JP4760484B2 - Camera shake correction apparatus, camera shake correction method, and program - Google Patents

Description

  The present invention relates to a camera shake correction apparatus and a camera shake correction method suitable for use in an imaging apparatus such as a digital camera or a digital video camera having a camera shake correction function.

Conventionally, in an imaging apparatus such as a digital camera or a digital video camera, as a method of correcting camera shake during moving image shooting or through image display, for example, the size of a moving image is made smaller than the size of an image captured by an image sensor. , cut out partial images required size from the captured image, while obtaining the partial image as a frame image, in the shooting, and detects the motion vector of the camera body, the partial image (frame in accordance with the detected motion vector There is a method of controlling the cutout position of the image), that is, the effective pixel area.

Then, for a method of detecting a motion vector as described above, it is known to detect by comparing the image obtained by the image and the currently captured the previously captured (for example, the following Patent Document 1).

  However, in the case where a motion vector is detected by comparing both images taken in succession as described above, the calculation amount is enormous if the entire area of the previous and next frame images is set as the comparison range (search range). Therefore, the search range must be limited to a part of the image (for example, four corners in the image). That is, there is a problem that the detection accuracy of the motion correction amount has to be lowered to some extent.

  The present invention has been made in view of such conventional problems, and a camera shake correction apparatus and a camera shake correction method capable of performing high-speed and high-precision camera shake correction, and a program used for realizing them. The purpose is to provide.

In order to solve the above-described problem, in the first aspect of the invention, the first histogram data and the second histogram data indicating each luminance distribution are obtained from the previous image and the current image captured by the image sensor. Histogram acquisition means, difference calculation means for calculating the difference between the first and second histogram data acquired by the histogram acquisition means, and the difference calculated by the difference calculation means, the same luminance is obtained. A luminance level determination unit that determines a specific luminance level having a difference in number of pixels that is greater than or equal to a predetermined value; Generation means for generating first and second specific luminance images, and first and second specific luminance images generated by the generation means The shake correction amount corresponding to the movement of the basis apparatus body was that a calculating means for calculating.

  Further, in the invention of claim 2, the luminance level determining means determines a plurality of the specific luminance levels for each luminance section obtained by dividing the entire luminance section into a plurality, and the generating means includes the previous image and From the current image, the first and second specific luminance images are generated for each of a plurality of specific luminance levels determined for each of a plurality of luminance intervals by the luminance level determination unit, and the arithmetic unit is A first computing means for computing a camera shake correction amount for each of the plurality of specific luminance levels based on the first and second specific luminance images generated for each of the plurality of specific luminance levels; And a second calculation means for calculating a final shake correction amount based on the shake correction amount calculated for each of the plurality of specific luminance levels by the first calculation means.

  Further, in the invention of claim 3, the brightness level determining means determines a plurality of the specific brightness levels for each brightness section obtained by dividing the entire brightness section into a plurality, and the generating means includes the previous image and From the current image, the first and second specific luminance images are generated for each of a plurality of specific luminance levels determined for each of a plurality of luminance intervals by the luminance level determination unit, and the arithmetic unit is Based on the first and second specific luminance images generated for each of the plurality of specific luminance levels, the camera shake correction amount corresponding to the movement of the apparatus body is calculated in a predetermined order for each of the plurality of specific luminance levels, After successfully acquiring the camera shake correction component for any one of the specific luminance levels, the acquisition of the camera shake correction component for another specific luminance level is stopped.

  According to a fourth aspect of the present invention, the histogram acquisition means acquires the first and second histogram data for each of the same divided areas in the previous image and the current image, and the luminance level. The determining unit determines the specific luminance level for each of the divided regions for the first and second histogram data acquired for each divided region by the histogram acquiring unit, and the generating unit is configured to determine the previous image and the current time. The first specific luminance image and the second specific luminance image are generated for each divided region determined by the luminance level determining unit, and the calculation unit is configured to generate the first and second specific luminance images generated for each divided region by the generating unit. Based on the second specific luminance image, a first calculation unit that calculates a camera shake correction amount corresponding to the movement of the apparatus main body for each divided region, and the first calculation unit It was assumed and a second calculating means for calculating a final shake correction amount based on the camera-shake correction amount calculated for each frequency band.

  According to a fifth aspect of the present invention, the first and second histograms in the previous image and the current image until the specific luminance level is determined by the luminance level determination unit. The data acquisition target region is gradually switched to a region having a larger area to acquire the first and second histogram data, and the generation unit determines the brightness level from the previous image and the current image. The first and second specific luminance images composed of pixels having the luminance of the specific luminance level first determined by the means are generated.

  According to a sixth aspect of the present invention, the first and second histograms in the previous image and the current image until the specific luminance level is determined by the luminance level determination unit. The data acquisition target region is switched to a region with a smaller area step by step to acquire the first and second histogram data, and the generation means determines the brightness level from the previous image and the current image. The first and second specific luminance images composed of pixels having the luminance of the specific luminance level determined last by the means are generated.

  In addition, in the invention of claim 7, each of the acquisition target areas that the histogram acquisition means switches stepwise is an independent area.

  In the invention according to claim 8, the histogram acquisition unit is configured to store the first and second histograms in the previous image and the current image until a specific luminance level is determined by the luminance level determination unit. The data acquisition target region is switched in a predetermined order to acquire the first and second histogram data, and the generation unit is first determined by the luminance level determination unit from the previous image and the current image. The first and second specific luminance images made up of pixels having specific luminance levels are generated.

Further, in the invention of claim 9, obtaining the first histogram data and the second histogram data indicating each luminance distribution from the previous image and the current image captured by the image sensor, The step of calculating the difference between the two from the acquired first and second histogram data, the step of determining a specific luminance level in which the difference in the number of pixels of the same luminance is greater than a predetermined value in the calculated difference, and the determined identification Generating a first and second specific luminance image made up of pixels having a luminance level from the previous image and the current image, and based on the generated first and second specific luminance images, and the method including the step of calculating the shake correction amount corresponding to the motion.

In the invention of claim 10, the first histogram data and the second histogram indicating the respective luminance distributions from the previous image and the current image captured by the image sensor are stored in the computer included in the camera shake correction apparatus. A process of acquiring histogram data, a process of calculating a difference between the acquired first and second histogram data, and a specific luminance level in which the difference in the number of pixels of the same luminance is greater than a predetermined value in the calculated difference. A process of determining, a process of generating first and second specific luminance images made up of pixels having the determined specific luminance level from the previous image and the current image, and the generated first and second and a program for executing a process of calculating the shake correction amount corresponding to the movement of the apparatus body based on the particular luminance image.
The invention of claim 11 further comprises setting means for setting a cutout position of an effective pixel area of a predetermined size cut out from the current image in accordance with the amount of camera shake correction calculated by the calculation means. .

As described above, according to the present invention, the efficiency based on the minimum necessary pixel information, which is a very small part constituting only the portion where the difference between the previous image and the current image is significant. the movement of the apparatus body because it has to allow you to detected manner, it is possible to perform the image stabilization of high speed and high accuracy.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a main part of an imaging apparatus such as a digital camera or a digital video camera that can shoot a moving image and includes the camera shake correction apparatus of the present invention.

  The imaging apparatus includes a CCD 2 that captures an optical image of a subject formed by the lens 1. The CCD 2 is driven by the vertical driver 3 and outputs a signal corresponding to the accumulated charge level of each pixel. This signal is sampled by a CDS (Correlated Double Sampling) circuit 4 based on a signal synchronized with the drive timing of the CCD 2, input to an AGC (Auto Gain Control) amplifier 5, multiplied by a predetermined gain, and then A / D converter 6. For example, it is digitized by 10 bits and is input as image data to the CCD interface unit 8 of the image processing unit 7 that supplies drive timing to the CCD 2. The CCD 2 may be an image sensor such as a CMOS sensor.

The image data input via the CCD interface unit 8 is subjected to processing such as data correction by the first data calculator 9 and then the external storage unit (in this case, SDRAM) via the DMA control unit 10. 11 is sequentially stored as Bayer data.

The image data (Bayer data) stored in the external storage unit 11 is read out by the second data calculator 12 via the DMA control unit 10 and then converted into a predetermined data format corresponding to the moving picture compressor 15 described later . Are converted, image data is corrected, and the like, and then sent to the reducer 13. The image data sent to the reducer 13 is stored in the external storage unit 11 via the DMA control unit 10 after being reduced from the recording size (pixel size) of the moving image. The external storage unit 11 is configured to hold the previous frame image data based on the data previously captured (captured) from the CCD 2, and when the new frame image data is stored, The data of the previous frame image is updated by the DMA control unit 10.

  Thereafter, the data of both the previous and current frame images stored in the external storage unit 11 are read out by the electronic image stabilizer 14, and the cutout position for the current frame image corresponding to the movement of the apparatus body is calculated there ( The details will be described later), and the data is output to the moving picture compressor 15.

The current frame image stored in the external storage unit 11 is read by the moving image compressor 15 and has a size corresponding to the recording size at the position indicated by the cutout position data input from the electronic camera shake corrector 14 there. After the image of the area (effective pixel area) is cut out, the data of the cut out image is finally compressed into data of a predetermined moving image format such as MPEG and stored in the external storage unit 11 as moving image data.

  FIG. 2 is a block diagram showing details of the electronic image stabilizer 14. As described above, the electronic camera shake corrector 14 outputs cutout position data indicating the position of the region to be cut out from the current frame image to the moving picture compressor 15, and includes a DMA interface unit 20 and a histogram calculation unit 21. , First and second histogram data storage units 22, 23, histogram data difference calculation unit 24, image conversion unit 25, first and second converted image storage units 26, 27, difference data calculation unit 28, motion correction amount It is comprised from each part of the calculating part 29 and the cut-out position setting part 30. FIG.

  The DMA interface unit 20 is a divided area in both the previous frame image and the current frame image with respect to the DMA control unit 10 at the timing when a new frame image is stored in the external storage unit 11. The data of 1/4 of the divided area divided into the left and right parts is requested, and each divided area data is transferred to the histogram calculation unit 21 and the image conversion unit 25.

  The histogram calculation unit 21 is a histogram acquisition unit of the present invention, and calculates the number of pixels at each luminance level by dividing the luminance into levels of 0 to 255 from the divided area data of both the previous and current frame images transferred. Then, histogram data indicating the luminance distribution in the divided areas of both frame images is generated. Then, the first histogram data on the previous frame image side is stored in the first histogram data storage unit 22, and the second histogram data on the current frame image side is stored in the second histogram data storage unit 23. Note that the resolution of the histogram may be half of the above (0 to 127).

  The histogram data difference calculation unit 24 is a difference calculation unit and a luminance level determination unit of the present invention, and calculates a difference between both from the histogram data stored in the first and second histogram data storage units 22 and 23. At the same time, a specific luminance level in which the difference in the number of pixels having the same luminance is greater than a predetermined value is determined, and data indicating the determined specific luminance level is output to the image conversion unit 25. FIG. 3A is a diagram conveniently showing a portion corresponding to a specific luminance level in which the difference in the number of pixels having the same luminance is greater than a predetermined value, out of the differences between the two histogram data.

Image conversion unit 25 is a generating unit of the present invention, respectively converting the divided area data have been both frame images transferred from the D MA interface unit 20, the particular luminance image composed of only the pixels having a luminance of the specific luminance level To do. Then, the converted image data on the previous frame image side is stored in the first converted image storage unit 26, and the converted image data on the current frame image side is stored in the second converted image storage unit 27. FIG. 3B is a diagram showing an example of the specific luminance image 100.

  The difference data calculation unit 28, together with the motion correction amount calculation unit 29, constitutes a calculation unit of the present invention, and both specific luminances stored in both the first and second converted image storage units 26 and 27 described above. While shifting the relative positional relationship of the image in the vertical and horizontal directions for each determined amount, the difference data between the two is calculated each time. That is, an operation is performed to integrate the absolute value of the difference between the pixels in the mismatch area between the two. Then, the calculation result (difference data) is output to the motion correction amount calculation unit 29.

  The motion correction amount calculation unit 29 calculates a motion correction amount based on the relative positional relationship between the two images when the difference data from the difference data calculation unit 28 is the smallest, and stores it in the internal register 29a.

  The histogram calculation unit 21 and the image conversion unit 25 described above each time the data of the second and subsequent 1/4 divided areas in the previous and current frame images are sequentially transferred from the DMA interface unit 20. The histogram data difference calculation unit 24, the difference data calculation unit 28, and the motion correction amount calculation unit 29 similarly repeat the above-described operations.

  Meanwhile, the motion correction amount calculation unit 29 calculates the motion correction amount for each divided region, stores the calculated motion correction amount in the register 29a as needed, and calculates the motion correction amount for all the divided regions. These average values are calculated, and the average value is output to the cutout position setting unit 30 as a motion correction amount for the entire image (a camera shake correction amount corresponding to the motion of the apparatus main body).

  The cutout position setting unit 30 is a setting unit of the present invention, calculates the position of the region to be cut out from the current frame image based on the motion correction amount from the motion correction amount calculation unit 29, and compresses the cutout position data to the moving image compression. Output to the device 15.

Based on according cutout position data, the video compressor 15 by shifting the cutout position of the image portion of the recording in the current frame image, the camera shake between frames is corrected.

  As described above, in the present embodiment, the difference between the histogram data of the previous frame image and the histogram data of the current frame image is calculated, and a specific luminance level in which the difference in the number of pixels having the same luminance is larger than a predetermined value is obtained. Determination is made, and a motion correction amount (camera shake correction amount) is calculated only for pixels having a specific luminance level in both frame images.

  Therefore, the motion correction amount can be calculated efficiently based on the minimum necessary pixel information as compared with the case where the motion correction amount is calculated using both frame images as they are. That is, the motion correction amount can be calculated in a short time. In addition, it is possible to obtain a motion correction amount with higher accuracy than when the motion correction amount is calculated only for specific portions of both frame images (for example, the four corners of the frame image). Therefore, camera shake correction can be performed at high speed and with high accuracy.

In the present embodiment, the electronic image stabilizer 14 uses the image data after being converted into a predetermined data format corresponding to the moving picture compressor 15 by the second data calculator 12 as a processing target. However, the motion correction amount may be calculated using the image data (Bayer data) before the conversion as a processing target.

  In addition, the previous and current frame images are divided into 1/4 divided areas, the motion correction amount for each divided area is calculated, and the average of them is used as the final motion correction amount. May be.

  That is, the previous and present frame images are not divided into four, but may be divided into two or eight, for example, and the motion correction amount may be calculated for each divided region, and the shape and area of each divided region are not necessarily the same. It doesn't have to be. Furthermore, the motion correction amount may be calculated by performing the above-described processing for the entire region without dividing both frame images.

  Further, when the motion correction amount for each divided region is averaged to calculate the final motion correction amount, each motion correction amount may be weighted for each region.

  Further, when the previous and current frame images are divided, the motion correction amount of any one divided region is finally calculated without calculating the motion correction amount for each divided region and averaging each motion correction amount. It may be adopted as a simple motion correction amount.

  In that case, for example, as shown in FIG. 4, the previous and current frame images 200 are divided into a first divided region 200 a to a fourth divided region 200 d whose area increases in order from the center toward the outside, The motion correction amount is calculated in order from the first divided area 200a, that is, in the order of FIG. 4A to FIG. 4D, while switching the processing target area (the hatched area in the figure). When the calculation of the motion correction amount is successful, the motion correction amount at that time is determined as the final motion correction amount, and thereafter, the calculation of the motion correction amount for other divided areas may be stopped. Good.

Contrary to the above, the motion correction amount is calculated while switching the processing target regions in order from the fourth divided region 200d, that is, in the order of FIG. 4 (d) to FIG. 4 (a). The motion compensation amount at the time when the effective motion compensation amount is successfully calculated first is determined as the final motion compensation amount, and thereafter, the motion compensation amount computation for other divided areas is stopped. Also good.

Further, the histogram data difference calculation unit 24 calculates the difference between both from the histogram data stored in the first and second histogram data storage units 22 and 23 and divides the entire luminance section into a plurality of sections. A plurality of specific luminance levels in which the difference in the number of pixels having the same luminance is greater than a predetermined value for each luminance section are determined , and the image conversion unit 25 causes the specific luminance image for each of the plurality of specific luminance levels from the previous and current frame images. May be calculated to calculate a motion correction amount for each of a plurality of specific luminance levels, and an average of them may be subjected to final motion correction.

In addition, the motion correction amount is not averaged as described above, and the motion correction amount is calculated in a predetermined order for each of the plurality of specific luminance levels, and the first effective motion correction amount is calculated for any specific luminance level. At the time of success, the motion correction amount at that time may be determined as the final motion correction amount, and thereafter, the calculation of the motion correction amount for other specific luminance levels may be stopped.

  In the above description, the case where the present invention is used for the purpose of correcting camera shake of a moving image recorded in the imaging apparatus is described. However, the present invention is not limited thereto, and the present invention is not limited to a still image in a digital camera. It can also be used for camera shake correction of a through image displayed on the monitor in the shooting standby state.

  In addition, the above-described data processing in each part of the electronic image stabilizer 14 shown in FIG. 2 is partly or entirely performed by, for example, a CPU (not shown) that controls the entire imaging apparatus. May be.

It is a block diagram which shows the principal part of the imaging device which concerns on this invention. It is a block diagram which shows the detail of an electronic camera shake correction device. (A) is a diagram showing a portion corresponding to a specific luminance level in which the difference in the number of pixels having the same luminance is greater than a predetermined value in the difference of histogram data, and (b) is a diagram showing an example of a specific luminance image. . It is a figure which shows the other example of the division area | region used as the calculation object of a motion correction amount.

Explanation of symbols

2 CCD
DESCRIPTION OF SYMBOLS 7 Image processing part 11 External storage part 14 Electronic camera shake corrector 15 Movie compressor 20 DMA interface part 21 Histogram calculation part 24 Histogram data difference calculation part 25 Image conversion part 28 Difference data calculation part 29 Motion correction amount calculation part 30 Cutout position setting Part

Claims (11)

Histogram acquisition means for acquiring first histogram data and second histogram data indicating respective luminance distributions from the previous image and the current image captured by the image sensor;
A difference calculating means for calculating a difference between the two from the first and second histogram data acquired by the histogram acquiring means;
In the difference calculated by the difference calculating means, a luminance level determining means for determining a specific luminance level in which the difference in the number of pixels having the same luminance is greater than a predetermined value;
Generating means for generating first and second specific luminance images composed of pixels having the luminance of the specific luminance level determined by the luminance level determining means from the previous image and the current image;
Calculating means for calculating a camera shake correction amount corresponding to the movement of the apparatus main body based on the first and second specific luminance images generated by the generating means;
Shake correction apparatus characterized by comprising a. The brightness level determination means determines a plurality of the specific brightness levels for each brightness section obtained by dividing the entire brightness section into a plurality of sections,
The generation unit generates the first and second specific luminance images for each of a plurality of specific luminance levels determined for each of a plurality of luminance sections by the luminance level determination unit from the previous image and the current image. ,
The calculation means calculates a camera shake correction amount corresponding to the movement of the apparatus main body for each of the plurality of specific luminance levels based on the first and second specific luminance images generated by the generation unit for each of the plurality of specific luminance levels. And a second calculation means for calculating a final camera shake correction amount based on the camera shake correction amount calculated for each of a plurality of specific luminance levels by the first calculation means. The camera shake correction device according to claim 1. The brightness level determination means determines a plurality of the specific brightness levels for each brightness section obtained by dividing the entire brightness section into a plurality of sections,
The generation unit generates the first and second specific luminance images for each of a plurality of specific luminance levels determined for each of a plurality of luminance sections by the luminance level determination unit from the previous image and the current image. ,
The arithmetic means predetermines a camera shake correction amount corresponding to the movement of the apparatus main body for each of the plurality of specific luminance levels based on the first and second specific luminance images generated by the generation unit for each of the plurality of specific luminance levels. The acquisition of a camera shake correction component for another specific luminance level is stopped after successful acquisition of a camera shake correction component for any one of the plurality of specific brightness levels. The described image stabilization device. The histogram acquisition means acquires the first and second histogram data for each same divided region in the previous image and the current image,
The brightness level determination means determines the specific brightness level for each of the divided areas for the first and second histogram data acquired for each divided area by the histogram acquisition means,
The generation unit generates the first and second specific luminance images for each divided region determined by the luminance level determination unit from the previous image and the current image,
The calculation means calculates first a camera shake correction amount corresponding to the movement of the apparatus main body for each divided area based on the first and second specific luminance images generated for each divided area by the generating means. And a second computing means for computing a final camera shake correction amount based on the camera shake correction amount computed for each divided region by the first computing means. apparatus. The histogram acquisition means is a region having a larger area as the acquisition target area of the first and second histogram data in the previous image and the current image until a specific brightness level is determined by the brightness level determination means. To obtain the first and second histogram data in a stepwise manner,
The generation means generates first and second specific luminance images composed of pixels having the luminance of the specific luminance level first determined by the luminance level determination means from the previous image and the current image. The camera shake correction apparatus according to claim 1, wherein: The histogram acquisition means is a region having a smaller area for the acquisition target areas of the first and second histogram data in the previous image and the current image until a specific brightness level is determined by the brightness level determination means. To obtain the first and second histogram data in a stepwise manner,
The generating means generates first and second specific luminance images composed of pixels having the luminance of the specific luminance level last determined by the luminance level determining means from the previous image and the current image. The camera shake correction apparatus according to claim 1, wherein:   The camera shake correction apparatus according to claim 5 or 6, wherein each of the acquisition target areas switched in stages by the histogram acquisition means is an independent area. The histogram acquisition means switches the acquisition target regions of the first and second histogram data in the previous image and the current image in a predetermined order until a specific brightness level is determined by the brightness level determination means. Obtaining the first and second histogram data;
The generating unit generates first and second specific luminance images each composed of pixels having luminance of a specific luminance level first determined by the luminance level determining unit from the previous image and the current image, respectively. The camera shake correction device according to claim 1. From an imaging element previous image and the current image captured by the steps of obtaining a first histogram data and the second histogram data indicating the respective luminance distribution,
A step of calculating a difference between the two from the acquired first and second histogram data;
A step of determining a specific luminance level in which the difference in the number of pixels having the same luminance is greater than a predetermined value in the calculated difference;
Generating first and second specific luminance images composed of pixels having the determined specific luminance level from the previous image and the current image;
Calculating a camera shake correction amount corresponding to the movement of the apparatus main body based on the generated first and second specific luminance images;
Shake correction method characterized by comprising a. In the computer that the image stabilization device has,
A process of acquiring first histogram data and second histogram data indicating respective luminance distributions from the previous image and the current image captured by the image sensor;
A process of calculating the difference between the two from the acquired first and second histogram data;
In the calculated difference, a process of determining a specific luminance level in which the difference in the number of pixels with the same luminance is greater than a predetermined value;
Processing for generating first and second specific luminance images made up of pixels having the determined specific luminance level from the previous image and the current image;
Processing for calculating a camera shake correction amount according to the movement of the apparatus main body based on the generated first and second specific luminance images;
Program for the execution. In accordance with the camera shake correction amount calculated by the calculation means, the image processing apparatus further includes setting means for setting a cut-out position of an effective pixel area of a predetermined size cut out from the current image.
9. The camera shake correction device according to claim 1, wherein

Images