JP5079284B2 - Camera shake amount detection device, camera shake correction device, imaging device, and camera shake amount detection method - Google Patents

Description

  The present invention relates to a camera shake amount detection apparatus, a camera shake correction apparatus, an imaging apparatus, and a camera shake amount detection method that perform camera shake correction.

In recent years, camera miniaturization has progressed as an imaging apparatus, and camera shake correction technology has attracted attention.
Due to the downsizing of the camera, the holdability is reduced in photography, and the degree of freedom is widened as compared with conventional photography forms such as photography with one hand as a photography method such as a camera equipped with a mobile phone.

As described above, when shooting with various methods or in an unstable state, when the shutter button is pressed, the camera shakes and camera shake occurs.
For example, when shooting under dark conditions with a long exposure time, there are many cases where the shutter speed becomes slow and blurring occurs.
In addition, when the sensitivity is increased under the exposure conditions and the exposure time is shortened, noise is added and the image becomes rough.

In order to solve this problem, a device for correcting camera shake is mounted on a single-lens reflex camera or recently a compact camera.
For example, an optical camera shake correction device that corrects an image by tilting the lens in accordance with a shake has been commercialized, but it is very difficult in terms of space in consideration of recent downsizing and mounting in a mobile phone camera.

  As other methods, various imaging devices that can obtain a blur-free image from a plurality of images by image processing have been proposed.

  For example, in Patent Literature 1, when combining captured image information with different exposure conditions to obtain an image with an expanded dynamic range, when synthesizing captured image information at substantially the same time obtained with pixels with different exposure conditions, An imaging apparatus that uses captured image information of at least one exposure condition in an overlapping manner is disclosed.

  In Patent Document 2, camera shake limit exposure time peculiar to a photographer is stored in advance based on detection information of an angle detection sensor provided in the camera, focal length information and aperture information of the camera, and appropriate exposure time is limited to camera shake. When the exposure time is longer than the exposure time, there has been disclosed a correction apparatus that performs camera shake correction by performing image capturing a plurality of times with a shorter exposure time and synthesizing images.

  In Patent Literature 3, two subject images having a luminance difference obtained by separating a light flux from a lens into two parts and attenuating one of the light fluxes are generated in different regions on the imaging surface of the image sensor. An image pickup apparatus is disclosed that obtains new image data having an expanded dynamic range by making the charge accumulation time and the charge accumulation start time equal for each piece of image data.

  In Patent Document 4, a motion vector is detected for each thinned block, and a motion vector is given to an undetected block by interpolation from the detected motion vector. A motion vector detection method for performing detection every time is disclosed.

  Patent Document 5 discloses an image compression method that performs image evaluation and transmits data with high definition in a portion judged to be important, and coarse definition in a portion with little movement or stillness.

Japanese Patent Laid-Open No. 10-108079 JP 2004-279514 A JP2003-32559A JP-A-7-177519 Japanese Patent Laid-Open No. 6-169552

However, the above-described camera shake correction method requires a large number of shots, and has a disadvantage that the processing time becomes long because a positional shift between images occurring during shooting is corrected.
An object of the present invention is to provide a camera shake amount detection device, a camera shake correction device, an imaging device, and a camera shake amount detection method that can perform high-precision camera shake amount detection with a small processing amount in order to eliminate the disadvantages described above. To do.

A camera shake amount detection apparatus according to the present invention is a camera shake amount detection apparatus that detects a camera shake amount by performing a matching process based on a reference image and a reference image having different exposure conditions, and each of the reference image and the reference image includes a plurality of macros. Macroblock dividing means for dividing into blocks, and a first block constituted by macroblocks of a predetermined size from the reference image divided by the macroblock dividing means, and the reference divided by the macroblock dividing means A block extraction unit that extracts a second block composed of macroblocks of a predetermined size from a macroblock of an image, and obtains a residual between the first block and the second block extracted by the block extraction unit and residual acquisition means for, data thinning unit for generating a thinned image by thinning the reference image and the reference image , Based on the residuals the residual acquisition means has acquired in rough search using the thinned-out image in which the data thinning unit has generated, acquires the residual integral value for each of the macroblocks, said residue difference integration A block selection unit that selects a predetermined number of macroblocks having a small value; and a shake amount that is a value of a predetermined order when the residual integral values of the macroblocks selected by the block selection unit are arranged in order from the largest or the smallest. And the residual acquisition means thins out a thinning amount smaller than the thinned-out image by a fine search for a macroblock having a small residual integral value selected by the block selection means. The residual of the image or the non-decimated image is acquired and output to the camera shake amount acquisition means.

A camera shake correction apparatus according to a second aspect of the present invention is a camera shake correction apparatus that detects a camera shake amount by performing matching processing based on a reference image and a reference image having different exposure conditions, and performs camera shake correction based on the camera shake amount. A macroblock dividing unit that divides the reference image and the reference image into a plurality of macroblocks, and a first block constituted by macroblocks of a predetermined size from the reference image divided by the macroblock dividing unit. Block extracting means for extracting a second block constituted by macroblocks of a predetermined size from the macroblock of the reference image divided by the macroblock dividing means; and a thinned image by thinning the reference image and the reference image a data thinning unit that generates, in coarse search using the thinned-out image in which the data thinning unit has generated Based on the residual acquired by the residual acquisition unit and the residual acquisition unit that acquires the residual of the first block and the second block extracted by the block extraction unit, a residual is obtained for each macroblock. A block selection means for obtaining a difference integral value and selecting a predetermined number of macroblocks having a small residual integral value; and arranging in order from the largest or smallest residual integral value of each macroblock selected by the block selection means. a shake amount acquiring means to shake amount the value of a given rank when the, on the basis of the shake amount calculated in the shake amount calculation means, have a image synthesizing means for synthesizing the images of the plurality, wherein The residual acquisition means is a thinned-out image that is smaller than the thinned-out image or a non-decimated-out image in a fine search for the macroblock having a small residual integrated value selected by the block selecting means. It gets the residual image, and outputs to the camera shake amount acquiring means.

An imaging apparatus according to a third aspect of the invention is an imaging apparatus that captures a subject image through an optical system,
Based on a reference image and a reference image with different exposure conditions, a camera shake amount detection device that detects a camera shake amount by performing a matching process, and the camera shake amount detection device converts each of the reference image and the reference image into a plurality of macroblocks. A first block composed of macroblocks of a predetermined size is extracted from the base image divided by the macroblock dividing unit, and the reference block divided by the macroblock dividing unit A block extracting means for extracting a second block composed of macroblocks of a predetermined size from a macroblock; a data thinning unit for thinning out the base image and the reference image to generate a thinned image; and the data thinning unit, wherein said block extracting means has extracted with coarse search using the generated the decimated image first Based on the residual acquired by the residual acquiring means for acquiring the residual of the block and the second block, and the residual acquired by the residual acquiring means, a residual integral value is acquired for each macroblock, and the residual integral is obtained. Block selection means for selecting a predetermined number of macroblocks having a small value, and a camera shake amount with a predetermined order value when the respective macroblocks selected by the block selection means are arranged in order from the smallest residual integral value possess a quantity acquisition unit, the residual acquisition means decimated image of the block the decimated image lesser amount of thinning in fine search the residual integral value selecting means has elected for small macroblock or, The residual of the non-decimated image is acquired and output to the camera shake amount acquisition means .

A camera shake amount detection method according to a fourth aspect of the present invention is a camera shake amount detection method for detecting a camera shake amount by performing a matching process based on a reference image and a reference image having different exposure conditions, and each of the plurality of reference images and reference images. A first block that is divided into macroblocks, and a first block constituted by macroblocks of a predetermined size from the reference image divided in the first step, and in the first step A second step of extracting a second block composed of macroblocks of a predetermined size from the macroblocks of the divided reference image, and a rough image using a thinned image generated by thinning out the reference image and the reference image a third step of obtaining a residual of the first and second blocks in the search, the residual obtained in the third step based on the front A fourth step of obtaining a residual integral value for each macroblock, selecting a predetermined number of macroblocks having a small residual integral value, and a residual integral of each macroblock selected in the fourth step; have a fifth step of the shake amount the value of a predetermined order when arranged from those values ascending order, the third step, the residual integral value is less was selected in the fourth step A sixth step of acquiring a residual of a thinned-out image or a non-thinned-out image that is smaller than the thinned-out image by a fine search with respect to a macroblock ;

  Provided are a camera shake amount detection device, a camera shake correction device, an imaging device, and a camera shake amount detection method that can perform high-precision camera shake amount detection with a small processing amount.

Hereinafter, the imaging device of the present invention will be described.
<First Embodiment>
FIG. 1 is a block diagram illustrating a configuration of the imaging apparatus 1 according to the first embodiment.
As illustrated in FIG. 1, the imaging apparatus 1 according to the first embodiment includes a camera 2, a camera shake amount detection unit 3, an image composition unit 4, and a buffer memory 5.

  The camera 2 includes a lens and an image sensor such as a CCD (Charged Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The image sensor photoelectrically converts light passing through the lens, and obtains color information by a color filter. It is a digital camera module that outputs as image data (RAW data). When performing camera shake correction at the time of image capture, the camera 2 captures a plurality of images that are continuously captured and that have different exposure conditions. In order to detect the amount of camera shake, one of the plurality of images is used as a standard image, and the remaining other images are used as reference images. The standard image and the reference image have different exposure conditions. Images taken by the camera 2 are temporarily stored in the buffer memory 5 for camera shake correction processing.

The camera shake amount detection unit 3 detects the amount of camera shake based on the standard image and the reference image that are captured by the camera 2 and stored in the buffer memory 5.
The camera shake amount detection unit 3 includes a data decimation unit 31, a macroblock division unit 32, a block extraction unit 33, a luminance difference comparison unit 34, a residual acquisition unit 35, a block selection unit 36, A quantity acquisition unit 37 is further included.

The data decimation unit 31 decimates the data of the standard image and the reference image raw data (RAW data) captured by the camera 2 to create a decimation standard image and a decimation reference image used for motion amount detection.
The RAW data is data as it is obtained from the image sensor of the camera 2. For example, RAW data taken by a camera having a primary color filter is Gr (green), R (red) as shown in FIG. ), B (blue), and Gb (green).
The data thinning unit 31 creates each thinned image obtained by extracting only the leading green data (Gr) from the four types of color data based on the RAW data of the standard image and the reference image captured by the camera 2. To do. In the present embodiment, only Gr is extracted, but the present invention is not limited to this, and any one of R, B, and Gb may be extracted.

  The macroblock dividing unit 32 divides the reference image and the reference image of the thinned image created by the data thinning unit 31 by thinning a part of the RAW data into a plurality of macroblocks. In the present embodiment, the macroblock dividing unit 32 divides each image into, for example, 10 × 10 macroblocks.

The block extraction unit 33 and the residual acquisition unit 35 calculate the absolute value of the difference between the pixel values of the block in the reference image and the block in the reference image corresponding to the residual sequential test method (SSDA: Sequential Similarity Detection Algorithms). Is calculated (residual) and template matching is performed.
Template matching is a technique for obtaining the degree of similarity or difference between a search target pattern (search block I) and a template (template block T) prepared in advance, and the residual sequential test method is one of the techniques. .
As shown in FIG. 3, the residual sequential test method includes a search block I composed of M × N pixels from the base image and a template block T composed of m × n pixels from the reference image. This is a method of extracting, moving the template block T on the search block I, performing comparison, and finding a position on the search block I that matches the template block T. Note that when M> m and N> n and the upper left coordinate of the template block T is (x, y), the residual, that is, the cumulative residual of the difference between the pixel values of the search block I and the template block T E (x, y), that is, a residual integral value E (x, y) obtained by adding all the pixels in the macro block for each macro block based on the residual is a function of (x, y). It is given by equation (1).

  The position of (x, y) at which the residual integral value E (x, y) is minimized is the position where the template block T and the search block I match.

In the present embodiment, as shown in FIG. 4A, the block extraction unit 33 extracts a search block I of 2 × 2 = 4 blocks from a 10 × 10 macroblock of the reference image, and further, a block extraction unit Reference numeral 33 denotes a block composed of two sides each having a length of 1/4 of the two vertical and horizontal sides of the macroblock located at the center of the 2 × 2 block of the corresponding reference image shown in FIG. A template block T composed of four is extracted.
Next, the residual acquisition unit 35 uses the search block I and the template block T extracted by the block extraction unit 33 to acquire a residual by the above-described residual sequential test method.

  In the residual sequential test method, the value of E (x, y) exceeds a predetermined threshold A while accumulating pixel value differences to calculate E (x, y). In this case, it means that the search block I and the template block T do not match at all, so that the calculation process of the residual integral value E (x, y) for the position (x, y) is stopped at that time, The process proceeds to calculation of a residual for the position (x, y). Thereby, the template matching by the residual sequential test method can shorten the processing time and the processing amount. The magnitude of the predetermined threshold A is not limited in the present invention.

Furthermore, in the present embodiment, the luminance difference comparison unit 34 includes each macroblock after the block extraction unit 33 extracts the search block I and the template block T until the residual acquisition unit 35 acquires the residual. On the other hand, the difference between the maximum value and the minimum value of the luminance value is taken, and if this difference is smaller than the predetermined threshold value B, the macroblock is excluded from the target from which the residual acquisition unit 35 acquires the residual. Perform the process. The threshold B is a predetermined ratio of the difference between the maximum value and the minimum value of the luminance values of all the pixels of the standard image or all the pixels of the reference image, or the maximum luminance value of all the pixels of the standard image and the reference image. What is necessary is just to set it as the predetermined ratio of the average value of the difference of a value and a minimum value.
That is, blocks with a small difference between the maximum and minimum luminance values are blocks with low contrast, such as white parts, and are not included in the motion amount detection process because the amount of camera shake is difficult to detect in such blocks. It ends up.
Note that the magnitude of the predetermined threshold B is not limited in the present invention.

The block selection unit 36 takes the difference between the maximum value and the minimum value of the luminance value for each macroblock, and if this difference is smaller than a predetermined threshold B, the block selection unit 36 After the block is excluded from the target from which the residual acquisition unit 35 acquires residuals, and the minimum values of the residual integral values E (x, y) in all search blocks I acquired by the residual acquisition unit 35 are compared. A predetermined number of search blocks I having a residual integral value with a small minimum value are extracted. Alternatively, instead of extracting a predetermined number of blocks, a predetermined percentage of the total number of blocks may be extracted.
In this embodiment, for example, 10 blocks are extracted.
The camera shake amount acquisition unit 37 obtains the median value when the integral value E (x, y) is obtained without decimating the macroblock selected by the block selection unit 36 in the search block I and arranged in order from the largest or smallest. The amount of camera shake of this image is calculated. In the present embodiment, the median value is used. However, the present invention is not limited to this and may be of a predetermined order.
The camera shake amount detection unit 3 detects the camera shake amount by the method described above.

Based on the camera shake amount detected by the camera shake amount detection unit 3, the image composition unit 4 synthesizes the reference image photographed by the camera 2 (not thinned out) and the reference image, and creates a corrected image in which camera shake correction is performed.
The buffer memory 5 is a RAM (Random Access Memory) that temporarily stores image data before correction used for camera shake correction processing.

Hereinafter, an operation example at the time of shooting of the imaging apparatus 1 of the present embodiment will be described.
FIG. 5 is a flowchart illustrating an operation example of the imaging apparatus 1.

Step ST1:
The camera 2 captures one standard image and one or a plurality of reference images while changing the exposure conditions.
Step ST2:
The data decimation unit 31 extracts only the Gr (green) component from the standard image and reference image raw data (RAW data) captured by the camera 2 in step ST1, and decimates the other components. A reference image and a thinned reference image are created.
Step ST3:
The macroblock dividing unit 32 divides the thinning standard image and the thinning reference image created by the data thinning unit 31 in step ST2 into 10 × 10 macroblocks.

Step ST4:
From the decimation reference image and decimation reference image which the block extraction unit 33 divides into 10 × 10 macroblocks in step ST3, as shown in FIG. The search block I of 2 × 2 = 4 blocks is extracted from 10 macroblocks, and as shown in FIG. 4B, the macroblock located at the center of the corresponding 2 × 2 block is extracted from the thinned reference image. A template block T composed of four blocks each composed of two sides each having a length of 1/4 of the two vertical and horizontal sides is extracted.

Step ST5:
The luminance difference comparison unit 34 takes the difference between the maximum value and the minimum value of the luminance value of each pixel. If this difference is smaller than a predetermined threshold value B, the residual acquisition unit 35 stores the macroblock. Exclude from the target to get the difference.
Step ST6:
The residual acquisition unit 35 uses the search block I and the template block T that are not excluded by the luminance difference comparison unit 34 in step ST5, and acquires a residual by the above-described residual sequential test method.

Step ST7:
The block selection unit 36 selects ten macroblocks acquired by the residual acquisition unit 35 and having a small residual integral value E (x, y) in all the search blocks I.
Step ST8:
The residual acquisition unit 35 reads from the buffer memory 5 the standard image and reference image from which RAW data has not been thinned out, and re-residuals the macroblock selected by the block selection unit in step ST7 based on the data that has not been thinned out. Calculate the integral value.
The processing of the residual acquisition unit 35 in this step is performed in order to increase the accuracy of the overall camera shake amount detection process of the camera shake amount detection unit 3.

Step ST9:
The camera shake amount acquisition unit 37 determines the position of the template block T in which the residual value has the minimum value in the search block I based on the residual integrated value for the non-thinned data acquired by the residual acquisition unit 35 in step ST8. Then, the integral value E (x, y) is obtained without thinning out the macroblocks selected by the block selection unit 36, and the median when they are arranged in order from the largest or smallest is extracted.

Step ST10:
Based on the camera shake amount acquired by the camera shake amount acquisition unit 37 in step ST9, the image composition unit 4 combines the reference image photographed by the camera 2 (not thinned out) with the reference image, and obtains a corrected image subjected to camera shake correction. create.

  As described above, according to the imaging apparatus 1 of the present embodiment, when detecting the amount of camera shake, the block selection unit 36 selects a small number of blocks having a small residual integral value by a predetermined number or ratio. Since the position of the template image that minimizes the residual integral value is obtained based on the predetermined number of blocks and the camera shake amount is detected based on the template image position, the overall amount of camera shake detection processing can be reduced. The processing time can be shortened.

  Further, according to the imaging apparatus 1 of the present embodiment, the data thinning unit 31 extracts only the Gr (green) component from the raw data (RAW data) of the standard image and the reference image, and extracts the other components. Since the thinned-out reference image and the thinned-out reference image are created by thinning out and the thinned-out image is used in the subsequent processing, the overall amount of camera shake detection processing can be reduced, and the processing time can be shortened. In addition, since the residual acquisition unit 35 acquires the residual integral value based on the thinned image and then acquires the residual integral value again based on the image that has not been thinned, highly accurate camera shake amount detection processing is performed. It can be carried out.

  Further, according to the imaging apparatus 1 of the present embodiment, the luminance difference between the time when the block extraction unit 33 extracts the search block I and the template block T and the time when the residual acquisition unit 35 acquires the residual integral value. The comparison unit 34 takes the difference between the maximum value and the minimum value of the luminance values for each macroblock, and if this difference is smaller than a predetermined threshold value B, the residual acquisition unit 35 selects the macroblock. Since the residual is excluded from the target for acquiring the residual, the block that is predicted to have a low contrast and cannot acquire the residual integrated value is excluded in advance from the process in which the residual acquiring unit 35 acquires the residual integrated value. The processing amount of the entire camera shake detection process can be reduced, and the processing time can be shortened.

  Further, according to the imaging apparatus 1 of the present embodiment, pattern matching is performed by the residual sequential test method that stops the residual calculation when the accumulated value exceeds a predetermined value when calculating the residual integral value. Therefore, the processing amount of the entire camera shake amount detection process can be reduced, and the processing time can be shortened.

Second Embodiment
In the second embodiment of the imaging apparatus, the data decimation unit 31 creates a 1/8 decimation image and a 1/2 decimation image, and the residual calculation unit 35 first uses the 1/8 decimation image to calculate the residual integral value. After obtaining (rough search), a predetermined number of blocks having a small residual integral value are extracted, and the residual integral value is also obtained again for the half-thinned image (fine search). It is the same as the imaging device 1 of one embodiment.

Hereinafter, each configuration different from the imaging device 1 of the first embodiment will be described. The data thinning unit 31 is a thinned image obtained by thinning the raw data of the standard image and the reference image captured by the camera 2 to 1/2 or 1/8. Create The half-thinned image is a thinned image created by extracting only one pixel in two of the vertical and horizontal pixels, and the 1/8 thinned image is one pixel in eight of the vertical and horizontal pixels. This is a thinned image created by extracting only.
The residual acquisition unit 35 first uses the search block I and the template block T of the 1/8 thinned-out image, and acquires a residual by the above-described residual sequential test method (coarse search). FIG. 6 shows the search block I and the template block T when the residual is acquired.
FIG. 6 is a diagram showing how the search block I is searched for the template block T. As shown in FIG. 6, the template block T searches the search block while moving one pixel at a time.

  The block selection unit 36 takes the difference between the maximum value and the minimum value of the luminance value for each macroblock, and if this difference is smaller than a predetermined threshold B, the block selection unit 36 After the block is excluded from the target from which the residual acquisition unit 35 acquires residuals, and the minimum values of the residual integral values E (x, y) in all search blocks I acquired by the residual acquisition unit 35 are compared. A predetermined number of search blocks I having a residual integral value with a small minimum value are extracted. Alternatively, instead of extracting a predetermined number of blocks, a predetermined percentage of the total number of blocks may be extracted.

Next, the residual integral value calculation unit 35 obtains an integral value E (x, y) as a half-thinned image for the nine macroblocks selected by the block selection unit 36 in the search block I (precisely). search).
When obtaining the residual integral value of the half-thinned image, the residual integral value obtained by the block extraction unit 33, the residual acquisition unit 35, and the block selection unit 36 described above for the 1/8 thinned image is small. Since it is only necessary to calculate the residual integral value for only the pixels in the vicinity of the pixel, the amount of calculation of the camera shake amount calculation unit is significantly smaller than that in the past.

Here, the relationship between the 1/8 thinned image and the 1/2 thinned image will be described in association with FIG.
The circles shown in FIG. 7 represent pixels for which the residual integral value calculation is performed in the 1/8 thinned image, and the x marks in FIG. 7 are used for the residual integral value calculation in the 1/2 thinned image. A pixel is shown. It can be seen that there are pixels for which the residual integral value calculation is performed in the 1/8 thinned-out image every four pixels in the vertical and horizontal directions of the pixels for which the residual integral value calculation is performed in the half-thinned image.
That is, in the 1/8 thinned image, if the pixel a in FIG. 7 is the pixel having the minimum residual integral value in the block, the 1/2 thinned image is within the range surrounded by the dotted line in FIG. It can be seen that there is a pixel having a minimum residual integral value.

Next, an operation example of the imaging device 1 of the present embodiment will be described.
FIG. 8 is a flowchart illustrating an operation example of the imaging apparatus 1 according to the second embodiment.

Step ST11:
The camera 2 captures one standard image and one or a plurality of reference images while changing the exposure conditions.
Step ST12:
The data decimation unit 31 creates a 1/8 decimation image and a 1/2 decimation image from the standard image and the reference image raw data captured by the camera 2 in step ST11.
Step ST13:
The macroblock dividing unit 32 divides the standard image and reference image of the 1/8 thinned image created by the data thinning unit 31 in step ST12 into, for example, 10 × 10 macroblocks.

Step ST14:
The block extraction unit 33 extracts the search block I from the standard image and the reference image of the 1/8 thinned-out image divided by the macroblock dividing unit 32 into 10 × 10 macroblocks in step ST13, and from the reference image, A template block T is extracted.

Step ST15:
The luminance difference comparison unit 34 takes the difference between the maximum value and the minimum value of the luminance value of each pixel. If this difference is smaller than a predetermined threshold value B, the residual acquisition unit 35 stores the macroblock. Exclude from the target to get the difference.
Step ST16:
The residual acquisition unit 35 uses the search block I and the template block T that are not excluded by the luminance difference comparison unit 34 in step ST15, and acquires the residual by the above-described residual sequential test method.

Step ST17:
The block selection unit 36 selects nine macro blocks acquired by the residual acquisition unit 35 and having a small residual integral value E (x, y) in all the search blocks I.
Step ST18:
The residual acquisition unit 35 calculates a residual integral value again with respect to the macroblock selected by the block selection unit in step ST <b> 17 for the standard image and the reference image of the half-thinned image.
The processing of the residual acquisition unit 35 in this step is performed in order to increase the accuracy of the overall camera shake amount detection process of the camera shake amount detection unit 3.

Step ST19:
The camera shake amount acquisition unit 37 determines the position of the template block T at which the residual is the smallest value in the search block I based on the residual integrated value for the half-thinned image acquired by the residual acquisition unit 35 in step ST18. In the macroblock selected by the block selection unit 36, the residual integrated value E (x, y) of the half-thinned image is obtained, and the median when they are arranged in order from the smallest is extracted.

Step ST20:
Based on the camera shake amount acquired by the camera shake amount acquisition unit 37 in step ST19, the image composition unit 4 combines the reference image photographed by the camera 2 (not thinned out) with the reference image, and obtains a corrected image that has undergone camera shake correction. create.

As described above, according to the imaging apparatus 1 of the second embodiment, after the block extraction unit 33 extracts the search block I and the template block T, the residual acquisition unit 35 acquires the residual integral value. In the meantime, the luminance difference comparison unit 34 takes the difference between the maximum value and the minimum value of the luminance value for each macroblock, and if this difference is smaller than a predetermined threshold value B, the macroblock is selected. Since the residual acquisition unit 35 excludes the residual from the target from which the residual is acquired, a process in which the residual acquisition unit 35 acquires the residual integral value in advance for blocks predicted to have low contrast and cannot acquire the residual integral value. Therefore, the processing amount of the entire camera shake detection process can be reduced and the processing time can be shortened.
As in the first embodiment, when detecting the amount of camera shake, the block selection unit 36 may select a small number of blocks having a small residual integral value by a predetermined number or ratio, and obtain the amount of camera shake. In the present embodiment, the unit 37 uses the median as the amount of camera shake in the present embodiment, but the present invention is not limited to this.

<Third Embodiment>
The third embodiment of the imaging apparatus 1 is different from the second embodiment except that only a residual in a predetermined range is calculated at the time of residual calculation of the residual calculation unit 35 for selecting a block having a small residual integral value. This is the same as the embodiment.
That is, the residual calculation unit 35 does not calculate the residual integral value for all the range of pixels when obtaining the residual integral value (coarse search) using the 1/8 thinned image, The residual is calculated for pixels in only the range.

A predetermined range in which the residual calculation unit 35 of the third embodiment calculates the residual during the coarse search will be described with reference to FIG.
As illustrated in FIG. 9, the residual calculation unit 35 performs the residual calculation only on a range where, for example, 5% of pixels exist in the image thinned out to 1/8. That is, if the image is 3 megapixels (2048 pixels × 1536 pixels), the residual is calculated only for 2048/5/20 = about 13 (pixels) in the horizontal direction, and 1535/8 in the vertical direction. Residual calculation is performed only for / 20 = about 10 (pixels). Therefore, the time required for calculating the residual is greatly reduced.

  The value of 5% is an example, and the present invention is not limited to this. The value of 5% is a value derived from experience that it is the minimum necessary to detect camera shake.

  As described above, the residual calculation unit 35 of the third embodiment uses the 1/8 thinned image to obtain the residual integrated value (coarse search), and applies to the pixels in the entire range. Since the residual is calculated only for pixels within a predetermined range without calculating the residual integral value, the time required for calculating the residual is greatly shortened, and the time required for camera shake correction can be reduced.

The present invention is not limited to the embodiment described above.
That is, those skilled in the art may make various modifications, combinations, subcombinations, and alternatives regarding the components of the above-described embodiments within the technical scope of the present invention or an equivalent scope thereof.

  In the present embodiment, the luminance difference comparison unit 34, after the block extraction unit 33 extracts the search block I and the template block T, until the residual acquisition unit 35 acquires the residual integral value, For each macroblock, the difference between the maximum value and the minimum value of the luminance value is taken, and if this difference is smaller than a predetermined threshold B, the residual acquisition unit 35 sets the residual integral value for that macroblock. However, the present invention is not limited to this. For example, the luminance difference comparison unit 34, the block selection unit 36, the residual integral value E (x, y) in all search blocks I. ) To obtain a luminance difference immediately after extracting a predetermined number of small search blocks I having a residual integral value having a small minimum value (that is, between steps ST9 and ST10 in FIG. 5). Brightness difference It may be performed to exclude processing from the object to get the residual integral values for the following block value.

  In this embodiment, the data thinning unit 31 extracts only the Gr component and thins out the remaining components. However, the present invention is not limited to this. For example, only the other components are extracted or two components are extracted. Or may be extracted.

  In the present embodiment, the camera shake detection unit 3 includes a data thinning unit 31, a macroblock division unit 32, a block extraction unit 33, a luminance difference comparison unit 34, a residual acquisition unit 35, and a block selection unit as a configuration for detecting the amount of camera shake. 36, the camera shake amount acquisition unit 37 is further included. However, each component of the camera shake detection unit 3 may be incorporated in the imaging apparatus 1 as hardware for performing the processing, or camera shake detection. Software that performs processing according to the control of the unit 3 may be used.

FIG. 1 is a block diagram illustrating a configuration of the imaging apparatus 1 according to the first embodiment. FIG. 2 is a diagram illustrating an example of the configuration of RAW image data. FIG. 3 is a diagram for explaining the residual sequential test method. FIG. 4 is a diagram illustrating a specific example of block extraction by the block extraction unit 33 in the present embodiment. FIG. 5 is a flowchart illustrating an operation example at the time of shooting of the imaging apparatus 1 of the present embodiment. FIG. 6 is a diagram showing how the template block T searches for the search block I. FIG. 7 is a diagram illustrating a range in which a pixel having a minimum residual integral value exists in the 1/8 thinned image and the 1/2 thinned image. FIG. 8 is a flowchart illustrating an operation example at the time of photographing of the imaging apparatus 1 according to the second embodiment. FIG. 9 is a diagram for explaining a predetermined range in which the residual calculation unit 35 of the third embodiment calculates the residual during the rough search.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 2 ... Camera, 3 ... Camera shake amount detection part, 31 ... Data thinning part, 32 ... Macroblock division part, 33 ... Block extraction part, 34 ... Luminance difference comparison part, 35 ... Residual acquisition part, 36 ... block selection unit, 37 ... blurring amount calculation unit, 4 ... image synthesis unit, 5 ... buffer memory

Claims (10)

Based on a reference image and a reference image with different exposure conditions, a camera shake amount detection device that detects a camera shake amount by performing a matching process,
Macroblock dividing means for dividing each of the standard image and the reference image into a plurality of macroblocks;
A first block composed of macroblocks of a predetermined size is extracted from the base image divided by the macroblock dividing unit, and a predetermined size is extracted from the macroblock of the reference image divided by the macroblock dividing unit. Block extraction means for extracting a second block constituted by macroblocks;
A residual acquisition unit that acquires residuals of the first block and the second block extracted by the block extraction unit;
A data thinning unit for thinning the standard image and the reference image to generate a thinned image;
Based on the residual acquired by the residual acquisition means in the coarse search using the thinned image generated by the data thinning unit , a residual integrated value is acquired for each macroblock, and the residual integrated value A block selection means for selecting a predetermined number of macroblocks having a small size,
A camera shake amount acquisition unit having a camera shake amount as a value of a predetermined order when a residual integral value of each macroblock selected by the block selection unit is arranged in order from the largest or the smallest;
Have
The residual obtaining unit obtains a residual of a thinned-out image or a non-thinned-out image that is smaller than the thinned-out image by a fine search with respect to a macroblock having a small residual integrated value selected by the block selecting unit. And a camera shake amount detection device for outputting to the camera shake amount acquisition means. The block extracting means extracts a block having a predetermined size located at the center of a plurality of adjacent macroblocks based on the reference image as the second block, and based on the reference image, the reference The camera shake amount detection apparatus according to claim 1, wherein blocks corresponding to a plurality of adjacent macroblocks corresponding to a plurality of adjacent macroblocks in an image are extracted as the first block. Before the residual acquisition unit acquires the residual, the difference between the maximum value and the minimum value of the luminance value is calculated for each macroblock, and the difference is smaller than a predetermined threshold value. 3. The camera shake amount detection apparatus according to claim 2, further comprising: a luminance difference comparison unit that, when determined, excludes a macroblock having the difference from a target from which the residual acquisition unit acquires the residual. After the block selection means selects the predetermined number of the macro blocks, the difference between the maximum value and the minimum value of the luminance value is calculated, and when it is determined that the difference is smaller than a predetermined threshold value, The camera shake amount detection device according to claim 2, further comprising: a luminance difference comparison unit that excludes a macroblock having a difference from a target from which the residual acquisition unit acquires the residual. The data thinning unit extracts only predetermined components of the RAW data of the reference image and the reference image, thinning and generating the decimated image,
The macroblock dividing unit and the block extracting unit use the thinned image generated by the data thinning unit instead of the standard image and the reference image. The camera shake amount detecting device according to the item. The residual acquisition means, when performing a fine search for a macroblock having a small residual integral value selected by the block selection means during the coarse search, in the neighborhood where the minimum residual within the block was acquired during the coarse search. The camera shake amount detection apparatus according to claim 1 , wherein residual acquisition is performed only on pixels. The residual acquisition unit does not acquire residuals of all ranges when acquiring residuals of the first block and second block extracted by the block extraction unit, but only residuals of a predetermined range. The camera shake amount detection device according to any one of claims 1 to 6, wherein a difference is acquired. A camera shake correction apparatus that performs a matching process based on a reference image and a reference image with different exposure conditions, detects a camera shake amount, and performs camera shake correction based on the camera shake amount,
Macroblock dividing means for dividing each of the standard image and the reference image into a plurality of macroblocks;
A first block composed of macroblocks of a predetermined size is extracted from the base image divided by the macroblock dividing unit, and a predetermined size is extracted from the macroblock of the reference image divided by the macroblock dividing unit. Block extraction means for extracting a second block constituted by macroblocks;
A data thinning unit for thinning the standard image and the reference image to generate a thinned image;
A residual acquisition means for acquiring residuals of the first block and the second block extracted by the block extraction means in a rough search using the thinned image generated by the data thinning unit ;
Based on the residual obtained by the residual obtaining means, obtaining a residual integrated value for each macroblock, and a block selecting means for selecting a predetermined number of macroblocks having a small residual integrated value;
A camera shake amount acquisition unit having a camera shake amount as a value of a predetermined order when a residual integral value of each macroblock selected by the block selection unit is arranged in order from the largest or the smallest;
Based on the shake amount calculated in the shake amount calculation means, have a image synthesizing means for synthesizing the images of the plurality,
The residual obtaining unit obtains a residual of a thinned-out image or a non-thinned-out image that is smaller than the thinned-out image by a fine search with respect to a macroblock having a small residual integrated value selected by the block selecting unit. And a camera shake correction apparatus that outputs the result to the camera shake amount acquisition means . An imaging device that captures a subject image through an optical system,
Based on a reference image and a reference image with different exposure conditions, a camera shake amount detection device that detects a camera shake amount by performing a matching process,
The camera shake detection device is
Macroblock dividing means for dividing each of the standard image and the reference image into a plurality of macroblocks;
A first block composed of macroblocks of a predetermined size is extracted from the base image divided by the macroblock dividing unit, and a predetermined size is extracted from the macroblock of the reference image divided by the macroblock dividing unit. Block extraction means for extracting a second block constituted by macroblocks;
A data thinning unit for thinning the standard image and the reference image to generate a thinned image;
A residual acquisition means for acquiring residuals of the first block and the second block extracted by the block extraction means in a rough search using the thinned image generated by the data thinning unit ;
Based on the residual obtained by the residual obtaining means, obtaining a residual integrated value for each macroblock, and a block selecting means for selecting a predetermined number of macroblocks having a small residual integrated value;
Have a hand shake amount acquiring means to shake amount the value of a predetermined order when said block selection means are arranged from those small residual integral values of the respective macroblocks selected in order,
The residual obtaining unit obtains a residual of a thinned-out image or a non-thinned-out image that is smaller than the thinned-out image by a fine search with respect to a macroblock having a small residual integrated value selected by the block selecting unit. And outputting to the camera shake amount acquisition means . A camera shake amount detection method for detecting a camera shake amount by performing a matching process based on a reference image and a reference image having different exposure conditions,
A first step of dividing each of the reference image and the reference image into a plurality of macroblocks;
A first block composed of macroblocks of a predetermined size is extracted from the base image divided in the first step, and a predetermined size is extracted from the macroblocks of the reference image divided in the first step. A second step of extracting a second block constituted by the macroblock;
A third step of acquiring residuals of the first block and the second block by a coarse search using a thinned image generated by thinning the standard image and the reference image ;
A fourth step of acquiring a residual integral value for each macroblock based on the residual obtained in the third step, and selecting a predetermined number of the macroblocks having a small residual integral value;
Have a fifth step of the shake amount the value of a predetermined order when arranged in order from the residual integral value of elected each macroblock in said fourth step is small,
In the third step, the residual of the thinned-out image or the non-thinned-out image, which is smaller than the thinned-out image in the fine search for the macroblock having a small residual integrated value selected in the fourth step, is obtained. A camera shake amount detection method comprising a sixth step of acquiring .

Images