JP2004343483A - Device and method for correcting camera-shake and device for detecting camera shake - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an inexpensive small-sized device which can perform camera-shake correction that is precise and excellently responsive. <P>SOLUTION: Input images YC1 and YC2 are contracted step by step over a plurality of steps and a scanning block in obtaining an amount of local movement at an upper layer level is established by utilizing an amount of global movement roughly obtained by using a contracted image at a lower layer level. Thus, it becomes possible to establish correlation with an image in a reference block set on a first image without enlarging the scanning block to be set on a second image so much. By allowing reduction in the size of the scanning block, the probability for a plurality of identical or similar patterns to exist in the scanning block becomes small, and inconveniences that an image in the reference block and an image in the scanning block are decided to be matched at a wrong position are suppressed. Also, since a range to be scanned becomes small, an amount of operation decreases and the responsiveness of camera-shake correction is improved. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a camera shake correction device and method, and a camera shake detection device, and more particularly to a camera shake correction function for preventing a deterioration in image quality of a captured image due to a camera shake applied to an imaging device.
[0002]
[Prior art]
Many imaging devices, such as still cameras and video cameras, have a camera shake correction function in order to prevent a reduction in image quality due to a camera shake applied to the camera. The camera shake at the time of shooting is a vibration in a frequency range of about 1 Hz to 15 Hz. As elementary technologies of a camera shake correction function for enabling shooting of an image without image blur even if such camera shake occurs, there are two technologies, a technology for detecting camera shake and a technology for correcting camera shake.
[0003]
The former technique of detecting camera shake is roughly classified into a motion vector detection method and an angular velocity detection method. The motion vector detection method is a method of electronically processing image data captured by a CCD (Charge Coupled Device) or the like to detect a motion vector of a camera. On the other hand, the angular velocity detection method is a method of detecting an angular velocity using a gyro sensor or the like. As a method of electronically detecting camera shake, a method of obtaining image blurring (image moving amount) by taking a cross-correlation between two image data captured at a time difference is common (for example, see Patent Document 1). ).
[0004]
[Patent Document 1]
JP-A-10-173992
[0005]
Generally, when calculating the image movement amount by looking at the cross-correlation of a plurality of image data, a block of a predetermined size (hereinafter, referred to as a reference block) cut out from a reference image is moved to anywhere in an image to be compared. A method of detecting whether or not it is used is taken. In this case, while scanning the image of the reference block on the image to be compared, the similarity between the two images at each scanning position is calculated, and it is determined that the reference block has moved to the scanning position where the similarity is the maximum. . The scanning range may be the entire comparison target image, but a scanning block having a size larger than the reference block is set assuming a range that can occur as a camera shake amount, and scanning is performed only inside the scanning block. Many.
[0006]
Further, the latter technique for correcting camera shake is roughly classified into an optical type and an electronic type. The optical camera shake correction is a method of physically displacing the optical axis in accordance with the detected movement of the camera. As a method for displacing the optical axis, there are a shift method in which a shift lens arranged in a part of the lens optical system is moved in accordance with the amount of camera shake, and a prism method in which a variable prism is deformed in accordance with the amount of camera shake.
[0007]
On the other hand, in the electronic camera shake correction, a CCD larger than an actual imaging area is prepared, and a part of the CCD is used as an imaging area. That is, this is a method of correcting camera shake by moving the image cutout area (imaging area) of the CCD up, down, left, and right in accordance with the detected movement of the camera.
[0008]
[Problems to be solved by the invention]
However, if an angular velocity detection method is used as a technique for detecting camera shake or an optical method is used as a technique for correcting camera shake, it is necessary to provide a mechanism such as a gyro or actuator. Therefore, there has been a problem that not only the size of the camera is increased, but also the cost for realizing the product increases.
[0009]
In addition, if the motion vector detection method is used as the technology for detecting camera shake and the electronic method is used as the technology for correcting camera shake, camera movement can be detected and the camera shake corrected using only software processing. Suitable for. However, in this case, the correction accuracy of the camera shake and the processing speed greatly depend on the contents of electronic processing.
[0010]
Generally, an attempt to increase the correction accuracy complicates electronic processing contents, increases the processing load, and lowers the processing speed. Conversely, if the processing content is simplified in order to increase the processing speed, erroneous detection of the image movement amount will increase, and the correction accuracy will decrease. In particular, there is a problem that it is extremely difficult to accurately detect the amount of image movement with a small amount of calculation.
[0011]
One of the causes of the erroneous detection of the image movement amount is that a simple method of cross-correlating a plurality of images is used. For example, when a similar pattern exists at a plurality of locations in a subject, the similar pattern appears in any number of locations in the captured image data. In this case, even if an attempt is made to detect the amount of image movement by a simple matching operation, a plurality of points that are considered to be matched are obtained, and it is not possible to determine which matching state is correct.
[0012]
On the other hand, it is conceivable to obtain the image movement amount using a more complicated calculation method than the matching calculation. However, the number of pixels in a recent CCD has increased dramatically, and the data amount of an image to be captured has also become extremely large. Therefore, it takes a lot of time to perform the operation on the image data. The longer the calculation time, the worse the responsiveness of hand shake detection. If a CPU having a large memory space at the time of calculation and capable of high-speed processing is used, the responsiveness of camera shake detection can be improved to some extent, but there is a problem that the cost increases.
[0013]
Further, in a case where the brightness of the subject is low, such as when shooting in a dark place, shooting is often performed by emitting strobe light for illuminating the subject. However, in places where flash photography is prohibited, strobe light cannot be emitted. Also, when the subject is far from the camera, the strobe light does not reach the subject and the brightness of the subject cannot be increased.
[0014]
On the other hand, a method has been proposed in which the sensitivity is increased by increasing the gain of image data captured without projecting light in a dark place. However, since image data captured in a dark place has a low S / N ratio, simply increasing the gain of the image data also increases the noise gain at the same time, making it impossible to obtain a high-quality image. There was a problem.
[0015]
The present invention has been made to solve such a problem, and an object of the present invention is to be able to perform accurate and responsive camera shake correction while being small in size and low in cost.
Another object of the present invention is to make it possible to accurately detect and correct camera shake even when shooting in a dark place where the brightness of the subject is small, and to obtain a bright and high-quality image. I do.
[0016]
[Means for Solving the Problems]
The image stabilization apparatus of the present invention reduces the image in one or more stages, including the original image, for each of a plurality of image data generated by imaging within a predetermined time by the imaging unit. Image reduction means for generating a plurality of levels of image data, and a plurality of levels including at least one reference image data of the plurality of image data generated by the imaging means and reference reduced image data generated therefrom by the image reduction means A feature point extracting unit for extracting a feature point satisfying a predetermined condition from the image for each of the image data, and a lowermost level to a highermost level of the multi-level image data generated by the image reducing unit. Using the feature points extracted by the feature point extraction means in order to obtain a cross-correlation between the reference image and the other images. Therefore, the image movement amount detection means for obtaining the image movement amount, and the reference image data at the uppermost level generated by the imaging by the imaging means and the other image data are compared with the image movement amount obtained by the image movement amount detection means. Image synthesizing means for synthesizing the image by shifting by an amount.
[0017]
In another aspect of the present invention, the feature point extracting unit performs an adjacent pixel difference integration operation on one image data for each of a plurality of levels of image data for each divided block of a predetermined size. The difference integrated data is subjected to one or more scans of a filter of a predetermined size in which a coefficient is set so as to apply a large weight to the median value and a small weight to the surrounding value, and to obtain a data value It is characterized in that corresponding coordinates are extracted as a plurality of feature points in order from a large block.
[0018]
BEST MODE FOR CARRYING OUT THE INVENTION
(1st Embodiment)
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a functional block diagram illustrating a schematic configuration of the camera shake correction apparatus according to the first embodiment. FIG. 2 is a block diagram showing the entire configuration of the digital camera 10 to which the camera shake correction device according to the first embodiment is applied.
[0019]
As shown in FIG. 2, the digital camera 10 includes an optical system 1 including a shutter 1a, a lens 1b, and an iris 1c, an imaging device 2 such as a CCD or a CMOS, a programmable gain amplifier (PGA) 3, an A / D It includes a converter 4, a signal processing unit 5, a camera shake correction unit 6, an AE / AF processing unit 7 for performing automatic exposure adjustment and automatic focus adjustment processing, and a controller 8 for controlling the entire digital camera 10. It is configured.
[0020]
In the digital camera 10 configured as described above, light incident on the optical system 1 is imaged by the image sensor 2 via the shutter 1a, the lens 1b, and the iris 1c. The imaging element 2 photoelectrically converts the formed incident light to generate an analog imaging signal corresponding to the incident light. After the gain of the imaging signal generated here is increased by the PGA 3, it is supplied to the A / D converter 4 and converted into digital image data. As described above, by increasing the gain of the analog imaging signal in the PGA 3, it is possible to reduce the adverse effect that the gradation in the A / D converter 4 becomes coarse due to the quantization.
[0021]
The image data obtained by the A / D converter 4 is supplied to a signal processing unit 5. The signal processing unit 5 performs color interpolation processing, color correction processing, RGB (red, green, and blue primary color signals) on input image data to YCbCr (a luminance signal, a blue color difference signal, and a red color difference signal). Various kinds of signal processing including conversion processing to, etc. are performed. The camera shake correction unit 6 is configured as shown in FIG. 1, synthesizes images while performing camera shake correction based on the luminance data received from the signal processing unit 5, and outputs a synthesized image.
[0022]
In the above configuration, the optical system 1 and the imaging device 2 constitute an imaging unit. The imaging unit according to the present embodiment captures a plurality of images within a predetermined time when a shutter button (not shown) is pressed. The A / D converter 4 sequentially A / D converts a plurality of imaging signals. Further, the signal processing unit 5 separates the plurality of A / D-converted image data into a luminance signal (luminance data) and a color difference signal (color difference data), and sequentially outputs them to the camera shake correction unit 6.
[0023]
As shown in FIG. 1, the image stabilization device (the image stabilization unit 6 shown in FIG. 2) of the present embodiment includes an image data input unit 11, an image reduction unit 12, a feature point extraction unit 13, and a global movement amount detection unit 14. , A local movement amount detection unit 15 and an image synthesis unit 16.
[0024]
The image data input unit 11 receives from the signal processing unit 5 a plurality (for example, two) of image data YC1 and YC2 generated by imaging within a predetermined time by the above-described imaging unit. The image reduction unit 12 reduces the luminance data of each of the plurality of image data YC1 and YC2 input by the image data input unit 11 in one or more steps, thereby including a plurality of levels including the original image. Is generated.
[0025]
For example, the input luminance data is sequentially reduced by 回 times over three times to obtain four levels of 1 × (original image), 倍 times, ４ times, １ ／ times. Is generated. By performing such reduction processing, four levels of luminance data Y1-1 to Y1-4 are generated from the first input image YC1, and four levels of luminance data Y2 are generated from the second input image YC2. -1 to Y2-4 are generated. Hereinafter, the highest level is referred to as a first level, and the lowest level is referred to as a fourth level.
[0026]
The content of the reduction process performed here is not particularly limited, but preferably, an average reduction process described below is performed. FIG. 3 is a diagram for explaining the contents of the average reduction process. As shown in FIG. 3, in the average reduction processing, adjacent four pixels of 2 × 2 pixels are regarded as one block, and luminance data of the reduced image is obtained by averaging the luminance data in block units. For example, the luminance data of the four pixels P1 to P4 constituting a certain block are averaged to obtain the luminance data of the pixel P1 ′ in the reduced image.
[0027]
The feature point extracting unit 13 includes at least one image data (hereinafter, referred to as reference image data) of the plurality of image data input by the image data input unit 11 and reduced image data (hereinafter, referred to as reference image data) generated by the image reducing unit 12. For each of a plurality of levels of image data including the reduced image data, feature points that satisfy a predetermined condition regarding luminance data are extracted from each image.
[0028]
For example, when the first input image YC1 is used as a reference image, the feature point extracting unit 13 extracts feature points satisfying a predetermined condition from each of the four levels of luminance data Y1-1 to Y1-4. Here, a point where the pattern is relatively complicated and a point where the same pattern does not exist around the point is extracted as a feature point. FIG. 4 is a diagram for describing extraction of a feature point. As shown in FIG. 4, when feature points are extracted from an image in which a house is drawn on a plain background, for example, seven feature points K1 to K7 shown in FIG. 4 are extracted. Detailed processing contents for extracting such a feature point will be described later.
[0029]
The global movement amount detection unit 14 corresponds to a first image movement amount detection unit of the present invention. The global movement amount detection unit 14 uses the luminance data Y1-4 and Y2-4 of the fourth level (１ ／) image generated by the image reduction unit 12 to generate the two luminance data Y1-4. , Y2-4 (reference reduced image data and other reduced image data) to obtain an image movement amount at the fourth level. The global movement amount obtained here is supplied to the local movement amount detection unit 15, and is used for detecting the local movement amount at the upper level.
[0030]
FIG. 5 is a diagram for explaining the meaning of the global movement amount. As shown in FIG. 5, when the image pickup position of the house is displaced between the first image YC1 and the second image YC2 due to the occurrence of camera shake, it is determined how much the position is displaced on the fourth level. What is represented is the global movement amount. That is, the house drawn in the first 1/8 reduced image and the house drawn in the second 1/8 reduced image are determined based on the first 1/8 reduced image. The movement amount of the first 1/8 reduced image necessary for making them substantially match is the global movement amount.
[0031]
The local movement amount detection unit 15 corresponds to a second image movement amount detection unit of the present invention. The local movement amount detection unit 15 calculates the luminance data of the image at each level (1 ×, １ ／ ×, １ ／ ×) other than the fourth level between the first image and the second image. By calculating the cross-correlation, a local image movement amount is obtained for each of a plurality of feature points included in the image. Then, one likely movement is selected from the plurality of local movement amounts obtained for each feature point, and one local movement amount at each level is determined.
[0032]
FIG. 6 is a diagram for explaining the meaning of the local movement amount. As shown in FIG. 6, when the position of the feature point is shifted between the first image YC1 and the second image YC2 due to the occurrence of camera shake, how much the position is shifted on the third to first levels. Is the local movement amount. That is, based on a local image around a feature point included in the first overall image, the first image required to make the first and second images substantially coincide with each other Is the local moving amount of the image.
[0033]
When calculating such a local movement amount, the local movement amount detection unit 15 places a predetermined size around the feature points K1 to K7 extracted by the feature point extraction unit 13 on the first image. Set the reference block. On the other hand, on the second image, a scanning block having a predetermined size larger than the reference block is set (the setting position of the scanning block will be described later). Then, by cross-correlating the luminance data of the reference block set in the first image and the luminance data of the scan block set in the second image, each of the feature points K1 to K7 is obtained. To obtain the local movement amount.
[0034]
FIG. 6 shows the local movement amount for one feature point K2. When seven feature points K1 to K7 are extracted as shown in FIG. 4, seven local movement amounts are obtained for each feature point. In the present embodiment, one of the seven local movement amounts having the highest similarity frequency is selected. The highest similarity frequency refers to a local movement amount corresponding to the largest group when all similar local movement amounts within a predetermined error range are regarded as the same group local movement amount.
[0035]
When there are a plurality of local movement amounts having the same similarity frequency, one of the local movement amounts having the highest equivalent frequency is selected from them. The one having the highest equivalent frequency means the local movement amount of a group including the most completely identical local movement amounts. If the equivalent frequencies are also the same, the one closest to the zero vector is selected from those having the highest equivalent frequency.
[0036]
Note that, here, one of the plurality of local movement amounts obtained for each of the plurality of feature points is selected as a likely one, but a predetermined calculation is performed using all or a part of the plurality of image movement amounts. Then, one local movement amount calculated in this way may be determined as the local movement amount at each level. Here, various calculations can be applied to the predetermined calculation. For example, it is possible to use an operation for calculating an average vector of a plurality of image movement amounts.
[0037]
FIG. 7 is a diagram illustrating a use relationship between the global movement amount and the local movement amount. As shown in FIG. 7, the global movement amount is obtained by the global movement amount detection unit 14 for the fourth level image. This global movement amount is used when a scan block is set in a process of obtaining a local movement amount for a third level image on one layer.
[0038]
On the other hand, for the images of the third level to the first level, the local movement amount detection unit 15 calculates the local movement amount. The local movement amounts obtained at the third level and the second level are used when a scan block is set in a process of obtaining a local movement amount at a higher level (second level and first level). The local movement amount obtained at the first level is supplied to the image synthesizing unit 16 as the finally obtained camera shake amount.
[0039]
The image synthesizing unit 16 converts the second image data YC2 with respect to the first image data YC1 input by the image data input unit 11 into the final camera shake amount obtained by the local movement amount detection unit 15. The combined image is shifted in the reverse direction by an amount to generate a combined image YC corrected for camera shake. Various modes such as addition synthesis and average synthesis can be applied to the synthesis method, but in the present embodiment, average synthesis is preferable. The composition may be performed after performing processing such as affine transformation.
[0040]
Here, the setting position of the above-described scanning block will be described. When calculating the local movement amount for the third level image, the local movement amount detection unit 15 starts from the positions corresponding to the feature points K1 to K7 included in the first image on the second image. A scanning block having a predetermined size larger than the reference block is set around a position shifted by the global movement amount obtained at the fourth level. Note that a scanning block of a predetermined size may be set at a position shifted from the position of the reference block set on the first image by the amount of the global movement by a method different from this.
[0041]
Further, when obtaining the local movement amount for the second and first level images, one local movement amount is obtained from the positions corresponding to the feature points K1 to K7 included in the first image on the second image. A scan block of a predetermined size larger than the reference block is set around a position shifted by the local movement amount obtained at the lower layer level. Note that a scanning block of a predetermined size is set by a different method from the position of the reference block set on the first image at a position shifted by the local movement amount obtained at the lower level. You may do it.
[0042]
By the way, when calculating the global movement amount at the fourth level, since the correlation between the 1/8 reduced images is taken, the pictures of the houses in both images can be made to substantially match as shown in FIG. However, when this is converted into a first level image, there is an error of less than 8 pixels. That is, the global movement amount includes an error of less than 8 pixels. However, it is broadly correct.
[0043]
In the present embodiment, the position of the scanning block is determined at the third level of the upper layer using the roughly matched image movement amount. That is, based on the global movement amount, the scanning block is set at a position where it is expected that the same image as the reference block exists. Thus, even if the scan block is not so large, the image in the scan block substantially matches the image in the reference block, and can be correlated with the image in the reference block.
[0044]
Similarly, the local movement amount obtained at the third level includes an error of less than four pixels, but this is also generally correct. In addition, the error amount is reduced to half compared with the fourth level. In the present embodiment, the position of the scanning block is determined at the second upper level using the image movement amount obtained at the third level. Similarly, the position of the scanning block is determined at the first level of the uppermost layer using the image movement amount obtained at the second level, and the local movement amount obtained at the first level is used as the camera shake amount. The local movement amount obtained at the first level does not include an error, and an accurate camera shake amount is obtained.
[0045]
FIG. 8 is a functional block diagram illustrating the configuration of the camera shake correction apparatus according to the present embodiment in more detail. Note that in FIG. 8, components denoted by the same reference numerals as those illustrated in FIG. 1 have the same functions, and thus redundant description will be omitted here. As shown in FIG. 8, the global movement amount detection unit 14 includes a reference histogram generation unit 21, a scanning histogram generation unit 22, and a histogram matching unit 23. The local movement amount detection unit 15 includes a reference block setting unit 24, a scanning block setting unit 25, and a block matching unit 26.
[0046]
The reference histogram generation unit 21 generates a histogram relating to a horizontal edge component and a vertical edge component for the ８ reduced luminance data Y1-4 generated from the first image data YC1. In addition, the scanning histogram generation unit 22 generates a histogram relating to a horizontal edge component and a vertical edge component with respect to the ８ reduced luminance data Y2-4 generated from the second image data YC2.
[0047]
FIG. 9 is a diagram for explaining the generation processing of the reference histogram and the scanning histogram. Since the generation processing of the reference histogram and the generation processing of the scanning histogram are almost the same, the generation of the reference histogram will be described here as a representative. As shown in FIG. 9, the reference histogram generation unit 21 first performs vertical edge extraction and horizontal edge extraction filter processing on the ８ reduced luminance data Y1-4, so that the vertical edge components and the horizontal The edge components in the directions are extracted.
[0048]
Then, a vertical direction histogram is generated by projecting and integrating pixel values in the vertical direction with respect to the image from which the edge components in the vertical direction have been extracted. Further, a horizontal direction histogram is generated by projecting and integrating pixel values in the horizontal direction with respect to the image from which the edge components in the horizontal direction have been extracted.
[0049]
By performing such processing on the two 1/8 reduced luminance data Y1-4 and Y2-4, a vertical reference histogram and a horizontal reference histogram are obtained from the first 1/8 reduced luminance data Y1-4. The vertical scanning histogram and the horizontal scanning histogram are generated from the generated 1/8 reduced luminance data Y2-4.
[0050]
The histogram matching unit 23 performs a matching process between the reference histogram generated by the reference histogram generation unit 21 and the scanning histogram generated by the scanning histogram generation unit 22, and obtains a cross-correlation between the two histograms to reduce the amount of global movement. Ask. FIG. 10 is a diagram for explaining the content of the histogram matching process, and shows a matching process of the horizontal direction histogram as an example.
[0051]
As shown in FIG. 10, when camera shake occurs, a positional shift occurs in the vertical direction between the reference histogram and the scanning histogram. This shift amount x corresponds to the vertical movement amount in the vertical direction. The histogram matching unit 23 calculates, for each scanning position, a difference between the horizontal reference histogram and the horizontal scanning histogram, for example, while shifting the position of the horizontal reference histogram little by little. Determined as the vertical component.
[0052]
As described above, in the example of FIG. 10, when the horizontal reference histogram is shifted to the right by x, the difference between the horizontal reference histogram and the horizontal scanning histogram is smallest, and the shift amount x is the vertical component of the global movement amount. The same process is performed on the vertical histogram to obtain the horizontal component of the global movement amount.
[0053]
Next, the reference block setting unit 24 sets a reference block of a predetermined size around the feature point extracted by the feature point extraction unit 13 around the first image.
[0054]
The scan block setting unit 25 calculates the local movement amount for the third level image from the position corresponding to the feature point included in the first image on the second image. A scanning block of a predetermined size larger than the reference block is set around the position shifted from the position shifted by the global movement amount obtained by the above. Alternatively, a scan block of a predetermined size larger than the reference block is set at a position shifted from the position of the reference block set by the reference block setting unit 24 by the amount of global movement obtained by the histogram matching unit 23.
[0055]
When calculating the local movement amount for the second and first level images, the block matching unit 26 starts from the position corresponding to the feature point included in the first image on the second image. A scan block of a predetermined size larger than the reference block is set around a position shifted by the local movement amount obtained at the level of one lower layer. Alternatively, a scan block of a predetermined size larger than the reference block is set at a position shifted from the position of the reference block set by the reference block setting unit 24 by the local movement amount of the lower level obtained by the block matching unit 26. I do.
[0056]
The block matching unit 26 performs a cross-correlation between the luminance data of the reference block set in the first image and the luminance data of the scan block set in the second image, thereby obtaining each feature point. To obtain the local movement amount. Then, one of the obtained local movement amounts having the highest similarity frequency is selected as the image movement amount at each level. When there are a plurality of local movement amounts having the same similarity frequency, one of the local movement amounts having the highest equivalent frequency is selected from them. If the equivalence frequencies are also the same, the one closest to the zero vector is selected.
[0057]
Next, details of the feature point extraction processing will be described. FIG. 11 is a flowchart showing the detailed processing contents of the feature point extracting unit 13. 11, first, the feature point extraction unit 13 receives the fourth-level luminance data Y1-4 generated from the first reference image YC1 by the image reduction unit 12 (step S1), and receives a predetermined size (for example, , 8 pixels in the vertical direction × 8 pixels in the horizontal direction (block S2).
[0058]
As shown in FIG. 12, the adjacent pixel difference integration refers to an operation of sequentially calculating the differences between adjacent pixels in a vertical direction and a horizontal direction in a block and adding them. By this adjacent pixel difference integration operation, 8 × 8 pixel values included in one block are replaced with one difference integration data. In other words, the luminance data Y1-4, which has been reduced by a factor of 8 from the original image, is further reduced by a factor of 8 by obtaining difference integration data one by one for each divided block. At this time, large integrated difference data is obtained for a block having a complicated pattern and a relatively large difference between adjacent pixels.
[0059]
Next, the feature point extracting unit 13 determines a predetermined size (for example, 3 vertical pixels × 3 horizontal pixels: one difference integration data is 1 pixel) for the 1/64 reduced image composed of the difference integration data. An eight-direction Laplacian filter (corresponding to one pixel) is applied (step S3). The Laplacian filter is a filter in which a coefficient is set so as to apply a large weight to the median value and a small weight to the surrounding value, and is configured, for example, as shown in FIG. In step S3, a filter coefficient is integrated with the above-described difference integrated data while scanning such a Laplacian filter on the screen.
[0060]
That is, when attention is paid to a certain pixel, the difference integrated data of the pixel of interest is multiplied by eight, the difference integrated data of the surrounding eight pixels is multiplied by −1, and all of them are added. Data. Such filter processing is sequentially performed while scanning the pixel of interest.
[0061]
By such a filtering process, at the position where the value of the difference integrated data of the target pixel is large and the value of the difference integrated data of the surrounding pixels is small as shown in FIG. growing. On the other hand, as shown in FIG. 14B, at the position where the value of the difference integrated data of the target pixel and the surrounding pixels is large, the calculated corrected difference integrated data is relatively small. Even at a position where the value of the difference integrated data of the target pixel and its surrounding pixels is both small, the calculated corrected difference integrated data is relatively small.
[0062]
In this way, by applying a Laplacian filter as shown in FIG. 13 to an image composed of the difference integrated data, it becomes possible to specify a position where the same pattern does not exist around a complicated pattern (where the frequency is high). . In the present embodiment, a Laplacian filter is again applied to the corrected difference integrated data obtained in step S3 (step S4). Thereby, as shown in FIG. 14 (c), the range for ensuring that the same picture does not exist around it can be widened.
[0063]
Through the above processing, corrected difference integrated data is obtained for the number of divided blocks of 8 × 8 units. The feature point extraction unit 13 selects n (n is preferably 3 to 9) in descending order of the value from the plurality of corrected difference integrated data, and assigns the corresponding coordinates to the fourth level feature point. (Step S5).
[0064]
When selecting n feature points, a priority area is arbitrarily set in a part of the entire image reduced to 1/64 (1/8 × 1/8), and (N <n) may be preferentially selected. For example, when seven (n = 7) feature points are to be extracted in total, if the setting is made to extract four feature points from the priority area, the value of the corrected difference integrated data is large in the priority area. Four (N = 4) feature points are selected in this order, and the remaining three feature points are selected in ascending order of the value of the corrected integrated difference data outside the priority area.
[0065]
Next, the feature point extracting unit 13 inputs the third-level luminance data Y1-3 generated from the first reference image YC1 by the image reducing unit 12 (Step S6). Then, on the input third-level luminance data Y1-3, a characteristic area composed of blocks of a predetermined size (for example, 64 pixels by 64 pixels) around the n characteristic points selected in step S5. Is cut out (step S7).
[0066]
Further, the feature point extracting unit 13 performs adjacent pixel difference integration for each cut-out feature region in blocks of a predetermined size (for example, 8 pixels vertically × 8 pixels horizontally) as in step S2 (step S8). . Next, the feature point extracting unit 13 applies an eight-directional Laplacian filter of a predetermined size (for example, 3 pixels vertically × 3 pixels horizontally: one difference integrated data corresponds to one pixel) to the image formed of the integrated difference data. Apply twice (steps S9 and S10).
[0067]
By these two Laplacian filter processes, corrected difference integrated data is obtained in each feature region by the number of 8 × 8 divided blocks. The feature point extracting unit 13 selects n pieces of the corrected difference integrated data in descending order of the value, and selects coordinates corresponding to the pieces as the third level feature points (step S11).
[0068]
Next, the feature point extracting unit 13 inputs the second-level luminance data Y1-2 generated from the first reference image YC1 by the image reducing unit 12 (Step S12). Then, on the input second-level luminance data Y1-2, a characteristic region consisting of a block of a predetermined size (for example, 64 pixels by 64 pixels) around the n characteristic points selected in step S11. Is cut out (step S13).
[0069]
Further, the feature point extracting unit 13 performs adjacent pixel difference integration for each cut-out feature region in blocks of a predetermined size (for example, 8 pixels vertically × 8 pixels horizontally) as in step S2 (step S14). . Next, the feature point extracting unit 13 applies an eight-directional Laplacian filter of a predetermined size (for example, 3 pixels vertically × 3 pixels horizontally: one difference integrated data corresponds to one pixel) to the image formed of the integrated difference data. Apply twice (steps S15 and S16).
[0070]
By these two Laplacian filter processes, corrected difference integrated data is obtained in each feature region by the number of 8 × 8 divided blocks. The feature point extracting unit 13 selects n pieces of the plurality of corrected difference integrated data in descending order of the value, and selects coordinates corresponding to the pieces as the second level feature points (step S17).
[0071]
Next, the feature point extracting unit 13 inputs the first-level luminance data Y1-1 generated from the first reference image YC1 by the image reducing unit 12 (Step S18). Then, on the input first-level luminance data Y1-1, a characteristic area composed of blocks of a predetermined size (for example, 64 pixels by 64 pixels) around the n characteristic points selected in step S17. Is cut out (step S19).
[0072]
Further, the feature point extracting unit 13 performs adjacent pixel difference integration for each of the cut-out feature regions in blocks of a predetermined size (for example, 8 pixels vertically × 8 pixels horizontally) as in step S2 (step S20). . Next, the feature point extracting unit 13 applies an eight-directional Laplacian filter of a predetermined size (for example, 3 pixels vertically × 3 pixels horizontally: one difference integrated data corresponds to one pixel) to the image formed of the integrated difference data. Apply twice (steps S21 and S22).
[0073]
By these two Laplacian filter processes, corrected difference integrated data is obtained in each feature region by the number of 8 × 8 divided blocks. The feature point extracting unit 13 selects n pieces of the plurality of corrected difference integrated data in descending order of value, and selects coordinates corresponding to the pieces as the first level feature points (step S23). As described above, n feature points are extracted for each of the fourth level to the first level.
[0074]
As described in detail above, according to the present embodiment, the input image is reduced stepwise in a plurality of steps, and the upper level is reduced using the image movement amount roughly obtained using the reduced image at the lower level. Since the scan block for calculating the image movement amount is constructed, the correlation with the image in the reference block can be obtained without making the scan block so large, and the range of the scan block can be narrowed. .
[0075]
If the range of the scanning block can be reduced, the probability that a plurality of the same or similar patterns exist in the scanning block decreases. Therefore, it is less likely that a portion in the reference block that appears to match the pattern is erroneously detected, and the matching accuracy can be improved. As a result, a more accurate camera shake amount can be detected, and camera shake correction can be performed more accurately.
[0076]
Further, if the range of the scanning block can be narrowed, the amount of calculation required for obtaining the correlation can be reduced. Further, in the present embodiment, a Laplacian filter is applied to an image to extract a feature point in a complex pattern where the same pattern does not exist around the complex pattern, so that the probability of occurrence of erroneous detection of the image movement amount is reduced. An optimal feature point can be extracted by a simple calculation. As a result, camera shake can be accurately detected with a small amount of calculation, and high responsiveness and high-precision camera shake correction can be achieved without using a high-cost CPU that has a large memory space for calculations and high-speed processing. can do.
[0077]
In addition, as for the reduced image in the lowermost layer, since the image movement amount cannot be obtained from the lower layer, the correlation operation is performed on the entire image. However, since the data amount is greatly reduced by the reduction, even if the entire image is scanned, the processing amount is not so large, and the calculation time does not become long. In the present embodiment, since the matching calculation using the histogram is performed instead of the normal block matching, the calculation load can be further reduced.
[0078]
In the present embodiment, instead of taking one image with long exposure to increase the relative sensitivity, the exposure time is shortened to continuously shoot a plurality of images, and the camera shake is corrected while correcting the camera shake between the images. Images are combined. Since the exposure time can be shortened, the amount of image blur due to camera shake per image can be reduced. At the same time, since the images are synthesized while correcting the amount of camera shake between the images, image blur due to camera shake can be almost completely eliminated in the combined image.
[0079]
Further, in the present embodiment, since a plurality of continuously shot images are averagely combined, many noise components are canceled out by averaging over time over a plurality of frames, and noise is reduced. This improves the S / N ratio of the composite image. Since the S / N ratio can be improved, noise becomes less noticeable even when the image is brightened by increasing the gain with PGA3. Therefore, even in a dark environment, a bright high-quality image with less noise can be obtained.
[0080]
In the first embodiment, an example in which the image reducing unit 12 reduces the input image in three steps has been described. However, the three steps are merely examples, and the present invention is not limited to this. Further, in the first embodiment, the example in which the Laplacian filter is applied twice when extracting the feature points has been described. However, the number of times is merely an example, and the present invention is not limited to this.
[0081]
Further, in the first embodiment, an example has been described in which matching is performed using a histogram when obtaining the global movement amount for the image at the lowest level. However, at the lowest level, the image data amount is significantly reduced by reduction. Therefore, the global movement amount may be obtained by ordinary block matching.
[0082]
In the first embodiment, an example in which two images are input and processed is described. However, a larger number of images may be input and processed. When n (n> 2) images are input, for example, using the first input image as a reference image, the amount of movement of the second image and the first image are calculated based on the correlation between the first and second images. From the correlation with the third image, the amount of image movement of the i-th (i = 2, 3,..., N) image is calculated by the above-described procedure, for example, the amount of image movement of the third image. .
[0083]
When synthesizing these n images, first, the second image is shifted in the reverse direction with respect to the first image by the moving amount of the second image. Next, the third image is combined with the combined image by shifting the third image in the reverse direction by the moving amount of the third image. In this way, the i-th (i = 2, 3,..., N) images are sequentially synthesized with the first image while being shifted by the respective image movement amounts.
[0084]
By doing so, it is possible to suppress image blur due to camera shake of the synthesized image and at the same time improve the S / N ratio. Therefore, even when shooting in a dark place where the subject brightness is considerably low, camera shake can be accurately detected and corrected, and noise should not be noticeable even if the image is brightened by increasing the gain. Can be. As a result, a bright high-quality image with less noise can be obtained even in a dark environment.
[0085]
When n (n> 2) images are input and processed, the reference image is not always set to only the first image. For example, the reference image may be set every fixed number of sheets. When a plurality of reference images are set, a plurality of images included between a certain reference image and the next reference image are regarded as one group, and the image movement amount is detected for each group in the above-described procedure to perform image synthesis. I do. At that time, feature points are extracted from the reference image in each group.
[0086]
When shooting a still image, the feature points may only move slightly due to camera shake.However, when shooting a moving image, even if there is no camera shake, the feature points will change according to the movement of the image. Also move. Therefore, the method of providing the reference image at regular intervals and extracting the feature points is particularly effective for camera shake correction at the time of capturing a moving image.
[0087]
In the first embodiment, a plurality of images are averagely combined, but the combining method is not limited to this. For example, a plurality of images may be added and synthesized.
[0088]
(Second embodiment)
Next, a second embodiment of the present invention will be described. FIG. 15 is a functional block diagram illustrating a configuration of a camera shake correction device according to the second embodiment. Note that in FIG. 15, components denoted by the same reference numerals as those illustrated in FIG. 8 have the same functions, and thus redundant description will be omitted here. The overall configuration of a digital camera to which the camera shake correction device according to the second embodiment is applied is the same as that in FIG.
[0089]
In FIG. 15, the image reduction unit 31 reduces the luminance data Y1-1 and Y2-1 to 1/8 times for each of the plurality of images YC1 and YC2 input by the image data input unit 11. Here also, the average reduction processing is preferably performed. That is, the adjacent 64 pixels of 8 × 8 pixels are defined as one block, and the average of the luminance data is calculated for each block to obtain luminance data Y1-4 and Y2-4 of the reduced image.
[0090]
The feature point extracting unit 32 extracts feature points that satisfy a predetermined condition with respect to the luminance data Y1-1 in the first reference image YC1 input by the image data input unit 11. The content of the extraction processing is almost the same as that of the first embodiment, but feature points for which the directionality determination unit 33 determines that the vertical, horizontal, and oblique directional correlations are strong are excluded from the extraction target. This is different from the feature point extracting unit 13 of the first embodiment.
[0091]
The direction determining unit 33 performs a filtering process (a process of applying a direction determining filter and a Laplacian filter), which will be described later, on the −４ reduced luminance data Y1-4 generated from the first input image YC1 by the image reducing unit 31. To obtain the directional values of the vertical, horizontal, diagonal left, and diagonal right (values obtained by applying the direction determination filter) and Laplacian values (values obtained by applying the Laplacian filter). The direction value is compared with the Laplacian value. If the direction value is larger, the corrected difference integrated data value obtained by the feature point extracting unit 32 in the corresponding block is set to zero. , So that feature points are not extracted from the block.
[0092]
The global movement amount detection unit 34 cross-correlates the two luminance data Y1-4 and Y2-4 with respect to the luminance data Y1-4 and Y2-4 of the ８ reduced image generated by the image reduction unit 31. To obtain the global movement amount. In the first embodiment, the global movement amount detection unit 14 obtains the global movement amount by histogram matching, whereas in the second embodiment, the global movement amount detection unit 34 performs global movement by ordinary block matching. Find the quantity.
[0093]
That is, the global movement amount detection unit 34 includes a reference block setting unit 35, a scanning block setting unit 36, and a block matching unit 37. The reference block setting unit 35 sets a reference block of a predetermined size (for example, a range of 80% in the vertical and horizontal directions of the reduced luminance data Y1-4) on the first reduced luminance data Y1-4.
[0094]
The scanning block setting unit 36 sets a scanning block of a predetermined size (for example, the entire reduced luminance data Y2-4) larger than the reference block on the second reduced luminance data Y2-4. The block matching unit 37 performs a global movement of the image by cross-correlating the luminance data of the reference block set on the first image and the luminance data of the scan block set on the second image. Find the quantity. That is, the difference is calculated while scanning the image of the reference block in the scan block, and a position where the difference value is minimum is found as the global movement amount.
[0095]
FIG. 16 is a flowchart showing the detailed processing contents of the feature point extracting unit 32 and the direction determining unit 33. In FIG. 16, first, the feature point extracting unit 32 inputs the first luminance data Y1-1 input by the image data input unit 11 (step S31), and obtains a predetermined size (for example, 8 vertical pixels × 8 horizontal pixels). The adjacent pixel difference integration is performed for each pixel (block) (step S32).
[0096]
Next, the feature point extraction unit 32 applies an 8-directional Laplacian filter of a predetermined size (for example, 3 pixels vertically × 3 pixels: one difference integrated data corresponds to one pixel) to the image composed of the difference integrated data. Apply twice (steps S33 and S34). As a result, the corrected difference integrated data is obtained by the number of divided blocks of 8 × 8 units.
[0097]
On the other hand, the direction determining unit 33 inputs the reduced luminance data Y1-4 generated from the first input image YC1 by the image reducing unit 31 (Step S41). Then, the input reduced luminance data Y1-4 is subjected to a vertical, horizontal, leftward and rightward diagonal direction determination filter of a predetermined size (for example, 3 vertical pixels × 3 horizontal pixels), and a predetermined size (for example, An eight-direction Laplacian filter (3 pixels vertically × 3 pixels horizontally) is applied. Then, for each pixel, the largest value among the four directional values of vertical, horizontal, leftward and rightward and one Laplacian value is selected and set as the value of each pixel (step S42). FIG. 17 shows a vertical, horizontal, diagonally left, and diagonally right direction determination filter used in the direction determination unit 33.
[0098]
Next, the direction determining unit 33 performs the same process as step S42 once again (step S43). As a result, it is determined for each pixel whether any one of the vertical, horizontal, diagonally left, and diagonally right direction values obtained by applying the direction determination filter is larger than the Laplacian value obtained by applying the Laplacian filter. A determination is made (step S44). If the direction value is larger than the Laplacian value, the corrected difference integrated data of the pixel corresponding to the position at which the large direction value is obtained is included in the corrected difference integrated data of each pixel obtained in step S34. The value is set to zero (step S45).
[0099]
The feature point extracting unit 32 sorts the corrected difference integrated data of each pixel including the corrected difference integrated data set to the zero value as needed by the direction determining unit 33 in this order from the largest value. The n points are selected, and the coordinates corresponding to them are selected as feature points (step S35).
[0100]
As described above in detail, according to the second embodiment, a pattern having strong correlation vertically, horizontally, and diagonally is detected, and a feature point is not extracted from such a position. In doing so, it can be ensured that the same pattern does not exist in the surroundings. As a result, it is possible to detect the optimum feature point used for reducing the probability of occurrence of erroneous detection of the image movement amount by a simple calculation. Therefore, camera shake can be accurately detected with a small amount of calculation, and high responsiveness and highly accurate camera shake correction can be realized.
[0101]
In the second embodiment, an example in which the image reducing unit 31 reduces the input image by one step has been described. However, the input image may be reduced in a plurality of steps as in the first embodiment. Further, in the second embodiment, the example in which the direction determination filter and the Laplacian filter are applied twice has been described. However, the number of times is merely an example, and the present invention is not limited to this.
[0102]
Further, in the second embodiment, an example in which block matching is performed when obtaining the global movement amount has been described. However, matching using a histogram may be performed in the same manner as in the first embodiment. In the second embodiment, an example in which two images are input and processed is described. However, a larger number of images may be input and processed. In this case, it is possible to set the reference image other than the first image.
[0103]
Further, the above-described camera shake correction method according to the first and second embodiments can be realized by any of a hardware configuration, a DSP, and software. For example, the image stabilization apparatus according to the present embodiment can be implemented as software for real-time processing that can be embedded and mounted in a digital still camera or digital video camera.
[0104]
In the case of using a hardware configuration, a DSP, or the like, it is possible to mount the functional configuration of the image stabilization apparatus described in the above embodiment on a semiconductor chip, a board module, or the like. It is not always necessary to mount the functional configuration of the entire image stabilizing apparatus on a single semiconductor chip or a substrate module, but may be mounted on a plurality of chips or a plurality of substrates. For example, it is possible to mount other functional components except for the image synthesizing unit 16 on one chip as a camera shake detection device.
[0105]
In addition, each of the above-described embodiments is merely an example of a specific embodiment for carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. That is, the present invention can be embodied in various forms without departing from the spirit or main features thereof.
[0106]
【The invention's effect】
As described above, according to the present invention, a camera shake detection method of detecting a motion vector from image data instead of using a gyro sensor is employed, and an electronic camera shake correction method is employed instead of an optical method. In addition, it is possible to realize a small and low-cost image stabilization apparatus. Moreover, since the camera shake is detected and corrected by an algorithm unique to the present invention, highly accurate camera shake correction can be realized with a small amount of calculation, and accurate and responsive camera shake correction can be performed. In addition, even when photographing in a dark place where the brightness of the subject is small, camera shake can be accurately detected and corrected, and a bright and high-quality image can be obtained.
[Brief description of the drawings]
FIG. 1 is a functional block diagram illustrating a schematic configuration of a camera shake correction device according to a first embodiment.
FIG. 2 is a block diagram illustrating an entire configuration of a digital camera to which the camera shake correction device according to the first and second embodiments is applied.
FIG. 3 is a diagram for explaining the contents of an average reduction process.
FIG. 4 is a diagram for describing extraction of a feature point;
FIG. 5 is a diagram for explaining the meaning of a global movement amount.
FIG. 6 is a diagram for explaining the meaning of a local movement amount.
FIG. 7 is a diagram illustrating a use relationship between a global movement amount and a local movement amount.
FIG. 8 is a functional block diagram illustrating a detailed configuration of a camera shake correction device according to the first embodiment.
FIG. 9 is a diagram for explaining a generation process of a reference histogram and a scanning histogram.
FIG. 10 is a diagram for explaining the contents of a histogram matching process;
FIG. 11 is a flowchart illustrating the content of a feature point extraction process according to the first embodiment.
FIG. 12 is a diagram for explaining the meaning of adjacent pixel difference integration.
FIG. 13 is a diagram illustrating a configuration example of an eight-direction Laplacian filter.
FIG. 14 is a diagram for explaining the significance of an eight-direction Laplacian filter.
FIG. 15 is a functional block diagram illustrating a detailed configuration of a camera shake correction device according to a second embodiment.
FIG. 16 is a flowchart illustrating details of a feature point extraction process and a direction determination process according to the second embodiment.
FIG. 17 is a diagram illustrating a configuration example of a direction determination filter.
[Explanation of symbols]
1 Optical system
1a Shutter
1b lens
1c Iris
2 Image sensor
3 PGA (Programmable Gain Amplifier)
4 A / D converter
5 Signal processing unit
6 Camera shake correction unit
7 AE / AF processing unit
8 Controller
10 Digital camera
11 Image data input section
12 Image reduction unit
13 Feature point extraction unit
14 Global movement detector
15 Local movement amount detection unit
16 Image synthesis unit
21 Reference histogram generator
22 Scanning histogram generator
23 Histogram matching unit
24 Reference block setting section
25 Scan block setting unit
26 Block matching section
31 Image reduction unit
32 feature point extraction unit
33 Direction determination unit
34 Global movement amount detector
35 Reference block setting section
36 Scan block setting unit
37 Block matching section

Claims (17)

For each of a plurality of image data generated and imaged within a certain period of time by the imaging means, the image is reduced by one or more stages to generate a plurality of levels of image data including the original image. Image reduction means;
For each of a plurality of levels of image data including at least one reference image data among the plurality of image data generated by the imaging means and the reference reduced image data generated by the image reduction means from the image data, Feature point extracting means for extracting feature points satisfying a predetermined condition from
In order from the lowest level to the highest level of the multi-level image data generated by the image reducing means, using the feature points extracted by the feature point extracting means, a reference image and other images are used. Image movement amount detection means for obtaining an image movement amount by taking a cross-correlation of
Image synthesizing means for synthesizing the reference image data and the other image data at the uppermost level generated by the imaging by the imaging means by shifting by the amount of image movement obtained by the image movement amount detecting means. A camera shake correction device comprising: The image movement amount detecting means performs cross-correlation between the reference reduced image data and other reduced image data with respect to the reduced image at the lowest level generated by the image reducing means, thereby obtaining the image at the lowest level. First image movement amount detection means for obtaining an image movement amount;
For each level image other than the lowest level, a reference block of a predetermined size is set around the feature points extracted by the feature point extraction means on the reference image data and the reference reduced image data, and On the image data, a scan block of a predetermined size larger than the reference block is set around a position shifted from the position corresponding to the feature point by the amount of image movement obtained at the lower level, and the image data of the reference block is set. And the image data of the scanning block are cross-correlated to determine an image movement amount at each level, and a second image movement to output the image movement amount obtained at the uppermost level as a final image movement amount The camera shake correction apparatus according to claim 1, further comprising an amount detection unit. The feature point extracting means extracts a plurality of feature points from one image data for each of the plurality of levels of image data,
The second image movement amount detection means obtains a local image movement amount for each of a plurality of feature points extracted by the feature point extraction means for an image at each level other than the lowest level. 3. The image stabilization apparatus according to claim 2, wherein one local movement amount selected from the above is set as the image movement amount at that level. The feature point extracting means extracts a plurality of feature points from one image data for each of the plurality of levels of image data,
The second image movement amount detection means obtains a local image movement amount for each of a plurality of feature points extracted by the feature point extraction means for an image at each level other than the lowest level. The one local movement amount calculated by performing a predetermined operation using all or a part of the plurality of obtained image movement amounts is set as the image movement amount at that level. Camera shake correction device. The feature point extracting means performs, for each of the plurality of levels of image data, an adjacent pixel difference integration operation on the one image data in divided block units of a predetermined size. , A filter of a predetermined size in which coefficients are set so as to apply a large weight to the median value and a small weight to the surrounding value is subjected to scanning processing at least once, and the corresponding blocks are sequentially processed in descending order of the data value. The camera shake correction apparatus according to claim 3, wherein coordinates to be extracted are extracted as a plurality of feature points. By performing cross-correlation between at least one reference image data and at least one other image data among a plurality of image data generated and imaged within a predetermined time by an imaging unit, each of the at least one image data is obtained. Image moving amount detecting means for calculating the image moving amount with respect to the reference image data,
Image synthesizing means for synthesizing the reference image data and the other one or more pieces of image data by shifting the image data by the image moving amount obtained by the image moving amount detecting means. Camera shake correction device. A feature point extraction unit that extracts a feature point that satisfies a predetermined condition from at least one reference image data among a plurality of image data generated by the imaging unit;
The image movement amount detection means sets a reference block of a predetermined size around the feature point extracted by the feature point extraction means on the reference image data, and sets the feature block on the one or more other image data. Setting a scan block of a predetermined size larger than the reference block around a position corresponding to a point, and obtaining the image movement amount by cross-correlating the image data of the reference block and the image data of the scan block; The camera shake correction apparatus according to claim 6, wherein: By inputting a plurality of image data generated by imaging within a certain time by the imaging means, and reducing the image of each of the input image data in one or more stages, including the original image, An image reduction step of generating multi-level image data by
At least one of the plurality of input image data is set as reference image data, and each of a plurality of levels of image data including the reference reduced image data generated by reducing the reference image data and the reference image data is referred to as an image. A feature point extracting step of extracting a feature point satisfying a predetermined condition from among them;
The cross-correlation between the reference image data and other image data and the cross-correlation between the reference reduced image data and other reduced image data in order from the lowest level to the highest level of the plurality of levels of image data. By calculating the image movement amount at each level by taking the above, when obtaining the image movement amount at a level other than the lowest level, the feature points by the amount of the image movement calculated at the lower level than itself A scanning block of a predetermined size to be scanned when cross-correlation is set around the position shifted from the corresponding position to obtain the image moving amount, and the image moving amount obtained at the uppermost layer level is finally calculated. Image movement amount detection step of outputting as a large image movement amount;
An image synthesizing step of synthesizing the input reference image data and the other image data at the uppermost layer level by shifting by the final image movement amount. In the feature point extracting step, a plurality of feature points are extracted from one image data for each of the plurality of levels of image data,
In the image moving amount detecting step, a local image moving amount is obtained for each of the plurality of feature points for an image at each level other than the lowest level, and one local moving amount selected from the image moving amounts is determined. 9. The method according to claim 8, wherein the image movement amount is a level. In the feature point extracting step, a plurality of feature points are extracted from one image data for each of the plurality of levels of image data,
In the image moving amount detecting step, a local image moving amount is obtained for each of the plurality of feature points for an image at each level other than the lowest level, and all or one of the obtained image moving amounts is obtained. 9. The camera shake correction method according to claim 8, wherein one local movement amount calculated by performing a predetermined operation using the unit is used as an image movement amount at that level. In the feature point extracting step, for each of the plurality of levels of image data, adjacent pixel difference integration calculation is performed on the one image data in units of divided blocks of a predetermined size. , A filter of a predetermined size in which coefficients are set so as to apply a large weight to the median value and a small weight to the surrounding value is subjected to scanning processing at least once, and the corresponding blocks are sequentially processed in descending order of the data value. 11. The method according to claim 9, wherein coordinates to be extracted are extracted as a plurality of feature points. By performing cross-correlation between at least one reference image data and at least one other image data among a plurality of image data generated and imaged within a predetermined time by an imaging unit, each of the at least one image data is obtained. An image moving amount detecting step of obtaining an image moving amount with respect to the reference image data;
An image synthesizing step of synthesizing the reference image data and the other one or more pieces of image data by shifting the image data by the image moving amount. For each of a plurality of image data generated and imaged within a certain period of time by the imaging means, the image is reduced by one or more stages to generate a plurality of levels of image data including the original image. Image reduction means;
For each of a plurality of levels of image data including at least one reference image data among the plurality of image data generated by the imaging means and the reference reduced image data generated by the image reduction means from the image data, Feature point extracting means for extracting feature points satisfying a predetermined condition from
In order from the lowest level to the highest level of the multi-level image data generated by the image reducing means, using the feature points extracted by the feature point extracting means, a reference image and other images are used. A camera shake detection device comprising: an image movement amount detecting unit that obtains an image movement amount by calculating a cross-correlation of the image movement amount. The image movement amount detecting means performs cross-correlation between the reference reduced image data and other reduced image data with respect to the reduced image at the lowest level generated by the image reducing means, thereby obtaining the image at the lowest level. First image movement amount detection means for obtaining an image movement amount;
For each level image other than the lowest level, a reference block of a predetermined size is set around the feature points extracted by the feature point extraction means on the reference image data and the reference reduced image data, and On the image data, a scan block of a predetermined size larger than the reference block is set around a position shifted from the position corresponding to the feature point by the amount of image movement obtained at the lower level, and the image data of the reference block is set. And the image data of the scanning block are cross-correlated to determine an image movement amount at each level, and a second image movement to output the image movement amount obtained at the uppermost level as a final image movement amount 14. The camera shake detection device according to claim 13, further comprising an amount detection unit. The feature point extracting means extracts a plurality of feature points from one image data for each of the plurality of levels of image data,
The second image movement amount detection means obtains a local image movement amount for each of a plurality of feature points extracted by the feature point extraction means for an image at each level other than the lowest level. 15. The camera shake detection apparatus according to claim 14, wherein one local movement amount selected from the above is set as the image movement amount at that level. The feature point extracting means extracts a plurality of feature points from one image data for each of the plurality of levels of image data,
The second image movement amount detection means obtains a local image movement amount for each of a plurality of feature points extracted by the feature point extraction means for an image at each level other than the lowest level. The one local movement amount calculated by performing a predetermined operation using all or a part of the plurality of obtained image movement amounts is set as the image movement amount at that level. Camera shake detection device. The feature point extracting means performs, for each of the plurality of levels of image data, an adjacent pixel difference integration operation on the one image data in divided block units of a predetermined size. , A filter of a predetermined size in which coefficients are set so as to apply a large weight to the median value and a small weight to the surrounding value is subjected to scanning processing at least once, and the corresponding blocks are sequentially processed in descending order of the data value. 17. The camera shake detection device according to claim 15, wherein coordinates to be extracted are extracted as a plurality of feature points.

Images