JP3252415B2 - Image stabilization device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image stabilizing apparatus for correcting the image pickup output of a handy type video camera by detecting the amount of movement of an image caused by a so-called camera shake contained in video data.

[0002]

2. Description of the Related Art Generally, in a handy type video camera, camera shake during photographing, that is, camera vibration appears as image vibration. Therefore, as an image stabilization device for correcting an image vibration caused by such a camera shake, for example, as disclosed in Japanese Patent Application Laid-Open No. 63-166370, a motion vector of an image is detected and based on this motion vector. A technique for correcting video data stored in an image memory has been proposed.

For detecting a motion vector of an image, for example, a block matching method is employed. In the detection of a motion vector of an image by the block matching method, a screen is divided into a number of regions (referred to as blocks), and a representative point pixel of a previous field located at the center of each block and a pixel of each pixel in a block of the current field are determined. The absolute value of the field difference from the image data is calculated, the absolute value of the field difference of each block is integrated for each corresponding pixel to obtain a correlation integral value, and the correlation integration having coordinates corresponding to the pixel array of one block Form a value table. Then, the motion vector of the entire screen is determined using the coordinate value of the minimum value of the correlation integral value in the correlation integrated value table as the coordinate value of the motion vector of the image.

[0004] The image blur correction device converts the detected motion vector into a correction signal, and performs a correction for moving the current image using the correction signal. The correction accuracy of such an image stabilization device depends on the detection accuracy of the motion vector of the image.

[0005]

By the way, as described above, the motion vector of the entire screen is determined by using the coordinate value of the minimum value of the correlation integrated value in the correlation integrated value table as the coordinate value of the motion vector of the image. In the detection of a motion vector of an image by the block matching method, a motion vector due to camera shake and a motion vector due to movement of a subject are simultaneously generated. Therefore, the image shake correction apparatus needs to determine the motion vector due to the camera shake and the motion vector due to the movement of the subject, and form a camera shake correction signal using only the motion vector due to the camera shake to perform the camera shake correction. If the motion vector due to the movement of the subject is erroneously determined to be the motion vector due to camera shake and the camera shake correction is performed, the background image that should be stationary moves due to the camera shake correction, which is unnatural. Becomes an image. Conversely, if the motion vector due to the camera shake is erroneously determined to be the motion vector due to the movement of the subject, the camera shake cannot be corrected.

The present invention has been made in view of such circumstances, and has as its object to enable high-performance camera shake correction in a handy-type video camera or the like.
It is an object of the present invention to provide an image stabilization apparatus for an image, in which it is reliably determined whether a detected motion vector of an image is a motion vector of an image due to camera shake or a motion vector of a movement of a subject, thereby performing camera shake correction. .

[0007]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, an image stabilizing apparatus for an image according to the present invention is a representative image for each block obtained by dividing a fixed unit image formed by an input video signal into a plurality of blocks. The image data I _k (0,0) of the point pixel is stored in the memory for a fixed unit period, and the image data I _k (x, y) of each pixel of the block of the current field and the representative of the block of the previous field read from the memory are stored. Absolute value | I of the difference from the image data I _k-1 (0,0) of the point pixel
a motion vector detecting unit for detecting a motion vector of an image based on _k-1 (0,0) -I _k (x, y) | Activity determining means for determining the activity of each image data in the vicinity of the representative point pixel of each block; and image data I of the representative point pixel for each block obtained by dividing the fixed unit image composed of the input video signal into a plurality of blocks.
_k (0, 0) is stored in the memory for a certain unit period, and for a block determined to be significant by the activity determining means, the image data I _k (x, y) of each pixel of the block in the current field is read out from the memory. Image data I _k-1 (0,
0)) and the absolute value | I _k-1 (0,0) −I _k (x,
y) | satisfies the first determination condition of I _k−1 (0,0) ≒ I _k (0,0), and the threshold of the determination is TH
_{As I} , every time a pixel that satisfies the second determination condition of | I _k-1 (0,0) −I _k (x, y) | ≧ TH _I is detected, A frequency distribution table is formed by incrementing the frequency f (x, y), and based on the frequency distribution table, when the frequency value of the coordinates specified by the motion vector detected by the motion vector detection unit is smaller than a predetermined value In addition, a camera shake determination unit that determines that the motion vector is caused by camera shake of the image, and a correction amount corresponding to the motion vector determined by the camera shake determination unit to be caused by camera shake of the image The image processing apparatus further includes a correction amount generation unit that forms a camera shake correction signal, and a correction unit that performs a camera shake correction process on an input video signal based on the camera shake correction signal supplied from the correction amount generation unit.

[0008]

In the image stabilization apparatus for an image according to the present invention, a motion vector of an image is obtained by block matching using image data of a representative point pixel for each block obtained by dividing a fixed unit image formed of an input video signal into a plurality of blocks. Is detected by the motion vector detection unit, the activity of each image data in the vicinity of the representative point pixel of each block is determined by the activity determination unit, and the image of a certain unit constituted by the input video signal is divided into a plurality of pieces. Based on the inter-field correlation of the image data of the representative point pixels for each block and the determination output by the activity determination unit, the camera shake determination unit determines whether the motion vector is due to camera shake, and determines whether the motion vector is due to camera shake. Forming a camera shake correction signal of a correction amount corresponding to a motion vector to be performed by a correction amount generation unit, Subjected to camera shake correction process on the input video signal by Tadashibu.

[0009]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram of an image stabilizing apparatus according to an embodiment of the present invention; An image stabilization apparatus for an image according to the present invention is configured, for example, as shown in FIG.

The image blur correction device shown in FIG. 1 is a device in which the present invention is applied to a camera shake correction device for correcting the motion of an image due to a camera shake in a handy type video camera. Activity determination unit 20, camera shake determination unit 30, correction amount generation unit 4
0 and a correction unit 50. In FIG. 1, input video data obtained by digitizing a video signal obtained as an imaging output by an imaging unit (not shown) of the video camera is supplied to a signal input terminal 1.

In this camera shake correction apparatus, the motion vector detecting section 10 includes a representative point memory 11 and a subtraction circuit 12 to which the input video data is supplied via the signal input terminal 1, and a subtraction output by the subtraction circuit 12. The circuit comprises a correlation integrated value table forming circuit 13 to which data is supplied, and a motion vector detecting circuit 14 to which a correlation integrated value of the correlation integrated value table formed by the correlation integrated value table forming circuit 13 is supplied.

The representative point memory 11 in the motion vector detecting unit 10 is composed of input video data.
The image data I _k (0,0) of the representative point pixel for each block obtained by dividing the image of the field into a plurality is stored. Specifically, for example, as shown in FIG. 2, a screen of one field is divided into blocks of m pixels × n lines, and as shown in FIG. 3, a pixel S (0,0) at the center of each block is set as a representative point. age,
The image data I _k (0,0) of each representative point pixel is stored in the representative point memory 11 for one field period. Note that the representative points are uniformly distributed on the screen. Then, the image data I _k−1 (0,0) of each representative point pixel one field before read from the representative point memory 11 is supplied to the subtraction circuit 12.

The subtraction circuit 12 is connected to the signal input terminal 1
, The image data of the current field, i.e., the image data I _k (x, y) of each pixel of m × n and the representative point memory 1
1 and the difference from the image data I _k-1 (0,0) of the representative point pixel of the corresponding block of the previous field, that is, the absolute value of the difference between the fields | I _k-1 (0,0) -I _k
(X, y) |. Then, the field difference absolute value obtained as the subtraction output data by the subtraction circuit 12 is supplied to the correlation integrated value table forming circuit 13.

The correlation integrated value table forming circuit 13 corresponds to the field difference absolute value | I _k-1 (0,0) -I _k (x, y) | of each block obtained by the subtraction circuit 12. The pixels are integrated over one field period to form a correlation integrated value table having m × n integer coordinates corresponding to the pixel arrangement of one block. This correlation integrated value table forming circuit 13
The correlation integrated value table formed by the following formula indicates the integrated value of m × n field difference absolute values | I _k-1 (0,0) −I _k (x, y) | The correlation integrated value of the coordinates having the strongest field correlation becomes the minimum value. Then, m × n correlation integrated values in the correlation integrated value table formed by the correlation integrated value table forming circuit 13 are supplied to the motion vector detecting circuit 14.

The motion vector detection circuit 14 detects the coordinates of the minimum value of the correlation integrated value in the correlation integrated value table formed by the correlation integrated value table forming circuit 13. The correlation integrated value of the correlation integrated value table formed by the correlation integrated value table forming circuit 13 is the image data I _k−1 of the representative point pixel of each block.
This indicates the inter-field correlation between (0, 0) and the image data I _k (x, y) of another pixel. The coordinates corresponding to a pixel having a higher correlation have a smaller value, and the correlation of the coordinates corresponding to a motion vector is smaller. The integrated value becomes the minimum value. That is, the motion vector detection circuit 14 detects the coordinate of the minimum value of the correlation integrated value in the correlation integrated value table as the coordinate indicating the motion vector.

The coordinates of the minimum value of the correlation integrated value, that is, the motion vector detected by the motion vector detection circuit 14 are supplied to the camera shake determination section 30 and the correction amount generation section 40.

Further, in the camera shake correction apparatus, the activity determining section 20 includes an activity determining circuit 21 to which the input video data is supplied via the signal input terminal 1 and a result determined by the activity determining circuit 21 as one. An activity memory 22 stores the field period.

The activity determining circuit 21 generates image data I _k (0,0) of a representative point pixel for each block.
For example, image data I _k (+1,
0), I _k (−1, 0), I _k (0, +1), I
Laplacian operator V ² ｛V ² = _k (0, -1)
_{I k (+1,0) + I k} (-1,0), I k (0, +
_{1) + I k (0,} -1) measured activity using -4I _k (0,0)}, it determines the magnitude of the activity that value due thresholding. As a result, a flat image having a small activity locally and an image such as a sharp edge of a pattern having a large activity locally are distinguished. Then, the activity determination circuit 2
The determination result of 1 is stored in the activity memory 22 for one field period.

The activity determination circuit 21 may determine the activity in the vicinity of the representative point pixel for each block by using image data of, for example, eight neighboring pixels in addition to the four neighboring pixels.

Then, the activity determining section 20
Is a determination output A _k (0,0) indicating the activity of each representative point pixel of the block of the current field obtained by the activity determination circuit 21 and each of the blocks of the previous field obtained via the activity memory 22 Judgment output A _k-1 indicating the activity of the representative point pixel
(0,0) is supplied to the camera shake determination unit 30.

Further, in the camera shake correction apparatus, the camera shake determination section 30 includes a representative point memory 31 and a subtraction circuit 32 to which the input video data is supplied via the signal input terminal 1, and a subtraction by the subtraction circuit 32. Output and each judgment output A from the activity judgment unit 20
_k (0, 0), A _k-1 (0, 0) is supplied to the condition determining circuit 33, and a frequency distribution table forming circuit 34 that forms a frequency distribution table based on the determination output by the condition determining circuit 33.
And a process of determining whether or not the motion vector of the image detected by the motion vector detection unit 10 is a motion vector of an image due to camera shake is performed based on the frequency distribution table formed by the frequency distribution table forming unit 34. Camera shake determination circuit 3
5 is provided.

In the camera shake judging section 30, the representative point memory 31 stores image data I _k (0,0) of representative point pixels for each block obtained by dividing an image of one field formed by the input video signal into a plurality of blocks. ) Is stored for one field period. Each of the blocks corresponds to each of the blocks obtained by dividing the image of one field into a plurality in the camera shake determination unit 20. Then, the representative point memory 31
Image data I _k-1 of each representative point pixel one field before
(0, 0) is read out and supplied to the subtraction circuit 32.

The subtraction circuit 32 is connected to the signal input terminal 1
, The image data of the current field, the image data I _k (x, y) of each pixel of m × n pixels and the representatives of the previous field read from the representative point memory 31. The absolute value | I _k-1 (0,0) of the difference from the image data I _k-1 (0,0) of the point pixel
0) -I _k (x, y) |, that is, the absolute value of the difference between the fields is detected. Then, the field difference absolute value obtained as the subtraction output data by the subtraction circuit 32 is supplied to the condition determination circuit 33.

The condition determining circuit 33 determines each of the determination outputs A _k (0,0), A from the activity determining section 20.
For at least one of greater than the threshold value T _A block of _k-1 (0,0), the field difference absolute value of each block detected by the subtractor circuit _{32 | I k-1 (0} ,
0) −I _k (0,0) | satisfies the first determination condition of I _k−1 (0,0) ≒ I _k (0,0), and | I _k−1 (0,0) It is determined whether or not a second determination condition of −I _k (x, y) | ≧ Th _{I is} satisfied. Here, the above Th _I is a threshold value for determining whether there is a level difference in the space.

That is, the condition determination circuit 33 detects a stationary block candidate having a representative point pixel having a strong field correlation according to the first determination condition, and determines the representative point pixel according to the second determination condition for the stationary block candidate. image data I _k of each pixel with respect to (0,0) I _k (x in,
Stillness determination based on the correlation of y) is performed.

In general, an image has a strong local correlation, and the properties near the representative point pixel are similar between the blocks. Therefore, in a block having low activity in a flat image,
Field difference absolute value of each block | I _k-1 (0,0)
−I _k (0,0) | satisfies the first determination condition. However, a block in which at least one of the determination outputs A _k (0,0) and A _k−1 (0,0) from the activity determination unit 20 is larger than the threshold T _A , that is, a block with a high activity, is actually moved. In the case of a block corresponding to the subject, the first determination condition is hardly satisfied. Therefore, it is possible to accurately detect a still block candidate based on the first determination condition. In the condition determination circuit 33, the activity determination unit 20
Performs _{A k-1 (0,0) ≧} T A becomes condition determination for determining the output A _k-1 only (0,0) indicating the activity of the representative point pixels of one field before the block obtained by this condition Even if a still block candidate is detected based on the first determination condition for a block that satisfies the above, the risk of erroneous detection of a still block candidate can be greatly reduced as compared with a case where the activity determination is not performed. . Condition determination circuit 33 of this embodiment
Then, the condition determination that A _k (0,0) ≧ T _A is performed for the determination output A _k (0,0) indicating the activity of each representative point pixel of the block of the current field obtained by the activity determination unit 20. Thus, the reliability of the detection of the still block candidate based on the first determination condition is improved.

The judgment output from the condition judgment circuit 33 is supplied to the frequency distribution table forming circuit 34.

The frequency distribution table forming circuit 34 determines the first and second determination conditions based on the memory table 34 A having coordinates corresponding to the pixel arrangement of one block and the determination output by the condition determination circuit 33. And an adder 34B that increments the frequency f (x, y) of the corresponding coordinate every time a pixel satisfying the following condition is detected. In the frequency distribution table forming circuit 34, the image data I
_k (x, y) and the absolute value | I _k-1 (0,0) of the difference between the image data I _k-1 (0,0) of each representative point pixel one field before.
0) -I _k (x, y) |, a conditional frequency distribution table showing the correlation between the one field of the representative point pixels is formed in the memory table 34A. Then, the frequency value of the frequency distribution table indicating the correlation between one field formed in the memory table 34A by the frequency distribution table forming circuit 34 is supplied to the camera shake determination circuit 35.

The camera shake determination circuit 35 calculates the frequency distribution table formed by the frequency distribution table forming circuit 34,
If the frequency value of the coordinates corresponding to the motion vector of the image detected by the motion vector detection unit 10 is smaller than a predetermined value, the motion vector of the image detected by the motion vector detection unit 10 is used as the image shake. If the frequency value is larger than a predetermined value, it is determined that the motion vector of the image detected by the motion vector detection unit 10 is not due to camera shake of the image.

Here, for example, as shown in FIG.
When the flat subject OB independently moves by v ₁ in the image with the motion vector V _k due to camera shake between the field and the k field, the image data I _k-1 (0,0) of the representative point pixel of the block A , I _k (0,0) have a high field correlation, and the image data I _k (x, y) at the coordinates of the motion vector V _k and the image data I _k-1 (0,0) of the representative point pixel in the previous field. If the activity determination as described above is not performed, the condition determination circuit 33 determines the pixel as a significant pixel that satisfies the first determination condition and the second determination condition, and the frequency distribution table forming circuit 34 The frequency f (x, y) of the coordinates in the memory table 34A is incremented, and the camera shake determination circuit 35 may make an erroneous determination. On the other hand, in the camera shake correction apparatus of this embodiment, since the activity of each image data in the vicinity of the representative point pixel for each block is determined by the activity determination unit 20, one field composed of the input video signal is used. It is determined whether or not the motion vector is caused by a camera shake based on the inter-field correlation of the image data of the representative point pixels for each block obtained by dividing the image into a plurality of blocks and the determination output by the activity determination unit 20. The determination can be reliably performed by the unit 30.

Then, the judgment output of the camera shake judging section 30 is supplied to the correction amount generating section 40.

When the determination output indicating that the motion vector detected by the motion vector detector 10 is caused by the camera shake of the image is supplied from the camera shake determiner 30, the correction amount generator 40 receives Motion vector detector 1
The motion vector detected by the motion vector 0 is represented by a shake vector V _t '.
As to form the image stabilization signal _{X t = X t-1 -V} t ' becomes the correction amount X _t, and supplies the image stabilization signal to the correcting unit 50. When the determination output indicating that the motion vector detected by the motion vector detection unit 10 is not caused by the camera shake of the image is supplied from the camera shake determination unit 30, the correction amount generation unit 40 sets the camera shake vector V _t 'is replaced by the zero vector V [0,0]
Is formed, and the camera shake correction signal is supplied to the correction unit 50.

As shown in FIG. 5, for example, the correction section 50 includes an address control circuit 51 and a select signal generation circuit 52 to which a camera shake correction signal is supplied from the correction amount generation section 40, and an address control circuit 51. Memory 53 and peripheral memory 54 in which writing / reading of video data is performed in accordance with an address signal supplied from
And a selector 55 for selectively outputting video data read from the field memory 53 and the peripheral memory 54 in accordance with a select signal supplied from the select signal generating circuit 52.

The input video data supplied via the signal input terminal 1 is sequentially written into the field memory 53. The read address of the field memory 53 is controlled by the camera shake correction signal in accordance with the camera shake vector. As a result, video data obtained by moving one field of input video data according to the camera shake vector is obtained from the field memory 53. Then, the video data read from the field memory 53 and the peripheral video data read from the peripheral memory 54 are combined by selection by the selector 55, and are output from the signal output terminal 2 as video data subjected to camera shake correction processing.

In the peripheral memory 54, video data of a peripheral portion corresponding to a correction range of an image based on video data subjected to camera shake correction output via the selector 55 is sequentially written as peripheral video data.

In this camera shake correction apparatus, the camera shake determination section 30 reliably determines whether or not the motion vector of the image detected by the motion vector detection section 10 is attributable to camera shake as described above. Since the correction can be performed, camera shake correction can be reliably performed, and a natural image output can be obtained.

[0037]

As is apparent from the above description, in the image stabilization apparatus for an image according to the present invention, a representative point pixel of each block obtained by dividing a fixed unit image formed by an input video signal into a plurality of blocks is divided. A motion vector of an image is detected by a motion vector detecting unit by block matching using image data, and the activity of each image data in the vicinity of a representative point pixel for each of the blocks is determined by an activity determining unit. Whether or not the motion vector is caused by camera shake based on the inter-field correlation of the image data of the representative point pixels of each block obtained by dividing the image of the fixed unit into a plurality of blocks and the determination output by the activity determining means Can be reliably determined by the camera shake determination unit, and the motion vector corresponding to the motion vector caused by the camera shake can be determined. The shake correction signal of the correction amount is formed by the correction amount generating section, it is possible to reliably apply the camera shake correction process on the input video signal by the correction unit.

As described above, in the image stabilizing apparatus for an image according to the present invention, it is possible to reliably determine whether or not the motion vector of an image is caused by a camera shake, and to perform a camera shake correction. High-performance camera shake correction in a video camera or the like can be performed.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a camera shake correction apparatus according to the present invention.

FIG. 2 is a diagram illustrating a state in which a screen is divided into blocks in a motion vector detection unit of the camera shake correction apparatus.

FIG. 3 is a diagram showing a structure of one block of the screen divided into blocks.

FIG. 4 is a diagram schematically showing a state of generation of a motion vector due to a motion of a subject.

FIG. 5 is a block diagram illustrating a configuration of a correction unit of the camera shake correction device.

[Explanation of symbols]

 Reference numeral 10: Motion vector detecting section 11: Representative point memory 12: Subtraction circuit 13: Correlation Integrated value table forming circuit 14 Motion vector detecting circuit 20 Activity determining unit 21 Activity determining circuit 22 Activity memory 30 Camera shake determination unit 31 Representative point memory 32 Subtraction circuit 33 ······ Condition determination circuit ········································································· • 50: Correction unit

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Ken Satoshi 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Tsukasa Hashino 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo (56) References JP-A-4-255178 (JP, A) JP-A-4-302590 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) H04N 5/232 G03B 5/00 G06T 7/20 100

Claims (1)

(57) [Claims] An image data I _k (0,0) of a representative point pixel for each block obtained by dividing a fixed unit image composed of an input video signal into a plurality of blocks is stored in a memory for a fixed unit period,
Image data I of each pixel of the block of the current field
_k (x, y) and image data I _k-1 (0, 0) of the representative point pixel of the block of the previous field read from the memory.
0)) and the absolute value | I _k-1 (0,0) −I _k (x,
y) a motion vector detecting unit for detecting a motion vector of an image based on |, and an activity of each image data in the vicinity of a representative point pixel for each block obtained by dividing a fixed unit image composed of the input video signal into a plurality of blocks. Activity determining means for determining the image data I _k of the representative point pixel of each block obtained by dividing the image of the fixed unit constituted by the input video signal into a plurality of blocks
(0, 0) is stored in the memory for a fixed unit period, and for the block determined to be significant by the activity determining means, the image data I of each pixel of the block in the current field
_k (x, y) and image data I _k-1 (0, 0) of the representative point pixel of the block of the previous field read from the memory.
0)) and the absolute value | I _k-1 (0,0) −I _k (x,
y) | satisfies the first determination condition of I _k−1 (0,0) ≒ I _k (0,0), and the threshold of the determination is TH
_{As I} , every time a pixel that satisfies the second determination condition of | I _k-1 (0,0) −I _k (x, y) | ≧ TH _I is detected, A frequency distribution table is formed by incrementing the frequency f (x, y), and based on the frequency distribution table, when the frequency value of the coordinates specified by the motion vector detected by the motion vector detection unit is smaller than a predetermined value A camera shake determination unit that determines that the motion vector is due to camera shake of the image; and a correction amount corresponding to the motion vector determined to be due to camera shake of the image by the camera shake determination unit. An image stabilization apparatus comprising: a correction amount generation unit that forms a camera shake correction signal; and a correction unit that performs a camera shake correction process on an input video signal based on a camera shake correction signal supplied from the correction amount generation unit. Apparatus.