JP3200890B2 - Image vibration correction device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image vibration correcting apparatus for correcting an image pickup output of a handy type video camera by detecting a moving amount of an image caused by camera shake contained in video data.

[0002]

2. Description of the Related Art Generally, in a handy type video camera, camera shake during photographing, that is, camera vibration appears as image vibration. Therefore, as an image stabilization device for correcting image vibration due to such a camera shake, for example, as disclosed in Japanese Patent Application Laid-Open No. 63-166370, a motion vector of an image is detected, and based on this motion vector, A device that corrects video data stored in an image memory has been proposed.

For detecting a motion vector of an image, for example, a block matching method is employed. In the detection of a motion vector of an image by the block matching method, a screen is divided into a number of regions (referred to as blocks), and a representative point pixel of a previous field located at the center of each block and a pixel of each pixel in a block of the current field are determined. The absolute value of the field difference from the image data is calculated, the absolute value of the field difference of each block is integrated for each corresponding pixel to obtain a correlation integral value, and the correlation integration having coordinates corresponding to the pixel array of one block Form a value table. Then, the motion vector of the entire screen is determined using the coordinate value of the minimum value of the correlation integral value in the correlation integrated value table as the coordinate value of the motion vector of the image.

[0004] The image blur correction device converts the detected motion vector into a correction signal, and performs a correction for moving the current image using the correction signal. The correction accuracy of such an image stabilization device depends on the detection accuracy of the motion vector of the image.

[0005]

By the way, as described above, the motion vector of the entire screen is determined by using the coordinate value of the minimum value of the correlation integrated value in the correlation integrated value table as the coordinate value of the motion vector of the image. In the detection of a motion vector of an image by the block matching method, a motion vector due to camera shake and a motion vector due to movement of a subject are simultaneously generated. Therefore, the image shake correction apparatus needs to determine the motion vector due to the camera shake and the motion vector due to the movement of the subject, and form a camera shake correction signal using only the motion vector due to the camera shake to perform the camera shake correction. If the motion vector due to the movement of the subject is erroneously determined to be the motion vector due to camera shake and the camera shake correction is performed, the background image that should be stationary will move due to the camera shake correction, and Becomes an image. Conversely, if the motion vector due to the camera shake is erroneously determined to be the motion vector due to the movement of the subject, the camera shake cannot be corrected.

In order to discriminate the motion vector due to the camera shake and the motion vector due to the movement of the subject, a conventional method is as follows.
It required expensive large-capacity memory.

[0007] The present invention has been made in view of such circumstances, and has as its object to enable high-performance camera shake correction in a handy-type video camera or the like.
An image vibration correction device that reliably determines whether a detected motion vector of an image is a motion vector of an image due to camera shake or a motion vector of a subject by using a small-capacity memory and performs vibration correction. To provide.

[0008]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, an image vibration correcting apparatus according to the present invention provides a representative image for each block obtained by dividing an image of one field constituted by an input video signal into a plurality of blocks. Point pixel image data I _k (0,0)
And a representative point memory for storing in a memory for one field period,
Image data I of each pixel of the block of the current field
_k (x, y) and image data I _k-1 (0, 0) of the representative point pixel of the block of the previous field read from the memory.
0)) and the absolute value | I _k-1 (0,0) −I _k (x,
y) A subtraction circuit for calculating |, and a field difference absolute value | I _k-1 (0,0) -I obtained by the subtraction circuit
a motion vector detecting circuit for detecting a motion vector of an image based on _k (x, y) |, and a field difference absolute value | I _k-1 (0,0) -I obtained by the subtraction circuit.
_k (x, y) | satisfies the first determination condition of I _k−1 (0,0) ≒ I _k (0,0), and finely moves away from the representative point pixel near the representative point pixel Field difference absolute value | I corresponding to a pixel roughly sampled according to
_{k-1 (0,0) -I k} (Δx, Δy) | is the threshold of the determination as _{TH I, | I k-1} (0,0) -I k (Δx, Δy) | ≧ TH in each time of detecting the second determination condition is satisfied pixel to be _I, the frequency distribution table formation circuit for forming a frequency distribution table increments the frequency f ([Delta] x, [Delta] y) of coordinates corresponding to the pixels, the frequency distribution table Based on the frequency distribution table formed by the formation circuit, when the frequency value of the coordinates specified by the motion vector detected by the motion vector detection circuit is smaller than a predetermined value, the motion vector is caused by image vibration. A vibration determination circuit that determines that the vibration vector, a correction amount generation unit that forms a vibration correction signal of a correction amount according to the motion vector determined to be caused by the vibration of the image by the vibration determination circuit, Supplied from the correction amount generator It is characterized in further comprising a correction unit that performs vibration correction processing to the input video signal by motion compensation signal.

[0009]

In the image vibration correction apparatus according to the present invention, the field difference absolute value obtained by the subtraction circuit from the image data of the representative point pixel for each block obtained by dividing the one-field image constituted by the input video signal into a plurality of blocks. | I _k-1 (0,
0) −I _k (x, y) |, a motion vector of an image is detected by block matching in a motion vector detection circuit, and the field difference absolute value | I
_k−1 (0,0) −I _k (x, y) |, the first determination condition of I _k−1 (0,0) ≒ I _k (0,0) is satisfied, and the representative point In the vicinity of the pixel, the field difference absolute value | I corresponding to the pixel coarsely sampled as the distance from the representative point pixel increases.
_{k-1 (0,0) -I k} (Δx, Δy) | is the threshold of the determination as _{TH I, | I k-1} (0,0) -I k (Δx, Δy) | ≧ TH in each time of detecting the second determination condition is satisfied pixel to be _I, to form a frequency distribution table is incremented frequency f a (Δx, Δy) of the coordinates corresponding to the pixels by the frequency distribution table formation circuit. Then, based on the frequency distribution table, the vibration determination circuit determines whether or not the motion vector is due to vibration, and generates a vibration correction signal of a correction amount corresponding to the motion vector due to vibration. The vibration correction processing is performed on the input video signal by the correction unit.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an image vibration correcting apparatus according to the present invention will be described below in detail with reference to the drawings. The image vibration correction apparatus according to the present invention is configured as shown in FIG. 1, for example.

The image vibration correcting apparatus shown in FIG.
The present invention is applied to a camera shake correction device that corrects image movement due to camera shake in a handy type video camera, and includes a field correlation detection unit 10, a motion vector detection unit 20, a camera shake determination unit 30, a correction amount generation unit 40, and a correction amount generation unit. A part 50 is provided. In FIG. 1, input video data obtained by digitizing a video signal obtained as an imaging output by an imaging unit (not shown) of the video camera is supplied to a signal input terminal 1. Then, the input video data is supplied from the signal input terminal 1 to the field correlation detection unit 10 and the correction unit 50.

In this camera shake correction apparatus, the field correlation detecting section 10 comprises a representative point memory 11 to which the input video data is supplied via the signal input terminal 1 and a subtraction circuit 12.

The representative point memory 11 in the field correlation detector 10 stores image data I _k (0,0) of representative point pixels for each block obtained by dividing an image of one field constituted by input video data into a plurality of blocks. Is stored. Specifically, for example, as shown in FIG. 2, a screen of one field is divided into blocks of m pixels × n lines, and as shown in FIG. Image data I _k (0,0) of the representative point memory 11
For one field period. Note that the representative points are uniformly distributed on the screen. Then, the image data I _k−1 (0,0) of each representative point pixel one field before read from the representative point memory 11 is supplied to the subtraction circuit 12.

The subtraction circuit 12 converts the input video data supplied through the signal input terminal 1, that is, the image data of the current field, to m × n for each block.
The difference between the image data I _k (x, y) of each pixel and the image data I _k-1 (0, 0) of the representative point pixel of the corresponding block of the previous field read from the representative point memory 11, that is, the field Absolute value of difference | I _k-1 (0,0)
−I _k (x, y) |.

The field correlation detector 10
Is the field difference absolute value | I _k-1 (0,0) -I obtained as subtraction output data by the subtraction circuit 12.
_k (x, y) | is supplied to the motion vector detection unit 20 and the camera shake determination unit 30.

The motion vector detecting section 20 is supplied with the field difference absolute value | I _k-1 (0,0) -I _k (x, y) | detected by the field correlation detecting section 10. A correlation integrated value table forming circuit 21 and a motion vector detecting circuit 22 to which a correlation integrated value of the correlation integrated value table formed by the correlation integrated value table forming circuit 21 is supplied.

The correlation integrated value table forming circuit 21 generates a field difference absolute value | I _k−1 (0,0) −I _k (x, x) of each block obtained by the field correlation detection unit 10.
y) | is integrated over one field period for each corresponding pixel to form a correlation integrated value table having m × n integer coordinates corresponding to a pixel array of one block. The correlation integrated value table formed by the correlation integrated value table forming circuit 13 is mx
n field difference absolute values | I _k-1 (0,0) −I _k
In other words, the distribution of the integrated value of (x, y) |, that is, the integrated value of the correlation is shown, and the integrated value of the correlation having the strongest field correlation is the minimum value. Then, m × n correlation integrated values of the correlation integrated value table formed by the correlation integrated value table forming circuit 21 are supplied to the motion vector detecting circuit 22.

The motion vector detecting circuit 22 detects the coordinates of the minimum value of the correlation integrated value of the correlation integrated value table formed by the correlation integrated value table forming circuit 21. The correlation integrated value of the correlation integrated value table formed by the correlation integrated value table forming circuit 21 is the image data I _k−1 of the representative point pixel of each block.
This indicates the inter-field correlation between (0, 0) and the image data I _k (x, y) of another pixel. The coordinates corresponding to a pixel having a higher correlation have a smaller value, and the correlation of the coordinates corresponding to a motion vector is smaller. The integrated value becomes the minimum value. That is, the motion vector detection circuit 22 detects the coordinate of the minimum value of the correlation integrated value in the correlation integrated value table as the coordinate indicating the motion vector.

Then, the motion vector detecting section 20
The coordinates of the minimum value of the correlation integrated value detected by the motion vector detection circuit 22, that is, the motion vector, are supplied to the camera shake determination unit 30 and the correction amount generation unit 40.

Further, in this camera shake correction apparatus, the camera shake determination section 30 determines the field difference absolute value | I _k-1 (0,
0) -I _k (x, y) |
A frequency distribution table forming circuit 32 for forming a frequency distribution table indicating a field correlation based on the condition determining circuit 31;
Camera shake determination for determining whether or not the motion vector of the image detected by the motion vector detection unit 20 is the motion vector of the image due to camera shake based on the frequency distribution table formed by the frequency distribution table forming unit 32 The circuit 33 is provided.

Here, in the image pickup output of an image accompanied by camera shake obtained by a handy type video camera, the entire screen moves because the camera itself moves due to camera shake, so that the motion obtained for each block as shown in FIG. Vector v
_In contrast, _A coincides, but in the image pickup output of an image without camera shake obtained by a fixed video camera, when the object OB moves from a position indicated by a broken line to a position indicated by a solid line as shown in FIG. OB is only the motion vector V _B is generated in the block are located. Accordingly, as shown in FIG. 6, the image data I of each representative point pixel is provided for each block obtained by dividing the screen of one field into a plurality.
_k (0, 0) and the image data I _k-1 (0, 0,
By performing a motion determination by comparing with (0) and detecting a still block, it is possible to discriminate between the image with the camera shake and the image without the camera shake.

The condition determining circuit 31 in the camera shake determining section 30 is used to
Field difference absolute value of each block detected by |
I _k−1 (0,0) −I _k (0,0) | satisfies the first determination condition of I _k−1 (0,0) ≒ I _k (0,0), and the representative In the vicinity of a point pixel, image data I _k (Δx, Δ
It is determined whether or not y) satisfies a second determination condition of | I _k−1 (0,0) −I _k (Δx, Δy) | ≧ Th _I. Here, the Th _I is a threshold for determining the presence or absence of level difference in space.

That is, the condition determination circuit 31 detects a stationary block candidate having a representative point pixel having a strong field correlation according to the first determination condition, and determines a representative point candidate according to the second determination condition for the stationary block candidate. I _k (Δ) of each pixel with respect to image data I _k (0,0) of the pixel
x, Δy) based on the correlation. Then, the judgment output of the condition judgment circuit 31 is supplied to the frequency distribution table forming circuit 32.

The frequency distribution table forming circuit 32 has a memory table 32A having coordinates corresponding to a pixel array of one block corresponding to pixels that are finely sampled in the vicinity of the representative point pixel and coarsely sampled away from the representative point pixel.
And the determination output by the condition determination circuit 31,
Each time a pixel satisfying the first and second determination conditions is detected, the adder 32B increments the frequency f (x, y) of the corresponding coordinate. In the frequency distribution table forming circuit 32, the image data I _k (x, y) of the current field and the image data I _k of each representative point pixel one field before are stored.
_k-1 (0,0) | I _k-1 (0,0) −
Based on I _k (Δx, Δy) |, a conditional frequency distribution table indicating the correlation between one field of the representative point pixels is formed in the memory table 32A. Then, the frequency value of the frequency distribution table indicating the correlation between one field formed in the memory table 32A by the frequency distribution table forming circuit 32 is supplied to the camera shake determination circuit 33.

Here, for the image pickup output of an image without camera shake obtained by a video camera fixed by a tripod, the condition judgment circuit 31 sets Δ to one pixel pitch and sets the field difference value | I _k− for image data of all pixels. ₁ (0,
0) −I _k (x, y) |, and the frequency distribution table formed by the frequency distribution table forming circuit 32 is, for example, as shown in FIG. due to the strong spatial correlation, whereas frequency value at the coordinates P (-1, 0) to be specified is a small value by the motion vector V _t, also in the large image of the motion, the motion as shown in FIG. 8 coordinates P (-4 specified by the vector V _t,
The value of 0) increases. As described above, in the image with small motion, the frequency value has a small value due to the strong spatial correlation near the representative point pixel, whereas in the image with large motion, the correlation is broken and the frequency value is large. Become. That is, since the correlation between pixels decreases as the spatial distance increases, it becomes easy to determine whether an object is moving or a camera shake is occurring as the detected motion vector increases. Therefore,
If the amount of motion is large, even if the accuracy of the frequency distribution table is reduced, it is possible to ensure the same determination accuracy as when the amount of motion is small.

Therefore, in the image stabilizing apparatus of this embodiment, as described above, near the representative point pixel, the field difference absolute value | I _k-1 (0, 0) corresponding to the pixel that is finely sampled as the distance from the representative point pixel increases. 0) -I
_k (Δx, Δy) |, the frequency distribution table forming circuit 32
Thus, as shown in FIG. 9, for example, a frequency distribution table of 49 coordinates with spatially different resolutions is formed. Therefore, the table memory 32A of the frequency distribution table forming circuit 32 only needs to have a storage capacity of 49 units. On the other hand, the condition judgment circuit 31 judges the condition of the field difference value | I _k-1 (0,0) -I _k (x, y) | For example, when a frequency distribution table of 17 × 17 = 289 coordinates is formed as shown in FIG. 10, a storage capacity of 289 units is required for the table memory 32A.

The camera shake determination circuit 33 calculates the frequency distribution table formed by the frequency distribution table forming circuit 32,
When the frequency value of the coordinates corresponding to the motion vector of the image detected by the motion vector detection unit 20 is smaller than a predetermined value, the motion vector of the image detected by the motion vector detection unit 20 is used as the image shake. If the frequency value is larger than a predetermined value, it is determined that the motion vector of the image detected by the motion vector detecting unit 20 is not due to camera shake of the image.

Then, the judgment output of the camera shake judging section 30 is supplied to the correction amount generating section 40.

When the judgment output indicating that the motion vector detected by the motion vector detecting section 20 is caused by the camera shake of the image is supplied from the camera shake judging section 30, the correction amount generating section 40 outputs Motion vector detector 2
The motion vector detected by the motion vector 0 is represented by a shake vector V _t '
As to form the image stabilization signal _{X t = X t-1 -V} t ' becomes the correction amount X _t, and supplies the image stabilization signal to the correcting unit 50. When the determination output indicating that the motion vector detected by the motion vector detection unit 10 is not caused by the camera shake of the image is supplied from the camera shake determination unit 30, the correction amount generation unit 40 sets the camera shake vector V _t 'is replaced by the zero vector V [0,0]
Is formed, and the camera shake correction signal is supplied to the correction unit 50.

As shown in FIG. 11, for example, the correction section 50 includes an address control circuit 51 and a select signal generation circuit 52 to which a camera shake correction signal is supplied from the correction amount generation section 40, and an address control circuit 51. Memory 53 and peripheral memory 4 in which video data is written / read in accordance with an address signal supplied from
4 and a selector 55 for selectively outputting video data read from the field memory 53 and the peripheral memory 54 in accordance with a select signal supplied from the select signal generating circuit 52.

Input video data supplied via the signal input terminal 1 is sequentially written into the field memory 52. The read address of the field memory 42 is controlled by the camera shake correction signal in accordance with the camera shake vector. As a result, video data obtained by moving one field of input video data in accordance with the camera shake vector is obtained from the field memory 52. Then, the video data read from the field memory 52 and the peripheral video data read from the peripheral memory 53 are combined by selection by the selector 55, and are output from the signal output terminal 2 as video data subjected to camera shake correction processing.

In the peripheral memory 53, the video data of the peripheral portion corresponding to the image correction range of the video data subjected to the camera shake correction output through the selector 55 is sequentially written as peripheral video data.

In this camera shake correction apparatus, the camera shake determination section 30 reliably determines whether or not the motion vector of the image detected by the motion vector detection section 10 is attributable to camera shake as described above. Since the correction can be performed, camera shake correction can be reliably performed, and a natural image output can be obtained.

[0034]

As is apparent from the above description, in the image vibration correcting apparatus according to the present invention, the representative point pixel of each block obtained by dividing the one-field image formed by the input video signal into a plurality of blocks is obtained. Field difference absolute value | I _k-1 (0,0) -I obtained by a subtraction circuit from image data
A motion vector of an image is detected by block matching in a motion vector detection circuit based on _k (x, y) |, and the field difference absolute value | I
_k−1 (0,0) −I _k (x, y) |, the first determination condition of I _k−1 (0,0) ≒ I _k (0,0) is satisfied, and the representative point In the vicinity of the pixel, the field difference absolute value | I corresponding to the pixel coarsely sampled as the distance from the representative point pixel increases.
_{k-1 (0,0) -I k} (Δx, Δy) | is the threshold of the determination as _{TH I, | I k-1} (0,0) -I k (Δx, Δy) | ≧ TH in each time of detecting the second determination condition is satisfied pixel to be _I, the frequency f ([Delta] x, [Delta] y) of coordinates corresponding to the pixel because it forms a frequency distribution table formation circuit a frequency distribution table that is incremented, the frequency distribution The memory capacity required for the table forming circuit can be reduced. That is, if the amount of motion is large, even if the accuracy of the frequency distribution table is reduced, it is possible to ensure the same determination accuracy as when the amount of motion is small.

Then, based on the frequency distribution table, it is determined whether or not the motion vector is due to vibration by a vibration determining circuit, and a vibration correction signal of a correction amount corresponding to the motion vector due to vibration is generated. The vibration correction processing can be reliably performed on the input video signal by the correction amount generation unit.

As described above, in the image vibration correcting apparatus according to the present invention, it is possible to reliably determine whether or not a motion vector of an image is caused by a camera shake with a small memory capacity and to perform a camera shake correction. This makes it possible to perform high-performance camera shake correction in a handy-type video camera or the like.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a camera shake correction apparatus according to the present invention.

FIG. 2 is a diagram illustrating a state in which a screen is divided into blocks in a motion vector detection unit of the camera shake correction apparatus.

FIG. 3 is a diagram showing a structure of one block of the screen divided into blocks.

FIG. 4 is a diagram schematically showing a state of occurrence of a motion vector due to camera shake.

FIG. 5 is a diagram schematically showing a state of generation of a motion vector due to a motion of a subject.

FIG. 6 is a diagram for explaining a principle of determining a motion vector due to a camera shake and a motion vector due to a motion of a subject;

FIG. 7 shows a motion of a frequency distribution table formed by detecting a field correlation of all pixels by a frequency distribution table forming circuit in the image stabilizing device when an amount of motion is small with respect to an image output obtained by a fixed video camera. It is a figure showing a frequency value near a vector.

FIG. 8 is a diagram showing a frequency distribution table generated by detecting a field correlation of all pixels by the frequency distribution table forming circuit when an amount of motion is large with respect to an image output obtained by a fixed video camera; It is a figure showing a numerical value.

FIG. 9 is a diagram showing coordinates of respective frequency values of a frequency distribution table formed by a frequency distribution table forming circuit in the camera shake correction apparatus.

FIG. 10 is a diagram showing coordinates of respective frequency values when a frequency distribution table is formed by detecting a field correlation of all pixels by a frequency distribution table forming circuit in the camera shake correction apparatus.

FIG. 11 is a block diagram illustrating a configuration of a correction unit of the camera shake correction apparatus.

[Explanation of symbols]

 10 Field correlation detection unit 11 Representative point memory 12 Subtraction circuit 20 Motion Vector detection unit 21 Correlation integrated value table forming circuit 22 Motion vector detection circuit 30 Camera shake determination unit 31 ······ Condition determination circuit 32 ······· Frequency distribution table forming circuit 32A ····· Table memory 33 ···· .......

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Ken Satoshi 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Tsukasa Hashino 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo (56) References JP-A-3-3571 (JP, A) JP-A-1-236880 (JP, A) JP-A-61-269475 (JP, A) JP-A-61-201588 (JP, A) (58) Field surveyed (Int. Cl. ⁷ , DB name) H04N 5/225-5/247

Claims (1)

(57) [Claims] 1. A representative point memory for storing image data I _k (0,0) of a representative point pixel for each block obtained by dividing an image of one field constituted by an input video signal into a plurality of blocks for one field period. The image data I of each pixel of the block of the current field
_k (x, y) and image data I _k-1 (0, 0) of the representative point pixel of the block of the previous field read from the memory.
0)) and the absolute value | I _k-1 (0,0) −I _k (x,
y) a subtraction circuit for calculating |, and a field difference absolute value | I obtained by the subtraction circuit
a motion vector detection circuit for detecting a motion vector of an image based on _k-1 (0,0) -I _k (x, y) |, and a field difference absolute value | I obtained by the subtraction circuit.
_k−1 (0,0) −I _k (x, y) |, the first determination condition of I _k−1 (0,0) ≒ I _k (0,0) is satisfied, and the representative point In the vicinity of the pixel, the field difference absolute value | I corresponding to the pixel coarsely sampled as the distance from the representative point pixel increases.
_{k-1 (0,0) -I k} (Δx, Δy) | is the threshold of the determination as _{TH I, | I k-1} (0,0) -I k (Δx, Δy) | ≧ TH in each time of detecting the second determination condition is satisfied pixel to be _I, the frequency distribution table formation circuit for forming a frequency distribution table increments the frequency f ([Delta] x, [Delta] y) of coordinates corresponding to the pixels, the frequency distribution table Based on the frequency distribution table formed by the formation circuit, when the frequency value of the coordinates specified by the motion vector detected by the motion vector detection circuit is smaller than a predetermined value, the motion vector is caused by image vibration. A vibration determination circuit that determines that the vibration vector, a correction amount generation unit that forms a vibration correction signal of a correction amount according to the motion vector determined to be caused by the vibration of the image by the vibration determination circuit, Supplied from the correction amount generator Vibration correction apparatus of the image, characterized in that it comprises a correction unit that performs vibration correction processing to the input video signal by the vibration correcting signal.