JP3223577B2 - Image stabilization device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention detects a motion vector of an image composed of an input video signal, and reads out image data of a missing portion of the image due to camera shake from peripheral memory based on the motion vector. The present invention relates to an image stabilizing device for an image to be interpolated by means of, for example, an image stabilizing device for detecting and moving an image moving amount due to a so-called image stabilization included in video data such as an imaging output of a handy type video camera.

[0002]

2. Description of the Related Art Generally, in a handy type video camera, camera shake during photographing, that is, camera vibration appears as image vibration. Therefore, as an image stabilization device for correcting an image vibration caused by such a camera shake, for example, as disclosed in Japanese Patent Application Laid-Open No. 63-166370, a motion vector of an image is detected and based on this motion vector. A technique for correcting video data stored in an image memory has been proposed.

For detecting a motion vector of an image, for example, a block matching method is employed. In the detection of a motion vector of an image by the block matching method, a screen is divided into a number of regions (referred to as blocks), and a representative point pixel of a previous field located at the center of each block and a pixel of each pixel in a block of the current field are determined. The absolute value of the field difference from the image data is calculated, the absolute value of the field difference of each block is integrated for each corresponding pixel to obtain a correlation integral value, and the correlation integration having coordinates corresponding to the pixel array of one block Form a value table. Then, the motion vector of the entire screen is determined using the coordinate value of the minimum value of the correlation integral value in the correlation integrated value table as the coordinate value of the motion vector of the image.

[0004] The image stabilization apparatus performs image stabilization by moving an output image in accordance with a motion vector and enlarging and outputting the output image, as shown in FIG. 11, for example. Alternatively, as shown in FIG. 12, camera shake correction is performed by interpolating image data of a missing portion of an image due to camera shake with peripheral image data read from a peripheral memory based on the motion vector and outputting the result.

In the detection of a motion vector by the block matching method, a motion vector of an image caused by camera shake and a motion vector caused by the motion of a subject are generated at the same time. Conventionally, as a method for improving the detection accuracy of a motion vector of an image due to camera shake, for example, a method of dividing a screen into a plurality of regions (macroblocks) and detecting a motion vector of the entire screen from the motion vector for each macroblock Is known.
The present applicant divides into a plurality of macroblocks constituting one screen in such a manner that macroblocks that are not adjacent to each other are generated, detects a motion vector of the entire screen from the motion vector for each macroblock, and detects Japanese Patent Application No. 3-10038 discloses a camera shake correction apparatus in which image data of a missing portion of an image is interpolated by peripheral image data read from a peripheral memory based on the motion vector.
No. 4 and others.

[0006]

As described above, in an image stabilizing apparatus for an image in which an output image is moved in accordance with a motion vector, and is enlarged and output to perform image stabilization, the image is always enlarged. Since output is performed, there is a disadvantage that resolution degradation cannot be avoided even when there is no camera shake at all. On the other hand, in an image stabilization apparatus for performing image stabilization using a peripheral memory, since only the image data of a portion lost due to the camera shake is replaced with past data stored in the peripheral memory, enlargement is performed. Since there is no resolution, there is an advantage that the resolution is not deteriorated.

However, as described above, the motion vector of the image composed of the input video signal is detected, and the image data of the missing portion of the image due to camera shake is interpolated with the peripheral image data read from the peripheral memory based on the motion vector. For example, as shown in FIG. 13, in the image stabilizing apparatus for a moving image, when a moving object X exists at a boundary portion between a peripheral image interpolated by peripheral image data and a main image based on an input video signal, mismatching of an output image May occur and the corrected image may be broken. Further, even when an erroneous detection occurs in the detection of a motion vector, for example, as shown in FIG. 14, a mismatch may occur in the output image, and the corrected image may be broken.

Accordingly, the present invention has been made in view of such circumstances, and has as its object to enable high-performance camera shake correction in a handy-type video camera or the like, and to eliminate a portion where an image is lost due to camera shake. In an image stabilization device for interpolating image data with peripheral image data read from a peripheral memory based on a motion vector,
It is an object of the present invention to provide an image stabilization apparatus having a function of stopping image stabilization in a case where a mismatch occurs in an output image and there is a risk of causing a failure in a corrected image.

[0009]

In order to solve the above-mentioned problems, the present invention detects a motion vector of an image composed of an input video signal, and converts the image data of a missing portion of the image due to camera shake into the motion vector. In the image stabilization device for an image interpolated with peripheral image data read from the peripheral memory based on the image data, the correlation of the boundary portion of the corrected image interpolated with the peripheral image data from the peripheral memory is detected, and when the correlation is small, the image stabilization is performed. Control means for stopping the operation.

Further, according to the present invention, a plurality of macroblocks constituting one screen are divided in such a manner that macroblocks not adjacent to each other are generated, and a motion vector of the entire screen is detected from the motion vector for each macroblock. An image shake correction apparatus for interpolating image data of a missing portion of an image due to camera shake with peripheral image data read from the peripheral memory based on the motion vector, wherein the peripheral memory is corrected based on the motion vector for each macroblock. And control means for detecting a correlation of a boundary portion of the corrected image interpolated with the peripheral image data from the camera and stopping the camera shake correction when the correlation is small.

[0011]

In order to solve the above-mentioned problem, the image stabilizing apparatus for an image according to the present invention detects the correlation of the boundary portion of the corrected image interpolated with the peripheral image data from the peripheral memory by the control means. When it is small, control is performed to stop camera shake correction.

Further, in the image stabilizing apparatus for an image according to the present invention, in order to solve the above-mentioned problems, the control means includes:
Based on the motion vector for each macroblock, a correlation is detected at the boundary of the corrected image interpolated with the peripheral image data from the peripheral memory, and if the correlation is small, control for stopping the camera shake correction is performed.

[0013]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram of an image stabilizing apparatus according to an embodiment of the present invention; An image stabilization apparatus for an image according to the present invention is configured, for example, as shown in FIG.

In the first embodiment shown in FIG. 1, the present invention is applied to a camera shake correction device for correcting an image movement caused by camera shake in a handy type video camera.
It comprises a motion vector detection unit 10, a correction signal generation unit 20, a correction unit 30, and a correction control unit 40. In FIG. 1, a signal input terminal 1 is supplied with input video data obtained by digitizing a video signal obtained as an imaging output by an imaging unit (not shown) of the video camera.

In the image stabilizing apparatus, the motion vector detecting section 10 includes a representative point memory 11 and a subtraction circuit 12 to which the input video data is supplied via the signal input terminal 1, and a subtraction output by the subtraction circuit 12. A correlation integrated value table forming circuit 13 to which data is supplied, and a motion vector determining circuit 14 to which correlation integrated value data of the correlation integrated value table formed by the correlation integrated value table forming circuit 13 are supplied.
And

The representative point memory 11 of the motion vector detecting section 10 is composed of input video data.
The image data I _k (0,0) of the representative point pixel for each block obtained by dividing the image of the field into a plurality is stored. Specifically, for example, as shown in FIG. 2, a screen of one field is divided into blocks of m pixels × n lines, and as shown in FIG. 3, a pixel S (0,0) at the center of each block is set as a representative point. age,
The image data I _k (0,0) of each representative point pixel is stored in the representative point memory 11 for one field period. Note that the representative points are uniformly distributed on the screen. Then, the image data I _k−1 (0,0) of each representative point pixel one field before read from the representative point memory 11 is supplied to the subtraction circuit 12.

The subtraction circuit 12 converts the input video data supplied through the signal input terminal 1, that is, the image data of the current field, to m × n for each block.
The difference between the image data I _k (x, y) of each pixel and the image data I _k-1 (0, 0) of the representative point pixel of the corresponding block of the previous field read from the representative point memory 11, that is, the field Absolute value of difference | I _k-1 (0,0)
−I _k (x, y) |. Then, the field difference absolute value | I obtained by the subtraction circuit 12
_k-1 (0,0) -I _k (x, y) | is supplied to the correlation integrated value table forming circuit 13.

The correlation integrated value table forming circuit 13 corresponds to the field difference absolute value | I _k-1 (0,0) -I _k (x, y) | of each block obtained by the subtraction circuit 12. The pixels are integrated over one field period to form a correlation integrated value table having m × n integer coordinates corresponding to the pixel arrangement of one block. This correlation integrated value table forming circuit 13
The correlation integrated value table formed by the above formula indicates the integrated value of m × n field difference absolute values | I _k−1 (0,0) −I _k (x, y) |, that is, the distribution of the correlation integrated value. Then, the m × n correlation integrated values of the correlation integrated value table formed by the correlation integrated value table forming unit 13 are used as the motion vector determining unit 14.
Supplied to

The motion vector determination circuit 14 detects the coordinates of the minimum value of the correlation integrated value in the correlation integrated value table formed by the correlation integrated value table forming circuit 13 and determines a motion vector. The correlation integrated value of the correlation integrated value table formed by the correlation integrated value table forming circuit 13 is represented by the image data I _k-1 (0,0) of the representative point pixel of each block and the image data I _k ( x, y) indicates the inter-field correlation. The coordinates corresponding to the pixel having a higher correlation have a smaller value, and the correlation integrated value of the coordinates corresponding to the motion vector has the minimum value. By doing so, a motion vector can be determined. Then, the motion vector detection unit 10 supplies the motion vector of the detected image to the correction signal generation unit 20.

[0020] The correction signal generation unit 20, _{'as, X t = X t-1} -V t' vector V _t shake the motion vector to form a shake correction signal becomes the correction amount X _t, the camera shake correction signal Is supplied to the correction unit 30.

The correcting section 30 is provided with an address control circuit 31 and a select signal generating circuit 32 to which a camera shake correction signal is supplied from the correction signal generating section 20 and a video signal according to an address signal supplied from the address control circuit 31. A field memory 33 and a peripheral memory 34 in which data is written / read, and video data read from the field memory 33 and the peripheral memory 34 are selectively output according to a select signal supplied from the select signal generating circuit 32. And a selector 35 that performs the operation.

Input video data supplied through the signal input terminal 1 is sequentially written into the field memory 33. The read address of the field memory 33 is controlled by the camera shake correction signal according to the camera shake vector. As a result, video data obtained by moving one field of input video data according to the camera shake vector is obtained from the field memory 33. Then, the video data read from the field memory 33 and the peripheral video data read from the peripheral memory 34 are combined by selection by the selector 35, and are output from the signal output terminal 2 as video data subjected to camera shake correction processing.

The peripheral memory 34 sequentially writes, as peripheral video data, video data of a peripheral portion corresponding to a correction range of an image based on video data subjected to camera shake correction processing output via the selector 35.

Further, the correction control section 40 is supplied with image data read from the field memory 33 of the correction section 30 and peripheral image data read from the peripheral memory 34. The correction control unit 40 compares the image data from the field memory 33 with the peripheral image data from the peripheral memory 34,
When the difference is large, it is determined that the motion vector detected by the motion vector detection unit 10 is not caused by camera shake of the image, and a correction stop control signal is supplied to the correction signal generation unit 20.

As a method of comparing the image data in the correction control unit 40, for example, a correlation is obtained for each data in an area A of a boundary portion shown by hatching in FIG. 4, and the correlation value is smaller than a threshold value. Sometimes, a correction stop control signal may be output. Further, the difference between the data may be detected in an area B indicated by oblique lines rising to the right and an area C indicated by oblique lines to the left in FIG.

When a correction stop control signal indicating that the motion vector is not caused by the camera shake is supplied from the correction control unit 40, the correction signal generator 20 generates the camera shake vector V _t ′. Zero vector V [0,
0] to form a camera shake correction signal, and supplies the camera shake correction signal to the correction unit 30.

That is, the correction control section 40 detects the correlation of the boundary portion of the corrected image interpolated with the peripheral image data from the peripheral memory 34, and performs control for stopping the camera shake correction when the correlation is small.

In the camera shake correction apparatus having such a configuration, the correction control section 40 detects the correlation of the boundary portion of the corrected image interpolated with the peripheral image data from the peripheral memory 34, and when the correlation is small, the camera shake correction is performed. Is performed, so that even if a moving object is present at the boundary portion or an erroneous detection occurs in the detection of a motion vector, a mismatch occurs in the output image, and the corrected image does not fail. In addition, the correction unit 30 can reliably perform the camera shake correction processing on the input video signal, and a natural image output can be obtained.

In the second embodiment shown in FIG. 6, the present invention is applied to a camera shake correction apparatus which employs macroblocks to improve the detection accuracy of a motion vector of an image due to camera shake. It comprises a vector detection unit 60, a correction control unit 70, a correction signal generation unit 80, and a correction unit 90. In FIG. 6, a signal input terminal 51 is supplied with input video data obtained by digitizing a video signal obtained as an imaging output by an imaging unit (not shown) of the video camera.

In this camera shake correction apparatus, the motion vector detecting section 60 stores the image data of the representative point pixels in each block obtained by dividing the image of one field constituted by the input video signal into a plurality of blocks in a memory for one field period. And
Detecting a motion vector of the image based on the absolute value of the difference between the image data of each pixel of the block of the current field and the image data of the representative point pixel of the block of the previous field read from the memory; A field difference detector 61 to which video data is supplied via the signal input terminal 51;
And a motion vector estimating unit 68 to which the correlation integrated value of the correlation integrated value table formed by the correlation integrated value table forming unit 64 is supplied. Become.

The field difference detector 61 comprises a representative point memory 62 to which the input video data is supplied via the signal input terminal 51, and a subtraction circuit 63.

The representative point memory 62 divides an image of one field composed of input video data into blocks of m pixels × n lines, similarly to the representative point memory 11 in the first embodiment. Pixel S at the center of the block
(0, 0) is a representative point, and the image data I of each representative point pixel is
_k (0, 0) is stored in the representative point memory 62 for one field period. The image data I of each representative point pixel one field before read from the representative point memory 62 is read.
_k-1 (0,0) is supplied to the subtraction circuit 63. Further, the subtraction circuit 63 converts the input video data supplied through the signal input terminal 51, that is, the image data of the current field, into image data I _k (x, y) of m × n pixels for each block. And the difference between the image data I _k-1 (0,0) of the representative point pixel of the corresponding block of the previous field read from the representative point memory 62, that is, the absolute value | I _k-1 (0,0) of the inter-field difference ) −I _k (x,
y) | is detected.

Then, the field difference detector 61
Supplies the field difference absolute value | I _k-1 (0,0) −I _k (x, y) | obtained by the subtraction circuit 63 to the correlation integrated value table forming unit 64.

The correlation integrated value table forming section 64 is supplied with the field difference absolute value | I _k-1 (0,0) -I _k (x, y) | obtained by the field difference detection section 61. a macro blocking circuit 65, a macroblock of the field difference absolute value by the macro blocking circuit _{65 | I k-1 (0,0} ) -I k (x, y) | first through is supplied It comprises thirteen absolute value integration circuits 66A to 66M.

The macroblocking circuit 65 converts the field difference absolute value | I _k-1 (0,0) -I _k (x, y) | In the form that blocks occur,
One screen is divided into a plurality of macro blocks. For example, as shown in FIG. 7, one screen is divided into 12 4 × 3 macro blocks B1 to B12.

The field difference absolute value | I macro-blocked by the macro-blocking circuit 65
_k−1 (0,0) −I _k (x, y) | is supplied to the first to twelfth absolute value integration circuits 66A to 66L corresponding to the macroblocks B1 to B9 for each macroblock. At the same time, all macroblocks are supplied to the thirteenth absolute value integration circuit 66M.

The first absolute value integration circuit 66A includes a first
Field difference absolute value | I _k-1 of macro block B1
For (0,0) −I _k (x, y) |, the field difference absolute values of the block composed of the m × n pixels are integrated for each corresponding pixel, and the first macro block B1 Is formed. Hereinafter, similarly, the second
The twelfth to twelfth absolute value integration circuits 66B to 66L respectively correspond to the second to twelfth macroblocks B2 to B1.
2 is formed. Further, the thirteenth absolute value integration circuit 66J forms a correlation integrated value table of the entire one screen including the first to ninth macroblocks B1 to B12.

The correlation integrated value table forming unit 64
The correlation integrated value of the correlation integrated value table of the macroblocks B1 to B12 formed by the first to twelfth absolute value integration circuits 66A to 66L and the thirteenth absolute value integration circuit 66
The correlation integrated value of the correlation integrated value table for one entire screen formed by M is supplied to the motion vector estimating unit 68 and is also supplied to the correction control unit 70.

The motion vector estimating section 68 detects the coordinates of the minimum value of the correlation integrated value of each correlation integrated value table formed by the correlation integrated value table forming section 64 and estimates the motion vector. Accumulated correlation value of each accumulated correlation value table is formed by the accumulated correlation value table forming circuit 64, image data I _k of image data I _k-1 (0,0) and the other pixels of the representative point pixels of each block It indicates the inter-field correlation with (x, y), and the coordinate corresponding to the pixel having a higher correlation has a smaller value, and the correlation integrated value of the coordinate corresponding to the motion vector has the minimum value. By detecting, a motion vector can be estimated.

Then, the motion vector detecting section 60
The motion vector of the image estimated by the motion vector estimation unit 68 is supplied to the correction signal generation unit 80.

Further, the correction control section 70 is based on the correlation integrated value of each correlation integrated value table formed by the first to thirteenth absolute value integration circuits 66A to 66M of the correlation integrated value table forming section 64. The motion vector of each macro block is compared, and when a macro block having a different motion vector is detected in a peripheral portion, it is determined that the motion vector is not caused by camera shake of the image, and a correction stop control signal Is supplied to the correction signal generator 80. That is,
For example, as shown in FIG.
, The motion vector of each macroblock is different from that of the other macroblocks as shown in FIG. Can be determined. As described above, the correction control unit 70 detects the correlation of the boundary portion of the corrected image interpolated with the peripheral image data from the peripheral memory 94 of the correction unit 90 described later based on the motion vector for each macroblock,
When the correlation is small, control is performed to stop camera shake correction.

[0042] The correction signal generation unit 80 _{'as, X t = X t-1} -V t' vector V _t shake the motion vector detected by the motion vector detecting unit 60 image stabilization becomes the correction amount X _t A signal is formed, and the camera shake correction signal is supplied to the correction unit 90. Further, the correction signal generation unit 80 outputs the correction stop control signal indicating that the motion vector detected by the motion vector detection unit 10 is not due to camera shake of the image.
When supplied from 0, a camera shake correction signal is formed with the camera shake vector V _t ′ as a zero vector V [0, 0], and this camera shake correction signal is supplied to the correction unit 80.

As shown in FIG. 10, for example, the correction section 90 includes an address control circuit 91 and a select signal generation circuit 92 to which a camera shake correction signal is supplied from the correction signal generation section 80, and an address control circuit 91. Memory and a peripheral memory 94 in which video data is written / read in accordance with an address signal supplied from the field memory 93 and the peripheral memory 94
And a selector 95 for selectively outputting video data read from the selector in response to a select signal supplied from the select signal generating circuit 92.

Input video data supplied via the signal input terminal 51 is sequentially written into the field memory 93. The read address of the field memory 93 is controlled according to the camera shake vector by the camera shake correction signal. As a result, video data obtained by moving one field of input video data in accordance with the camera shake vector is obtained from the field memory 93. Then, the video data read from the field memory 93 and the peripheral video data read from the peripheral memory 94 are combined by selection by the selector 95, and output from the signal output terminal 52 as video data subjected to camera shake correction processing.

In the peripheral memory 94, video data of a peripheral portion corresponding to a correction range of an image based on video data subjected to image stabilization processing output via the selector 95 is sequentially written as peripheral video data.

In the camera shake correction apparatus having such a configuration, the correction control unit 70 performs the correlation of the boundary of the corrected image interpolated with the peripheral image data from the peripheral memory 94 based on the motion vector for each macroblock. Is detected, and if the correlation is small, control is performed to stop image stabilization.
Even when a moving object is present at the boundary portion or when erroneous detection occurs in the detection of a motion vector, a mismatch does not occur in the output image and the corrected image does not break down. The camera shake correction processing can be reliably performed on the signal, and a natural image output can be obtained.

[0047]

As is apparent from the above description, according to the present invention, a motion vector of an image constituted by an input video signal is detected, and image data of a missing portion of the image due to camera shake is converted into the motion vector. In the image stabilizing apparatus for an image interpolated with peripheral image data read from the peripheral memory based on the control unit, the control unit detects a correlation between boundary portions of the corrected image interpolated with the peripheral image data from the peripheral memory,
Since the control for stopping the camera shake correction is performed when the correlation is small, a mismatch occurs in the output image even when a moving object is present in the boundary portion or an erroneous detection occurs in the detection of the motion vector, and the corrected image is corrected. Thus, the correction unit can reliably perform the camera shake correction process on the input video signal, and a natural image output can be obtained.

According to the present invention, a plurality of macroblocks constituting one screen are divided in such a manner that macroblocks that are not adjacent to each other are generated, and the motion vector of the entire screen is calculated from the motion vector for each macroblock. In the image stabilization device of the image that interpolates the image data of the missing part of the image due to camera shake with the peripheral image data read from the peripheral memory based on the motion vector,
Based on the motion vector for each macroblock, the control unit detects the correlation of the boundary portion of the corrected image interpolated with the peripheral image data from the peripheral memory, and performs control to stop the camera shake correction when the correlation is small. Even when a moving object is present at the boundary portion or erroneous detection occurs in the detection of a motion vector, a mismatch occurs in the output image, and the corrected image does not break down. Can be reliably subjected to a camera shake correction process, and a natural image output can be obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a first embodiment of an image stabilization apparatus according to the present invention.

FIG. 2 is a diagram illustrating a state in which a screen is divided into blocks in a motion vector detection unit of the camera shake correction apparatus.

FIG. 3 is a diagram showing a structure of one block of a screen divided into blocks.

FIG. 4 is a diagram provided for describing an example of an operation of comparing image data in a correction control unit of the camera shake correction apparatus.

FIG. 5 is a diagram for explaining another example of the comparison operation of the image data in the correction control unit of the camera shake correction apparatus.

FIG. 6 is a block diagram illustrating a configuration of a second embodiment of an image stabilization apparatus according to the present invention.

FIG. 7 is a diagram showing a state of a macroblock obtained by dividing one screen into 12 parts.

FIG. 8 is a diagram illustrating a state of an image when a moving object has entered.

9 is a diagram illustrating a state of a motion vector of a macroblock in a state of the image illustrated in FIG. 8;

FIG. 10 is a block diagram illustrating a configuration of a correction unit according to the second embodiment.

FIG. 11 is a diagram provided for describing a camera shake correction operation of an image enlargement method.

FIG. 12 is a diagram provided for describing a camera shake correction operation of the peripheral memory system.

FIG. 13 is a diagram provided to explain a camera shake correction operation of the peripheral memory method when a moving object is present.

FIG. 14 is a diagram provided for describing a camera shake correction operation of the peripheral memory system when an error occurs in detection of a motion vector.

[Explanation of symbols]

 10, 60... Motion vector detecting section 20, 80... Correction signal generating section 30, 90... Correcting section 33, 93. ..Peripheral memories 40, 70...

Continuation of the front page (72) Inventor Tsukasa Hashino 6-7-35 Kita Shinagawa, Shinagawa-ku, Tokyo Sony Corporation (56) References JP-A-4-255179 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 5/232 G03B 5/00

Claims (2)

(57) [Claims] 1. A camera shake of an image in which a motion vector of an image composed of an input video signal is detected, and image data of a missing portion of the image due to camera shake is interpolated by peripheral image data read from a peripheral memory based on the motion vector. A correction device for detecting a correlation of a boundary portion of the corrected image interpolated by the peripheral image data from the peripheral memory, and stopping the camera shake correction when the correlation is small; apparatus. 2. A screen is divided into a plurality of macroblocks constituting one screen in such a manner that macroblocks that are not adjacent to each other are generated. In an image stabilization apparatus for interpolating image data of a missing portion of an image with peripheral image data read from a peripheral memory based on the motion vector, a peripheral image from the peripheral memory based on the motion vector for each macroblock An image stabilization apparatus for an image, comprising: a control unit that detects a correlation at a boundary portion of a corrected image interpolated by data and stops image stabilization when the correlation is small.