JP3303311B2 - Image vibration correction apparatus and image vibration correction method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image vibration correcting apparatus and a method for correcting an image output of a handy type video camera by detecting a moving amount of an image caused by a so-called camera shake contained in video data. About.

[0002]

2. Description of the Related Art Generally, in a handy type video camera, camera shake during photographing, that is, camera vibration appears as image vibration. Therefore, as an image motion compensating device for compensating image vibration due to such camera shake, for example, as disclosed in Japanese Patent Application Laid-Open No. 63-166370, a motion vector of an image is detected and based on this motion vector. A device that corrects video data stored in an image memory has been proposed.

For detecting a motion vector of an image, for example, a block matching method is employed. In the detection of a motion vector of an image by the block matching method, a screen is divided into a number of regions (referred to as blocks), and a representative point pixel of a previous frame located at the center of each block and a pixel of each pixel in a block of the current frame are determined. The absolute value of the frame difference with the image data is calculated, the absolute value of the frame difference of each block is integrated for each corresponding pixel to obtain a correlation integral value, and the correlation integration having coordinates corresponding to the pixel array of one block Form a value table. Then, the motion vector of the entire screen is determined using the coordinate value of the minimum value of the correlation integral value in the correlation integrated value table as the coordinate value of the motion vector of the image.

[0004] The image motion compensator converts the detected motion vector into a compensation signal, and carries out compensation for moving the current picture using the compensation signal. The correction accuracy of such an image motion correction device depends on the detection accuracy of a motion vector of an image.

[0005]

By the way, as described above, the motion vector of the entire screen is determined by using the coordinate value of the minimum value of the correlation integrated value in the correlation integrated value table as the coordinate value of the motion vector of the image. In the image motion vector detecting apparatus using the block matching method, the correlation integral value is discretely calculated for each pixel, and the motion vector of the image is obtained only by the integer coordinate values in the correlation integrated value table. Therefore, there is a problem that the detection accuracy is low. Therefore, when a camera shake correction signal is formed based on a motion vector of an image detected by a conventional image motion vector detection device and a camera shake correction is performed, sufficient correction accuracy cannot be obtained, and the image becomes unnatural. Would.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and enables a handy type video camera or the like to perform high-performance image stabilization without giving a drift to an image that should be stationary. It is an object of the present invention to provide an image vibration correcting apparatus and an image vibration correcting method which are capable of detecting a motion vector of an image with high accuracy by interpolation estimation processing.

[0007]

An image vibration correcting apparatus according to the present invention is a memory for storing image data of a representative point pixel for each block obtained by dividing an image of one frame constituted by an input video signal into a plurality of blocks. Difference detection means for detecting, for each pixel, an absolute value of a difference between image data of each pixel of the block of the current frame and image data of a representative point pixel of the block of the previous frame stored in the memory; The absolute value of the difference detected for each pixel in each block is integrated for each pixel corresponding to the one frame, and the correlation integrated value having the respective integration results at coordinates corresponding to the pixel array of one block A correlation integrated value table forming means for forming a table, and a minimum value coordinate of the correlation integrated value of the correlation integrated value table formed by the correlation integrated value table forming means is detected as a minimum value coordinate. When the coordinates are different from the coordinates corresponding to the representative point pixel, each correlation integrated value corresponding to the minimum correlation integrated value of the minimum value coordinate and coordinates around the minimum value coordinate on the first axis passing through the minimum value coordinate The first coordinate on the first axis, which is proportionally divided from the peripheral coordinates on the first axis, is obtained from the difference value between the first coordinate and the minimum coordinate integrated value of the minimum coordinate and the minimum coordinate. On the second axis, which is proportionally divided from the peripheral coordinates on the second axis by a difference value with each correlation integrated value corresponding to the coordinates around the minimum value coordinate on the second axis passing through the second axis Motion vector estimating means for obtaining a second coordinate, further estimating a motion vector of an image by combining at least the first coordinate and the second coordinate, and a motion vector estimated by the motion vector estimating means The correction signal of the correction amount according to the A correction amount generating means for, characterized in that the correction signal supplied from the correction amount generating means and a correcting means for applying vibration correction processing on the image constituting the frame. Also, the image vibration correction method according to the present invention divides one frame into a plurality of blocks, and calculates a difference between each pixel data in the block of the current frame and representative pixel data of a previous frame corresponding to the block. Detecting the absolute value; and calculating the absolute value of the difference detected for each pixel in each of the blocks by the 1
Forming a correlation integrated value table having the respective integration results at coordinates corresponding to one block of pixel arrangement by integrating for each corresponding pixel over a frame; Detecting the value coordinates as the minimum value coordinates; and, when the minimum value coordinates are different from the coordinates corresponding to the representative point pixel, a first correlation integration value of the minimum value coordinates and a first value passing through the minimum value coordinates. The first coordinate on the first axis proportionally divided from the peripheral coordinate on the first axis by a difference value with each correlation integrated value corresponding to the coordinate around the minimum value coordinate on the axis Is calculated by using a difference value between a minimum correlation integrated value of the minimum value coordinate and each correlation integrated value corresponding to coordinates around the minimum value coordinate on a second axis passing through the minimum value coordinate. Second proportional division of the peripheral coordinates on the axis Estimating a motion vector of an image by obtaining the above second coordinates and combining at least the first coordinates and the second coordinates; and forming a correction signal having a correction amount corresponding to the motion vector. And a step of subjecting the image forming the frame to a vibration correction process based on a correction signal supplied from the correction amount generating means.

[0008]

According to the present invention, one frame is divided into a plurality of blocks, and each pixel data in the block of the current frame is divided into:
The absolute value of the difference from the representative pixel data of the previous frame corresponding to the block is detected, and the absolute value of the difference detected for each pixel in each block is integrated for each pixel corresponding to the one frame. Then, a correlation integrated value table having the respective integrated results at coordinates corresponding to the pixel arrangement of one block is formed. The minimum value coordinate of the correlation integration value in the correlation integration value table is detected as the minimum value coordinate, and when the minimum value coordinate is different from the coordinate corresponding to the representative point pixel, the minimum correlation integration value of the minimum value coordinate is detected. And a value obtained by proportionally dividing the peripheral coordinates on the first axis by a difference value between each of the correlation integrated values corresponding to the peripheral coordinates of the minimum value coordinate on the first axis passing through the minimum value coordinate. A first coordinate on one axis is obtained, and a minimum correlation integrated value of the minimum value coordinate and a second coordinate passing through the minimum value coordinate are obtained.
A second coordinate on the second axis proportionally divided from a peripheral coordinate on the second axis by a difference value with each correlation integrated value corresponding to a coordinate around the minimum value coordinate on the axis ,
A motion vector of an image is estimated by combining at least the first coordinates and the second coordinates. Then, a correction signal of a correction amount corresponding to the motion vector is formed, and the image forming the frame is subjected to vibration correction processing by the correction signal supplied from the correction amount generating means.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below in detail with reference to the drawings. The present invention is applied to, for example, a camera shake correction apparatus having a configuration as shown in FIG.

The image motion vector detecting device 10 shown in FIG. 1 constitutes a camera shake correction device for correcting image movement caused by camera shake in a handy type video camera together with the correction amount generator 20 and the correction unit 30. . In FIG. 1, input video data obtained by digitizing a video signal obtained as an imaging output by an imaging unit (not shown) of the video camera is supplied to a signal input terminal 1.

The motion vector detecting device 10 includes a representative point memory 11 and a subtraction circuit 12 to which the input video data is supplied through the signal input terminal 1, and a correlation to which subtraction output data from the subtraction circuit 12 is supplied. An integrated value table forming circuit 13 and a motion vector estimating circuit 14 to which correlation integrated value data of the correlation integrated value table formed by the correlation integrated value table forming circuit 13 are supplied.

The representative point memory 11 stores image data of representative point pixels for each block obtained by dividing one frame image composed of the input video data into a plurality of blocks. Specifically, for example, as shown in FIG. 2, a screen of one frame is divided into blocks of m pixels × n lines, and as shown in FIG. 3, a pixel at the center of each block is set as a representative point. Are stored in the representative point memory 11 for one frame period. Note that the representative points are uniformly distributed on the screen. Then, the image data of each representative point pixel one frame before read from the representative point memory 11 is supplied to the subtraction circuit 12.

The subtraction circuit 12 is connected to the signal input terminal 1
, The input video data supplied via the CPU, ie, the image data of the current frame, the image data of m × n pixels for each block and the representative point pixels of the corresponding block of the previous frame read from the representative point memory 11 are read out. A difference from the image data, that is, an absolute value of a difference between frames is detected. Then, frame difference absolute value data obtained as subtraction output data by the subtraction circuit 12 is supplied to the correlation integrated value table forming circuit 13.

The correlation integrated value table forming circuit 13 integrates the frame difference absolute value of each block obtained by the subtracting circuit 12 for each corresponding pixel over one frame period, and forms a pixel array of one block. A correlation integrated value table having corresponding m × n integer coordinates is formed. The correlation integrated value table formed by the correlation integrated value table forming circuit 13 is m × n
The distribution of the integrated value of the frame difference absolute values, that is, the correlation integrated value, is shown, and the integrated value of the correlation having the strongest frame correlation is the minimum value. Then, the correlation integrated value table forming circuit 1
3 are supplied to the motion vector estimating circuit 14.

The motion vector estimating circuit 14 detects the coordinates of the minimum value of the correlation integrated value in the correlation integrated value table formed by the correlation integrated value table forming circuit 13, and determines the coordinates of the minimum value as the representative point pixel. When the coordinates are other than the coordinates P (0,0), the motion of the image is calculated based on the correlation integrated value at the peripheral coordinates where the minimum coordinate of the correlation integrated value is located at the center and the minimum integrated value. Estimate the vector.

That is, in the motion vector estimating circuit 14 in this embodiment, as shown in the flow chart of FIG. 4, the correlation integrated value table formed by the correlation integrated value table forming circuit 13 in step 1 is correlated in step 2. The coordinates of the minimum value of the integrated value are detected. The coordinates of the minimum value of the correlation integrated value detected in step 2 are shown by integer coordinate values in the correlation integrated value table.

In the next step 3, a process for determining whether or not the coordinate of the minimum value of the correlation integrated value detected in the above step 2 is the coordinate P (0,0) corresponding to the representative point pixel. When the determination result is “NO”, that is, when the coordinates are other than the coordinates P (0, 0), the process proceeds to step 4 in order to perform motion vector estimation processing on the assumption that there is no image motion.

Here, the processing of estimating the motion vector of the image in the motion vector estimating circuit 14 is performed, for example, as shown in FIG. 5 by the coordinates P (x, y) of the minimum value of the correlation integrated value.
And its eight neighboring coordinates P (x + 1, y), P (x + 1, y +
1), P (x, y + 1), P (x-1, y + 1), P
(X-1, y), P (x-1, y-1), P (x, y-
1), P (x + 1, y-1) can be two-dimensionally separable using four axes S ₁ , S ₂ , S ₃ , S ₄ passing through two coordinates.

In step 4 above, the variable N is set to N =
In step 5, a determination process is performed to determine whether or not N = 4. If the determination result is NO, step 6 is performed.
Then, the process of estimating the coordinates of the minimum value of each axis is performed.

The one-dimensional estimation of the minimum value coordinate for a certain axis is performed, for example, as shown in FIG. 6 on the axis S ₁ , where the correlation integrated value of the central coordinate P (x, y) is the minimum value and the adjacent coordinate P Assuming that there is a difference between a and b between each correlation integrated value of (x + 1, y), P (x-1, y) and the minimum integrated value, the distance L between adjacent coordinates is a / (a + b). : B / (a +
This is performed by obtaining coordinates P (X, Y) that are proportionally divided in b). As a result, the minimum value coordinates can be obtained with a finer resolution than the integer coordinate values in the correlation integrated value table. By changing the weights α and β of the proportional coefficients for proportionally dividing the distance L between adjacent coordinates, such as α · a / (a + b): β · b / (a + b), the adjacent coordinates P (x
+1, y) and P (x-1, y) can also be estimated in accordance with the change of each correlation integrated value.

When the process of estimating the coordinate of the minimum value of one axis in step 6 is completed, the process proceeds to step 7 where the variable N is incremented (N = N + 1), and then the process returns to the determination process of step 4 above. Steps 5 to 7 are repeated. Then, the four axes S ₁ , S ₂ , S
_3, S ₄ performs estimation processing of the minimum value of the coordinates for, the process proceeds to step 8 if the decision result in the step 5 is to YES.

In this step 8, each axis S ₁ , S ₂ , S
_3, by combining the estimated coordinates of the minimum value of S ₄ to calculate the two-dimensional coordinate values, determining the motion vector of the image from the 2-dimensional coordinate values. Then, it is confirmed that the motion vector of the image determined in step 7 is within the estimated existence range, and the process of estimating the motion vector of the image is completed.

That is, in the image motion vector detecting device 10 having such a configuration, the correlation integrated value forming circuit 13 is used.
The integer coordinate value is obtained by detecting the coordinate of the minimum value of the correlation integrated value in the table correlation integrated value table formed by the above equation, and the integer coordinate of this minimum value is the coordinate P (0,
In the case of coordinates P (x, y) other than (0), it is represented by coordinates P (x, y) of the minimum value of the correlation integrated value with respect to coordinates P (0,0) corresponding to the representative point pixel. From the motion vector V _t = (x, y), the correlation integrated value at the peripheral coordinates P (x ± n, y ± n) where the coordinate P (x, y) is located at the center is calculated as the correlation integrated value and the minimum correlation. The motion vector V _t ′ = I _P (V _t ) of the image is estimated with high accuracy based on the integrated value. If the minimum integer coordinate is the coordinate P (x, y) of the coordinate P (0,0) corresponding to the representative point pixel, the motion vector V _t ′ = V _t = (0,0 ).

Then, the motion vector V _t 'detected by the motion vector detecting device 10 is supplied to the correction amount generator 20. The correction amount generating section 20 uses the motion vector V _t ′ detected by the motion vector detection device 10 as a camera shake vector to form a camera shake correction signal having a correction amount X _t of X _t = X _t−1 + V _t ′, The camera shake correction signal is supplied to the correction unit 30.

As shown in FIG. 7, for example, the correction section 30 includes an address control circuit 31 and a select signal generation circuit 32 to which a camera shake correction signal is supplied from the correction amount generation section 20, and an address control circuit 31. Memory 33 and peripheral memory 34 in which video data is written / read in accordance with an address signal supplied from
And the video data read from the frame memory 33 and the peripheral memory 34 are transferred to the select signal generating circuit 3.
And a selector 35 for selectively outputting in response to the select signal supplied from the selector 2.

Input video data supplied via the signal input terminal 1 is sequentially written into the frame memory 32. The read address of the frame memory 32 is controlled by the camera shake correction signal in accordance with the camera shake vector. As a result, video data obtained by moving one frame of input video data according to the camera shake vector is obtained from the frame memory 32. Then, the video data read from the frame memory 32 and the peripheral video data read from the peripheral memory 33 are combined by selection by the selector 35, and are output from the signal output terminal 2 as video data subjected to camera shake correction processing.

In the peripheral memory 33, video data of a peripheral portion corresponding to a correction range of an image by the video data subjected to the camera shake correction output via the selector 35 is sequentially written as peripheral video data.

In the camera shake correction apparatus having such a configuration, in the motion vector detecting apparatus 10, the minimum value of the correlation integrated value of the table integrated by the correlation integrated value table formed by the correlation integrated value forming circuit 13 is assumed to be the coordinate P (0). , 0), the motion vector V _t ′ is interpolated and estimated as described above. For example, if a motion vector V _t ′ having a decimal point or less as shown in FIG. correction amount as shown in FIG. 9 in accordance with the following motion vector V _t 'point X _t =
X _t-1 + image stabilization by image stabilization signal V _t 'is will be applied to the video data, so that the drift is generated in the image without camera shake. However, in the camera shake correction apparatus of this embodiment, in the motion vector detecting apparatus 10, the minimum value of the correlation integrated value of the table correlation integrated value table formed by the correlation integrated value forming circuit 13 is the coordinate P (0,0).
, The interpolation estimation of the motion vector V _t ′ as described above is not performed, so that the motion vector V _t ′ = indicating that there is no motion in the image as shown in FIG.
Since (0, 0) is output, the correction amount _Xt is also zero as shown in FIG. 11, and no drift occurs in the still image.

The minimum value of the correlation integrated value in the table correlation integrated value table formed by the correlation integrated value forming circuit 13 is the coordinate P.
For an image with camera shake located at coordinates P (x, y) other than (0,0), a highly accurate image having information of a decimal point or less obtained by interpolation estimation of the motion vector V _t ′ as described above is used. Since the motion vector V _t ′ is output from the motion vector detection circuit, high camera shake correction accuracy can be secured, and a natural image output can be obtained.

[0030]

As is apparent from the above description, according to the present invention, one frame is divided into a plurality of blocks, and each pixel data in the above-described block of the current frame and a representative of the previous frame corresponding to the block. The absolute value of the difference from the pixel data is detected, and the absolute value of the difference detected for each pixel in each block is integrated for each corresponding pixel over the one frame to form a pixel array for one block. A correlation integrated value table having the respective integrated results at corresponding coordinates is formed. The minimum value coordinate of the correlation integration value in the correlation integration value table is detected as the minimum value coordinate, and when the minimum value coordinate is different from the coordinate corresponding to the representative point pixel, the minimum correlation integration value of the minimum value coordinate is detected. And a value obtained by proportionally dividing the peripheral coordinates on the first axis by a difference value between each of the correlation integrated values corresponding to the peripheral coordinates of the minimum value coordinate on the first axis passing through the minimum value coordinate. The first coordinate on the first axis is obtained, and the minimum correlation integrated value of the minimum value coordinate and each correlation integrated value corresponding to coordinates around the minimum value coordinate on the second axis passing through the minimum value coordinate Calculating a second coordinate on the second axis proportionally divided from peripheral coordinates on the second axis by a difference value between the first coordinate and the second coordinate. The motion vector of the image is estimated by It is possible to detect. In addition, when the coordinates of the minimum value are the coordinates corresponding to the representative point pixel, the motion vector of the image is not estimated. Is not output. Then, a correction signal having a correction amount corresponding to the motion vector is formed, and the image forming the frame can be subjected to the vibration correction processing based on the correction signal.

Therefore, according to the present invention, it is possible to detect a motion vector of an image with high accuracy and to perform a vibration correction process on the image.
It is possible to perform high-performance camera shake correction without giving a drift to an image that should be stationary.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a camera shake correction apparatus provided with an image motion vector detection apparatus according to the present invention.

FIG. 2 is a diagram illustrating a state of block division of a screen in the motion vector detection device.

FIG. 3 is a diagram showing a structure of one block of the screen divided into blocks.

FIG. 4 is a flowchart showing a motion vector estimation processing procedure in the motion vector detection device.

FIG. 5 is a diagram for explaining an axis near a minimum value on a correlation integrated value table used for a motion vector estimation process in the motion vector detection device.

FIG. 6 is a diagram illustrating an example of one-dimensional estimation of a motion vector in the motion vector detection device.

FIG. 7 is a block diagram illustrating a configuration of a correction unit of the camera shake correction device provided with the motion vector detection device.

FIG. 8 is a diagram illustrating a motion vector having a decimal point or less obtained by an estimation process in a case where the coordinate of the minimum value of the correlation integrated value in the table correlation integrated value table is the coordinate corresponding to the representative point pixel. it is a diagram showing a V _t '.

9 shows a motion vector V having a decimal point or less shown in FIG.
FIG. 14 is a diagram illustrating a camera shake correction amount corresponding to _t ′.

FIG. 10 is a diagram illustrating a motion vector detection apparatus according to an embodiment of the present invention. In a case where the coordinates of the minimum correlation integrated value in the table correlation integrated value table are the coordinates corresponding to the representative point pixels, the motion vector V _t ′ is output. FIG.

FIG. 11 is a diagram showing a camera shake correction amount corresponding to the motion vector V _t ′ of a decimal point or less shown in FIG. 10;

[Explanation of symbols]

 Reference numeral 10: Motion vector detecting device 11: Representative point memory 12: Subtraction circuit 13: Correlation Integrated value table forming circuit 14 ... Motion vector estimating circuit

────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Ken Satoshi 6-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Tsukasa Hashino 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo (56) References JP-A-2-118888 (JP, A) JP-A-2-79585 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) H04N 5/225-5/247

Claims (2)

(57) [Claims] 1. A memory for storing image data of a representative point pixel for each block obtained by dividing an image of one frame constituted by an input video signal into a plurality of blocks, an image data of each pixel of a block of a current frame, and the memory Difference detecting means for detecting the absolute value of the difference between the image data of the representative point pixel of the block of the previous frame and the image data of each of the pixels, and the absolute value of the difference detected for each of the pixels in each of the blocks. , For each corresponding pixel over one frame,
Correlation integrated value table forming means for forming a correlation integrated value table having the respective integration results at coordinates corresponding to a pixel array of one block, and correlation integration of the correlation integrated value table formed by the correlation integrated value table forming means The coordinates of the minimum value of the value are detected as the minimum value coordinates, and when the minimum value coordinates are different from the coordinates corresponding to the representative point pixel, the minimum correlation integrated value of the minimum value coordinates and the second coordinate passing through the minimum value coordinates are detected. The first axis on the first axis, which is proportionally divided from the peripheral coordinates on the first axis by a difference value with each correlation integrated value corresponding to the coordinates around the minimum value coordinate on the first axis The coordinates are obtained, and the difference value between the minimum correlation integrated value of the minimum value coordinate and each correlation integrated value corresponding to coordinates around the minimum value coordinate on the second axis passing through the minimum value coordinate is used as the first value. This is proportionally divided from the surrounding coordinates on axis 2. Motion vector estimating means for obtaining a second coordinate on a second axis and further estimating a motion vector of an image by combining at least the first coordinate and the second coordinate; Correction amount generation means for forming a correction signal of a correction amount corresponding to the motion vector estimated by the above, and correction means for performing a vibration correction process on an image constituting the frame by a correction signal supplied from the correction amount generation means; An image vibration correction device comprising: 2. One frame is divided into a plurality of blocks,
Detecting the absolute value of the difference between each pixel data in the block of the current frame and the representative pixel data of the previous frame corresponding to the block; and detecting the absolute value of the difference detected for each pixel in the block. Is integrated for each corresponding pixel over the one frame,
Forming a correlation integrated value table having the respective integration results at coordinates corresponding to a pixel array of one block; and detecting a minimum value coordinate of the correlation integrated value of the correlation integrated value table as a minimum value coordinate. When the minimum coordinate is different from the coordinate corresponding to the representative point pixel, the minimum correlation integrated value of the minimum coordinate and the coordinates around the minimum coordinate on the first axis passing through the minimum coordinate. The first coordinate on the first axis, which is proportionally divided from the peripheral coordinates on the first axis, is calculated from the difference value with each correlation integrated value corresponding to the minimum integrated coordinate of the minimum coordinate. The value obtained by proportionally dividing the peripheral coordinates on the second axis by a difference value between the value and each correlation integrated value corresponding to the peripheral coordinates of the minimum coordinate on the second axis passing through the minimum coordinate. Second
Estimating a motion vector of an image by combining at least the first coordinates and the second coordinates; and calculating a correction signal of a correction amount according to the motion vector. An image vibration correction method, comprising: forming; and performing a vibration correction process on an image forming the frame based on a correction signal supplied from the correction amount generation unit.