JP3216147B2 - Image stabilization device for video data - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a camera shake correction device for correcting a camera shake contained in video data such as a photographing output of a handy type video camera.

[0002]

2. Description of the Related Art When shooting using a handy type video camera, there is a problem that the playback screen fluctuates due to camera shake. To solve this problem, we detect motion vectors,
A device that corrects video data stored in an image memory based on this motion vector has been proposed (for example, Japanese Patent Application Laid-Open No. 63-166370). The detection of the motion vector is performed by, for example, block matching. That is, the screen is divided into a number of areas (called blocks)
The absolute value of the frame difference between the representative point of the previous frame located at the center of each block and the pixel data in the block of the current frame is calculated, the absolute value of the frame difference is integrated for one screen, and the integrated frame difference data is calculated. The motion vector is detected from the position of the minimum value. Also, as described in this document, in order to prevent video data from being lost when motion is corrected, the image is enlarged by, for example, about 15% by controlling the reading of the image memory and the interpolation circuit. ing.

[0003]

The camera shake vector is detected from the movement between frames, and is not the same as the camera shake correction amount. For example, in the period of three consecutive frames,
When camera shake occurs in the same direction during the second and third frames, a camera shake vector V1 between the first frame and the next frame is detected, and the second and third frames are detected. A shake vector V2 between frames is detected. The correction amount for the second frame may be V1, but the correction amount for the third frame is (V1
+ V2). Therefore, the conventional camera shake correction apparatus requires a circuit for generating a correction amount obtained by integrating a motion vector. SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a video data camera shake correction apparatus which does not need to integrate a camera shake vector and convert it into a correction amount unlike the related art, and can achieve simplification of a circuit.

[0004]

According to the present invention, a single screen is copied.
Divided into a number of areas, and for each area,
Detects level difference with pixel data in each area of current frame
And the absolute value of the difference is accumulated for one screen.
Form difference data and calculate 1 for one screen from accumulated difference data
Detecting two motion vectors, detecting means for detecting a camera shake vector indicating the direction and amount of camera shake from the motion vector, and means for correcting camera shake based on the output signal of the detection means , the detection means, a memory for storing a representative point in the previous frame, at least and a subtraction means for subtracting the pixel data in each region of the data and the current frame of the representative points stored in the memory, the detected vector by the detection means When the camera shake vector is determined, the representative point is updated
Stops , and after that, two frames or more ago
Wherein the level difference is calculated using the data of the representative point of the video data.

[0005]

The motion vector is detected by block matching using the representative points. A camera shake vector is detected from the motion vector. When the camera shake vector is detected, updating of the memory storing the representative points is stopped. Therefore, the detected shake vectors after that correspond to the movement of two frames, three frames,..., Not the movement between the frames. Accordingly, the camera shake vector corresponds to the camera shake correction amount, and there is no need to integrate the camera shake vector to form a camera shake correction signal, and the circuit configuration can be simplified.

[0006]

An embodiment of the present invention will be described below with reference to the drawings. In the camera shake detection in one embodiment of the present invention, since the camera shake is a motion of the video camera, the entire screen moves, while the motion of the object is that a certain part of the screen moves, that is, the camera stops moving in the screen. Focusing on the fact that there is a part, there is a distinction between the two. In FIG. 1, reference numeral 1 denotes an input terminal for digital video data.
The video data is, for example, captured by a CCD of a video camera, and is a series of interlaced scans.

The input video data is supplied to the field memory 2 and to the representative point memory 3 through the terminal a of the switching circuit SW1. From the output of the field memory 2, data corrected for camera shake as described later is read. When processing is performed for each frame, a frame memory is used instead of the field memory 2.
The output of the representative point memory 3 is supplied to the subtraction circuit 4 and returned to itself through the input terminal b of the switching circuit SW1. Therefore, when the input terminal b of the switching circuit SW1 is selected, the updating of the representative point memory 3 is stopped, and the stored data of the representative point is held.

Further, the output of the representative point memory 3 is supplied to the representative point memory 5 through the input terminal c of the switching circuit SW2. The output of the representative point memory 5 is supplied to the subtraction circuit 6 and returned to itself through the input terminal d of the switching circuit SW2. Therefore, the switching circuit SW2
Is selected, the updating of the representative point memory 5 is stopped, and the stored data of the representative point is held. Input video data is supplied to the subtraction circuits 4 and 6. The representative point memory 3 stores data of a representative point one field before, and the representative point memory 5 stores data of a representative point two fields (ie, one frame) before.

As shown in FIG. 2, a screen of one field is subdivided into blocks of m pixels × n lines, and a spatially central pixel of each block is set as a representative point. The representative point is before
It is evenly distributed on the screen of the field . The subtraction circuit 4 detects a difference (that is, a field difference) between each of the m × n pixel data of a certain block in the current field and the data of the representative point of the block at the same position in the previous field. This field difference is supplied to the absolute value integration circuit 7. Similarly, the subtraction circuit 6 detects a frame difference between the current field block and the representative point two fields before. The frame difference is supplied to the absolute value integration circuit 8 and the frequency table creation circuit 9. Set the xy coordinates of the origin to (0,0)
Is defined as the coordinates (x, y) of the block. example
If the block size is (m = n = 25), the origin and
The position in the x direction is the same, and the position of the pixel at the top of the block is
The coordinates become (0, -12), and the origin and the position in the x direction are the same.
The coordinates of the position of the pixel at the lower end of the block are (0, +1)
2), the origin and the position in the y direction are the same,
The coordinates of the position of the rightmost pixel are (+12, 0), and the origin is
And the position in the y direction are the same, and the position of the pixel at the left end of the block
Are (−12, 0. In this way, in the block
(25 × 25) of all pixels are defined by xy coordinates
Is done. Then, the current frame (the k-th frame)
(25 × 25) pixel values Ik (-12, -12),
Ik (-11, -12), ..., Ik (0,0), ..., Ik (+ 12, + 12)
And spatially identical to the previous frame (the k-1st frame)
Each of the pixel values Ik-1 (0,0) of the representative point of the block at the position
The difference is calculated as a frame difference by the subtraction circuit 6.
You. The field difference is similarly obtained.

The motion vector is detected by an absolute value integrating circuit 8.
Is done using the frame difference. The movement between the fields obtained by the absolute value integration circuit 7 from the difference between the fields is used as auxiliary means. That is, the difference between the fields has a feature that the time accuracy is good but the position accuracy is bad, and the difference between the frames has the feature that the time accuracy is bad but the position accuracy is good. Therefore, when a motion vector is detected from a frame difference, the detection accuracy can be improved by using the inter-field difference.

As shown in FIG. 3, a motion search area has the same size as a block. For example, (m = n = 2
5). The subtraction circuit 6 generates m × n differences between the representative point two fields before and the pixel data in the block of the current field for each block. In the absolute value integration circuit 8, the frame difference (Ik-1 (0,0)) at each position on the xy coordinates in the block is calculated.
−I k (x, y)) is integrated over one frame period to form a distribution of m × n integrated frame difference data, where for one block of the current field,
The absolute value of the frame difference corresponds to the xy coordinates of the block
(25 × 25) are generated. Similarly, one field
Number of times equal to the number of blocks in the xy coordinates
, An absolute value of the frame difference is generated. This one
Field that occurs the number of times equal to the number of all blocks in the field.
The absolute value of the frame difference is accumulated (integrated) at each position on the xy coordinates
The distribution of accumulated frame difference data
Is done. The minimum value in this distribution is detected as a motion vector for one entire screen . That is, the position of the minimum value is
(X, y), the motion vector is V (x, y) (V
Indicates the size, (x, y) indicates the direction of movement)
Is detected. The detection of this motion vector is performed by the inventor of the present application.
It is disclosed in Japanese Patent Application Laid-Open No. 61-105178 previously proposed.
Can no Similar to what's.

The frequency difference table creation circuit 9 is supplied with the frame difference from the subtraction circuit 6. Frequency distribution table creation circuit 9
Is the search area of the above-mentioned motion, for example, the area of (25 × 25).
Corresponds to the frequency, and includes a memory with a capacity of appropriate depth direction at each position. And all the positions of a certain block
When the following conditions are satisfied for the pixel of, the frequency at that position is incremented , a frequency distribution table is created, and
To the final count by accumulating over the entire screen
Create a cloth table . Final frequency distribution table integrated frequency distribution table
Called. Ik-1 (0,0) ≒ Ik (0,0) and ｛Ik-1 (0,0) −Ik (x, y)｝ Absolute value ≧ Th

Here, Ik (0,0) is the value of the pixel data at the center of the block of the current field, and Ik-1 (0,0) is the value of the block at the same spatial position as the block of the current field. located in the center, one frame (2 fields) the previous value of the representative point data, Th is a representative point of the previous frame, the current off
Level with the pixel at (x, y) in the frame block
This is a threshold value for determining the difference in the distance. When the above condition is satisfied, the frequency f (x, y) in the frequency distribution table is incremented. The cumulative frequency distribution table shows each position (x, y)
Indicates the number of times that the condition of the above-described formula is satisfied. This frequency is the frequency f (x, y) of each position (x, y).
y). The process of creating this frequency distribution table is one block.
At each position (x, y).
The results obtained for all blocks are accumulated,
A number distribution table is required. This accumulation process is based on the motion described above.
Of integrated frame difference data when calculating vector
It is similar to forming a distribution. This condition is that there is no frame difference and the position (center position) of (0,0)
Between the position of (x, y) and the threshold Th or more
It means that there is a difference . The presence of the level difference means that a frame difference always occurs if there is motion. Conversely, if it is flat, even if there is movement, the frame
There is no difference, so in a flat case it is
Cannot be determined based on the frame difference.

The motion vector detected by the absolute value integration circuit 8 and the output of the frequency distribution table creation circuit 9 are supplied to the motion amount determination circuit 10. The motion amount determination circuit 10 verifies whether the motion vector can be used as the camera shake vector with reference to the integrated frequency distribution table created by the frequency distribution table creation circuit 9 . That is, as described above, in the case of the motion vector V (x, y), the frequency f of the position (x, y)
(X, y) is examined, and the frequency f (x, y) is compared with a threshold value. This frequency f (x, y) is expressed in the frame
Blocks with almost no difference, above the threshold
Since it corresponds to the number of pixels having a difference , the frequency f (x,
If y) is equal to or smaller than the threshold value for determining a camera shake vector, it is determined that a motion vector has occurred due to the camera shake, and the motion vector is determined to be a camera shake vector. Otherwise, the block has almost no frame difference.
That is, the number of pixels having a level difference equal to or larger than the threshold value exists to some extent, in other words, the motion indicated by the motion vector is the motion of a large-area object,
It is determined that the motion is not the motion of the entire screen (camera shake) caused by the motion of the camera. Note that the frequency distribution table creating circuit 9 uses a representative point as the center instead of creating an integrated frequency distribution table having the same size as the motion vector search area (3).
X3) An integrated frequency distribution table for an area of about pixels may be created.

The reason why the motion vector between the fields from the absolute value integration circuit 7 is supplied to the motion amount determination circuit 10 is that, as described above, the continuity of the detected motion vector is checked and the accuracy is determined. This is for improvement. Sand
The continuity is determined by detecting each motion vector in the time direction.
Indicates whether the value is fluttering. Stationarity
Is good for each detection of each motion vector in the time direction.
Means that the value does not flutter. Further, the motion amount determination circuit 10 generates a state signal indicating that a camera shake vector has been detected from the motion vector and the frequency distribution table. By this state signal, switching circuits SW1 and SW2
Is controlled.

If the state signal indicates that the camera shake vector has been detected, the switching circuit SW
1 selects the input terminal b and the switching circuit S
W2 selects the input terminal d. Otherwise, the switching circuits SW1 and SW2 are connected to the input terminals a and c.
Select each. In this way, when the camera shake vector is detected, the updating of the representative point memories 3 and 5 is stopped, and one of the frames in which the camera shake vector is first detected is stopped.
The representative point before the frame is stored. As a result, detection of a motion vector (camera shake vector) after the camera shake vector is detected is performed between two frames, between three frames,...
This is done using the absolute value of the difference signal. This means that the detected camera shake vector no longer indicates the camera shake during one frame, but is the camera shake correction amount itself.

The output (camera shake vector) of the motion amount determination circuit 10 is supplied to a correction amount generation circuit 11. As described above, since the camera shake vector corresponds to the camera shake correction amount, the correction amount generation circuit 11 does not need to integrate the camera shake vector, and the circuit is simplified. Therefore, providing the correction amount generation circuit 11 is not essential. However, when the camera shake correction is performed in units of fields as in this embodiment, the correction amount generation circuit 11 converts the amount of movement between frames into the amount of movement between fields. The motion vector is reduced to 1/2.

The camera shake correction signal from the correction amount generation circuit 11 is supplied to the address control circuit 12 and the select signal generation circuit 1.
3 is supplied. Address control circuit 12 generates an address signal for field memory 2 and peripheral memory 16. An image of one field is written in the field memory 2, and its read address is controlled according to the correction amount. Accordingly, from the field memory 2, video data obtained by moving one field of input video data according to the correction amount is obtained. The output of the field memory 2 is supplied to the selector 14, and the output of the selector 14 is taken out to the output terminal 15 and supplied to the peripheral memory 16. Peripheral data output from the peripheral memory 16 is supplied to the selector 14. The selector 14 selects the video data from the field memory 2 and the peripheral data stored in the peripheral memory 16 in which the camera shake has been corrected, in response to the select signal from the select signal generating circuit 13.

As shown in FIG. 4A, the field memory 2
Is stored in the peripheral memory 16 (the area outside the one-dot chain line) of the one-field image (the image frame is represented by 21) taken into the peripheral memory 16. The width of the peripheral portion 22 is set in consideration of the range of camera shake correction, and is set to, for example, a width of about 10 to 20%. In FIG. 4B, as shown by an image frame 21a, when an image to be at the position of FIG. 4A is moved to the drawing, for example, rightward due to camera shake, the entire image is moved to a position indicated by a broken line by the camera shake correction amount V. Will be corrected.
In such a case, the image of the portion 23 indicated by the diagonal line on the left side of the moved image is missing because the image does not originally exist in the captured image. The missing portion 23 is replaced with an image at a corresponding position stored in the peripheral memory 16. This replacement is performed by the address control by the address control circuit 12 and the switching operation of the selector 14. Further, peripheral image data other than the missing portion 23 in the captured video data is written to the peripheral memory 16, and the content of the peripheral memory 16 is updated.

In the present invention, the camera shake correction of the image pickup output of the video camera may be performed in real time, or the camera shake correction may be performed on the image recorded by the VTR. When correcting the imaging output of the video camera in real time, it is preferable to display a marker indicating that camera shake has occurred on the screen of the viewfinder.

As shown in FIG. 5A, an image frame 21 indicated by a broken line
Is an actually captured image, which has been corrected by the above-described camera shake correction as in an image 21b indicated by a solid line. In this case, as shown in FIG. 5B, a vertical line marker 25 is displayed at a position corresponding to the edge of the image frame 21 that is actually imaged in the screen of the viewfinder 24 of the video camera. The photographer looks at the marker 25 and knows the direction and amount of camera shake or the direction and amount of camera shake correction, and moves the video camera in the direction to correct the camera shake. In addition to the above-described camera shake correction, feedback based on the vision of the photographer is applied, so that it is possible to considerably prevent the range of the camera shake from exceeding the correction range. If the camera shake is not displayed on the viewfinder, the screen may suddenly move at a certain point when the camera shake cannot be corrected. The marker 25 is not limited to a vertical line, but may be an arrow or the like.

[0022]

According to the present invention, when a motion vector is determined as a camera shake vector, updating of previous pixel data at the time of detecting a difference between frames is stopped, and a camera shake vector detected thereafter is corrected. Can correspond with quantity. Therefore, there is no need to integrate the camera shake vector and convert it into a camera shake correction amount, and the circuit configuration can be simplified.

[Brief description of the drawings]

FIG. 1 is a block diagram of one embodiment of the present invention.

FIG. 2 is a schematic diagram illustrating block division according to an embodiment of the present invention.

FIG. 3 is a schematic diagram illustrating a search range of a motion vector according to an embodiment of the present invention.

FIG. 4 is a schematic diagram used for explaining camera shake correction according to an embodiment of the present invention.

FIG. 5 is a schematic diagram used to explain a camera shake display on a viewfinder screen of a video camera.

[Explanation of symbols]

Reference Signs List 1 Video data input terminal 2 Field memory 3 Representative point memory for storing representative point data one field before 5 Representative point memory for storing representative point data two fields before 9 Frequency distribution table creation circuit 10 Motion amount determination circuit SW1, SW2 switching circuit

Claims (1)

(57) [Claims] 1. One screen is divided into a plurality of areas, and each area is divided into
At the representative point of the previous frame and the
The level difference from the pixel data is detected, and the absolute value of the difference is
The accumulated difference data is formed by accumulating data for one screen.
From the accumulated difference data, one motion
Means for detecting a vector and detecting a shake vector indicating the direction and amount of the shake from the motion vector ;
And means for correcting the camera shake based on an output signal of said detecting means, said detecting means includes a memory for storing the representative point of the previous frame, the representative points stored in the memory data and the current At least subtraction means for subtracting the pixel data from each of the regions in the frame. When the vector detected by the detection means is determined to be a camera shake vector, the representative point is stopped from being updated.
After that , the data of the representative point two or more frames earlier
A camera shake correction device for video data, wherein the level difference is calculated using data.