JP3060433B2 - Motion vector calculation method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the calculation of a motion vector from a moving image.

[0002]

2. Description of the Related Art Many studies have been published on motion vector calculation methods. These calculation methods are the gradient method
(BKHone and BGSchunck, “Determing optical flo
w ”, Artificial Intelligence, Vol. 17, pp. 185-203 (188
1)) are roughly divided into the Fourier transform method and the matching method.
The matching method has been used for stereo matching of still images for the purpose of extracting three-dimensional information. Recently, the matching method has been used for extracting motion information from a real-time moving image. I have. As a method roughly classified into a matching method, a block matching method is mainly used from a practical viewpoint. For example, the motion compensation calculation method of the image coding method in the image communication is generally the block matching.

In the block matching method, M (u, v) that minimizes the following evaluation function is a motion vector representing this block.

[0004]

(Equation 1)

Here, ft (xi, yj) represents the luminance level at the screen coordinates (xi, yj) of the (t + k) th frame. Similarly, ft + k (xi, yj) represents the luminance level at the screen coordinates (xi, yj) of the t + k-th frame. n and m represent the spatial resolution of the motion vector, and n × m
Represents the block size. k represents time resolution, and a ×
b represents a search range. The function represented by min (A) is a function that returns (u, v) having the minimum value of A in the search range a × b.

[0006] In this conventional block matching method, only a method that brings in spatial resolution has been reported (RYWong and ELHall, "Sequential Hier
archical Scene Mathing ”, IEEE Trans.Comput., Vol.C-
27, No. 4, pp. 359-366 (1978)). This method achieves high accuracy and high efficiency by introducing the concept of coarse-to-fine into a variable spatial resolution.

[0007]

However, this method is
Regarding still image matching, it was necessary and sufficient to change the spatial resolution. On the other hand, in the case of the accumulated moving image, when the camera operation that captured the moving image is moving slowly, calculating the motion vector using the conventional spatio-temporal resolution fixed block matching method, If the spatial resolution is not appropriate, quantization errors are seen.

An object of the present invention is to provide a method of calculating a motion vector by a block matching method of an accumulated moving image, in which a variable spatial resolution is changed to a variable temporal resolution in order to achieve high accuracy and high efficiency of the calculation. To make it possible to cope with a slow movement, and furthermore, to optimize a search range of a block to be matched, thereby making it possible to cope with a fast movement.

[0009]

According to the present invention, there is provided a method of calculating a motion vector using block-correlation image correlation, wherein a spatial resolution, a time resolution, and a search range of block matching of the motion vector calculation method are variable. This is a motion vector calculation method.

[0010]

According to the above means, high-precision,
A motion vector can be calculated with high efficiency.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings.

According to the present invention, in a motion vector calculating method using block-divided image correlation using a computer, a spatial resolution, a temporal resolution, and a block matching search range of the motion vector calculating method are made variable. First,
A method for making each variable will be described.

(1) As a method for making the spatial resolutions n and m variable, a conventionally used method is used. That is, the calculation according to the above [Equation 1] is first performed with a coarse block size. As a result, if a favorable result does not appear, the calculation is performed hierarchically by sequentially reducing the block size.

(2) A method of making the temporal resolution k variable will be described with reference to FIGS. 1 and 2
3 conceptually shows a frame of a moving image. In FIG. 2, a frame 1 is divided into four blocks 5 to 8, and the target of each block is a block 5 having no motion,
It is assumed that there is a block 6 having a small motion, a block 7 having a medium motion, and a block 8 having a large motion.

With respect to such a moving image, the above [Equation 1]
In the calculation of, as shown in FIG. 1, the leftmost frame is defined as a t-th frame 1 and the t-th frames 1 and 3
The calculation of [Equation 1] is performed between the (t + 3) th frame 4 after the frame. As a result, the temporal resolution of the block 5 with no motion and the block 6 with little motion are coarse, so that good results are obtained, but good results are not obtained for the blocks 7 and 8. Therefore, for blocks 7 and 8, the calculation is performed again for the (t + 2) th frame 3 two frames later. Then, a good result can be obtained for the block 7 because the intermediate motion has an intermediate temporal resolution with respect to the intermediate motion.
Good results cannot be obtained for block 8. Therefore, the calculation is performed on the block 8 with respect to the (t + 1) th frame 2 after one frame. As a result, since a fine temporal resolution is taken for a large motion, a good result is obtained also for the block 8, and the calculation for all the blocks in the t-th frame 1 is completed. Hereinafter, the same calculation is sequentially performed for the next frame. However, for the next (t + 1) -th frame 2, the calculation has been completed for the blocks 5 to 8, so the calculation is performed only for the block 8. For the (t + 2) th frame 3, it is needless to say that the calculation is performed for the block 8, but the block 7 also needs to be newly calculated. Hereinafter, similar calculations are sequentially performed.

(3) Next, a method of making the ranges a and b in which the image correlation of the block is searched variable will be described. When making the search range variable, the search range is increased when the motion is large, and the search range is reduced when the motion is small. Therefore, it is sufficient to first reduce the search range, sequentially increase the search range, and continue the calculation until a good result is obtained.

In the above, the method of varying each of the spatial resolution, the temporal resolution, and the range in which the image correlation of the block is searched has been described. Next, how to apply each of the above methods will be described in detail.

First, how the spatiotemporal resolution k is made variable will be described. If the camera operation moves slowly, the motion vector is calculated using the conventional spatio-temporal resolution block matching method. If the spatio-temporal resolution is not appropriate for the motion, a quantization error is detected. Can be This quantization error occurs when the motion is extremely small compared to the spatio-temporal resolution. This quantization error causes the spread of the primary statistic at the time of global motion vector separation and the variation in the feature space due to the Hough transform of the motion vector.

This leaves a problem in the extraction of the feature amount from each feature space by the primary statistic and the Hough transform. When the camera operation includes a zoom operation, the spread and the variation are remarkably observed. The primary statistic relating to the distance of a motion vector by a zoom operation without quantization error has a value only at a position where the distance is zero. However, when a quantization error is included, the first order statistics have a spread. This spreading is shown in FIG. 3a. The image used was CCITT S
It is a test image of a GXV image coding expert meeting. Similarly, the distribution in the feature space by the Hough transform when there is no quantization error in the motion vector is a straight line with a distance of 0, but when the quantization error is included, the distribution shows variation. This variation is shown in FIG. 3b.

As a method of avoiding this, in the use of the spatio-temporal resolution variable block matching in the calculation of the motion vector, the decrease in the spatial resolution does not increase the information amount.
On the other hand, a decrease in temporal resolution leads to an increase in the amount of information.
FIGS. 4A and 4B show the spread and the variation distribution when the time resolution is reduced from k = 1 to k = 3, respectively. The decrease in the spread and the variation distribution indicates that small motions are reliably detected and the calculated motion vectors do not overlap in space and time.

Next, a method of making the ranges a and b in which the image correlation of a block is searched variable will be described. Conversely, when the motion is large, the current TV system uses 30 frames per second, and therefore the time resolution k cannot be further increased. For this reason, when using a block matching method with a fixed search range, when a person looks at a moving image, it is difficult to capture fast-moving objects and objects clearly appearing at a maximum time resolution of 30 images per second. Happens. In order to cope with this, the search range is made variable, and when the motion is large, the search range of the block matching is expanded to be able to cope with the human sense.

Here, for the sake of simplicity, the above-described method has been described as a method of sequentially expanding the method corresponding to a slow movement to a violent movement.
By combining the two methods, compared to the conventional simple fixed search range and fixed time resolution vector calculation method,
It is possible to calculate a vector in a wider range.

For example, in order to capture an extremely slow motion, block matching is performed with a coarse time resolution and a coarse block size, hierarchical matching is sequentially performed with a finer block size, and for an object having a large motion, , The time resolution is made hierarchically finer. In addition, a hierarchical method of increasing the matching search range for intense movements can be used to deal with all kinds of movements.

One example of a method for combining two of the spatial resolution, the temporal resolution, and the search range is shown in FIG.
This will be described with reference to FIGS.

Now, the spatial resolution is S, the temporal resolution is T,
Assuming that the search range is W, S = f
(T), T = g (W), and S = h (W). Examples of these functions f (T), g (W), and h (W) are shown in the graphs of FIG. 5, FIG. 6, and FIG.

FIG. 5 shows a method in which the spatial resolution S and the temporal resolution T are combined. First, make a rough setting for both, and if good results are not obtained,
First, the calculation is performed with a finer time resolution, and then the time resolution T is returned to a coarse value, and the calculation is performed with a finer spatial resolution S. Hereinafter, each resolution is alternately made finer in accordance with the arrows in the graph, and the calculation is continued until a good result is obtained.

FIG. 6 shows the relationship between the time resolution T and the search range W. The search range W is sequentially reduced in the direction of increasing the search range W from a combination in which the search range W is small and the time resolution T is coarse.
This shows that the calculation is performed by changing the time resolution T in the fine direction.

FIG. 7 shows an example of the relationship between the spatial resolution S and the search range W. The search range W is sequentially reduced in the direction from the combination in which the search range W is large and the time resolution T is coarse. 2 shows that the calculation is performed by changing the time resolution T in a fine direction.

FIGS. 5 to 7 show two combinations,
In each case, the other one has a fixed resolution. In the case of three combinations, it is possible to change the remaining one resolution sequentially.

[0030]

According to the present invention described above, a motion vector can be calculated with high accuracy and high efficiency from an accumulated moving image. Further, according to this method, (1) it is possible to reliably capture a small change. (2) The calculated motion vectors do not overlap in space and time. (3) Since the calculated motion vector is hierarchical, it is effective when indexing a moving image.

[Brief description of the drawings]

FIG. 1 is a conceptual diagram illustrating variable temporal resolution according to the present invention.

FIG. 2 is a conceptual diagram illustrating a temporal resolution variable according to the present invention.

FIG. 3 is a diagram illustrating a first-order statistic and a feature space according to a conventional example.

FIG. 4 is a diagram showing a first-order statistic and a feature space according to the present invention.

FIG. 5 is a view for explaining a method of combining spatial resolution and temporal resolution according to the present invention.

FIG. 6 is a view for explaining a method of combining a time resolution and a search range according to the present invention.

FIG. 7 is a view for explaining a method of combining a spatial resolution and a search range according to the present invention.

[Explanation of symbols]

1, 2, 3, 4,... Moving image frames, 5, 6, 7, 8,.
Block in frame.

──────────────────────────────────────────────────続 き Continued on front page (58) Field surveyed (Int.Cl. ⁷ , DB name) G06T 7/20 H04N 5/232 H04N 7/32

Claims (2)

(57) [Claims] 1. A method for calculating a motion vector using block-correlation image correlation, wherein two or three of a range for searching for a spatial resolution, a time resolution, and a block image correlation in the motion vector calculation method are variable. A motion vector calculation method characterized by using: 2. A method for calculating a motion vector using block-correlation image correlation, wherein a time resolution or an image correlation of a block in a search range of a spatial resolution, a time resolution, and a block image correlation of the motion vector calculation method is determined. A motion vector calculation method characterized by using any one of search ranges variably.