JPH0787494A - Method and device for detecting motion vector - Google Patents

Abstract

PURPOSE:To reduce the arithmetic operation quantity and to preclude the possibility of mis-detection when a motion vector is detected by the block matching method. CONSTITUTION:A size of a motion vector detection object block is selected to be 16X16 and divided into small blocks in a size 4X4 by a small block processing circuit. A steady-state component is extracted by a circuit 7a for each small block comprising 4X4 picture elements and a transient component is extracted by a circuit 8a. The steady-state component and the transient component are extracted for each block as to data of a reference data. A difference of the steady-state component and the transient component is detected respectively and an evaluation value is formed by applying weight means to accumulated absolute differences. A motion vector is detected based on the evaluation value.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and method for detecting a motion vector representing the direction and amount of motion of an image by a block matching method.

[0002]

BACKGROUND OF THE INVENTION One application of motion vectors is motion compensation in predictive coding of digital image data. As an example, MPEG (Moving Picture Coding Experts Gros), which is an international standard method for high-efficiency encoding of moving images, is used.
up) method is proposed. This MPEG system is DC
It is a combination of T (Discrete Cosine Transform) and motion compensation predictive coding.

FIG. 8 shows an example of a motion compensation predictive coding apparatus. In FIG. 8, digital video data from the input terminal 61 is supplied to the motion vector detection circuit 62 and the subtraction circuit 63. In the motion vector detection circuit 63, the current frame and the reference frame (for example, the previous frame in time)
The motion vector between and is detected. This motion vector is supplied to the motion compensation circuit 64.

The image stored in the frame memory 65 is motion-compensated by the motion compensation circuit 64 based on the motion vector, and then supplied to the subtraction circuit 63 and the addition circuit 66. The subtraction circuit 63 subtracts the video data of the current frame and the decoded video data of the previous frame from the motion compensation circuit 64 for each pixel. The difference data from the subtraction circuit 63 is DCT-converted in the DCT circuit 67.
The coefficient data from the DCT circuit 67 is requantized by the quantization circuit 68. The output data of the quantization circuit 68 is taken out to the output terminal 69 and supplied to the inverse quantization circuit 70.

Inverse quantization circuit 70 and inverse D connected to it
The CT circuit 71 is the DCT circuit 67 and the quantization circuit 6
A local decoding circuit for performing processing opposite to that of No. 8 is constructed. The decoded difference data from the inverse DCT circuit 71 is supplied to the addition circuit 66. The output data of the adder circuit 66 is supplied to the motion compensation circuit 64 via the frame memory 65.
The decoded data of the previous frame from the motion compensation circuit 64 is supplied to the addition circuit 66 to form decoded data,
This decoded data is stored in the frame memory 65.

The motion vector detection circuit 62 detects a motion vector by the block matching method. In this method, a motion vector is obtained by moving an inspection block of a reference frame within a predetermined search range and detecting a block that best matches the standard block of the current frame.

In the block matching method, as shown in FIG. 9A, one image, for example, an image of one frame of horizontal H pixels and vertical V lines is P pixels × Q as shown in FIG. 9B.
It is subdivided into blocks of lines. In the example of FIG. 9B, P
= 5 and Q = 5. c is the central pixel position of the block.

FIG. 10 shows the positional relationship between a reference block whose center pixel is c and an inspection block whose center is c '. It is assumed that the reference block whose center pixel is c is a certain reference block of interest in the current frame, and the inspection block of the reference frame that matches the image is at the position of the block centered on c ′ in the reference frame. There is. In the block matching method, a motion vector is detected by finding an inspection block that best matches the reference block within the search range. In the case of FIG. 10A, +1 pixel in the horizontal direction and +1 line in the vertical direction,
That is, the motion vector of (+1, +1) is detected. In FIG. 10B, the motion vector of (+3, +3) is detected, and in FIG. 10C, the motion vector of (+2, −1) is detected. The motion vector is obtained for each reference block of the current frame.

Assuming that the range for searching the motion vector is ± S pixels in the horizontal direction and ± T lines in the vertical direction, the reference block is displaced ± S horizontally and ± T vertically from the center c thereof. It needs to be compared with the test block with center c '. FIG. 11 shows that when the position of the center c of a certain standard block in the current frame is set to R, it is necessary to compare (2S + 1) × (2T + 1) check blocks of the reference frame to be compared. That is, this FIG.
All the inspection blocks in which c'exists at the 1st cell position are comparison targets. FIG. 11 shows an example in which S = 4 and T = 3. Evaluation value obtained by comparison within the search range (that is, sum of absolute values of frame differences, sum of squares of this frame difference,
Alternatively, the motion vector is detected by detecting the minimum value in the sum of the absolute values of the frame differences (nth power).
The search range in FIG. 11 is a region where the center of the inspection block is located, and the size of the search range including the entire inspection block is (2S + P) × (2T + Q).

FIG. 12 shows an example of the configuration of a conventional motion vector detecting device. In FIG. 12, reference numeral 81 is an input terminal for the image data of the current frame, and this image data is stored in the current frame memory 83. Reference numeral 82 is an input terminal for the image data of the reference frame, and this image data is stored in the reference frame memory 84.

The reading / writing of the current frame memory 83 and the reference frame memory 84 is controlled by the controller 85. Pixel data of the standard block of the current frame is read from the current frame memory 83, and pixel data of the inspection block of the reference frame is read from the reference frame memory 84. An address moving circuit 86 is provided in association with the reference frame memory 84. As a result of the controller 85 controlling the address moving circuit 86, the center position of the inspection block is moved within the search range by one pixel step.

The output of the current frame memory 83 and the output of the reference frame memory 84 are supplied to the difference detection circuit 87,
The difference for each pixel is detected. The output of the difference detection circuit 87 is converted into an absolute value by the absolute value conversion circuit 88, and this absolute value is supplied to the accumulation circuit 89. The accumulator circuit 89 accumulates the absolute value difference generated in one block, and the output is the judgment circuit 9
Supplied to zero. The determination circuit 90 detects a motion vector from the sum of absolute values of the differences generated when the inspection block is moved within the search range. That is, the position of the inspection block that generates the minimum absolute difference sum is detected as a motion vector.

In the above-mentioned conventional block matching method, it is necessary to perform the processing for obtaining the sum of absolute values of frame differences between the reference block and the inspection block within the search range. In the examples of FIGS. 9, 10, and 11 described above, it is necessary to accumulate the absolute value difference (P × Q) times (2S + 1) × (2T + 1) times. From this relationship, the calculation amount is (P × Q) × (2S
It is represented by (+1) × (2T + 1). Therefore, the block matching method described above has a problem that the scale of hardware is large and the amount of calculation is enormous.

As a countermeasure, two types of methods have been proposed. One is a method of reducing the number of elements of the block, and the other is a method of simplifying the search.
As a method of reducing the number of elements, a method has been proposed in which the reference block and the inspection block are further divided into small blocks in the horizontal direction and the vertical direction, respectively, and the feature amount is extracted for each small block. That is,
Between the reference block and the inspection block, the horizontal small block feature amount and the vertical small block feature amount are compared separately, the absolute values of the comparison results are accumulated, and the weighted average of the accumulated results is calculated. It is used as the comparison result between blocks. The feature amount of a small block is, for example, a cumulative result of pixel data in the small block. This method can reduce the required operations on the total number of pixels in one block to the number of horizontal and vertical small blocks.

As a method for simplifying the search, when the inspection block is moved within the search range, the first step is to move the inspection block at intervals of several pixels to detect a rough motion vector and then In the step, there is known a method (two-step method) of finally obtaining a motion vector by moving the inspection block at intervals of one pixel in the vicinity of the detected position. A three-step method with three steps is also possible. According to this method, it is possible to reduce the number of calculations required for the full search, which corresponds to the total number of pixels in the search range, to the number of times corresponding to the number of pixels around the motion vector detected in each step. .

Further, as a motion vector detecting method intended to both reduce the number of elements and simplify the search, a method using a hierarchical structure has been proposed. One is what can be called the decimation method. That is, the number of pixels in the block is thinned out by subsampling (for example, 4 pixels).
Pixels are thinned to 1 pixel), block matching is performed on the block composed of the thinned pixels, then the origin is moved to the position of the detected minimum value, for example, 2 pixels are thinned to 1 pixel. Block matching is performed on the block structure, the origin is moved to the position of the detected minimum value, and finally the motion vector is detected by block matching of one pixel step. As a result of thinning, both the number of elements in the block and the number of operations in the search range are reduced.

Another that is intended to both reduce the number of elements and simplify the search is to use a low pass filter. The original image (referred to as the first layer) and the low-pass filter and subsampling from the first layer
Second with the number of pixels halved in the horizontal and vertical directions
The hierarchy and the second hierarchy are halved horizontally and vertically by a low-pass filter and subsampling.
It defines a hierarchical structure consisting of the third layer thinned out in the above. Then, block matching is performed on the third layer,
The origin is moved to the detected minimum value position to perform block matching for the second layer, and the origin is moved to the detected minimum value position to perform block matching for the first layer.

[0018]

The above-mentioned various improvements of the block matching can reduce the amount of calculation, but have a drawback that the motion vector is erroneously detected. This is because, as a result of simplification, the amount of information contained in the original image is lost.

More specifically, in the method of reducing the number of elements in a block, the feature amount of the small block is obtained by passing the image data of the small block through a low pass filter. In the method of simplifying the search, when the rough motion vector is detected, the accuracy is low, and thus the erroneous detection may occur. In the method of reducing the number of elements and simplifying the search, there is still a risk of erroneous detection when detecting a motion vector based on a decimated image or an image passed through a low-pass filter.

Therefore, an object of the present invention is to provide a motion vector detecting device and a detecting method in which the amount of calculation can be reduced and the hardware only has a simple feature, in which erroneous detection is prevented.

The present invention focuses on the fact that the activity of an image is necessary to correctly detect a motion vector by block matching. The activity means the level change or the amount of high frequency components. The conventional simplification method results in the loss of activity, and causes the problem of false detection by performing block matching based on the image with lost activity. This point will be described below using specific numerical values for easy understanding.

FIG. 3A shows an example of pixel values included in an area of (8 × 8) pixels. This value represents the level (0 to 255) of each 8-bit pixel. Where (2 ×
FIG. 3B shows the result of calculating the average value as the stationary component feature amount for each region (2 = 4 pixels), and FIG. 3C shows the result of calculating the standard deviation as the transient component feature amount for each region. In FIG. 3B, when the average value is viewed, adjacent average values “164” and “163” are extremely close values. However, the standard deviations corresponding to each are “0.5” and “7”, as shown in FIG. 3C, which are very different. This is because the degree of level change differs between the two (2 × 2) regions.

Next, FIG. 4A shows an area (8 × 8) in which the leftmost column in FIG. 3A is removed and a new column is newly added to the rightmost column. Similar to the above, the result of calculating the average value for this region is shown in FIG. 4B, and the result of calculating the standard deviation is shown in FIG. 4C. This FIG. 4B and FIG. 4C
Also in the above, the average values of the positions corresponding to “164” and “163” described above are “164” and “162”, and the standard deviations are “2” and “10.8”. Also in this example, it can be seen that the difference in standard deviation is larger than that of the average value.

Further, FIG. 5 shows an example of data in the area of (16 × 16) pixels. The result of calculating the average value for each (4 × 4) pixel is shown in FIG. 6, and the result of calculating the standard deviation is shown in FIG. As can be seen from FIGS. 6 and 7, even in this example, although the average value is the same as “145”, the standard deviations are different from “4.91” and “4.05”.

In this way, (2 × 2) or (4 ×
The local features of the image of 4) are
It cannot be expressed sufficiently and can be expressed accurately only when the transient component feature amount is also added. Since the block matching method examines the degree of image matching between the reference block and the inspection block, it makes sense to include the transient component as the feature amount of the block as in the present invention. There is.

[0026]

According to a first aspect of the present invention, there is provided a means for extracting image data of a reference block from first image data, and a means for indicating a stationary component of the block from the image data of the reference block. A unit for extracting a first transient component feature amount indicating a stationary component feature amount and a transient component, a unit for extracting image data of the inspection block from the second image data, and an image data of the inspection block Means for extracting a second stationary component characteristic amount indicating a stationary component and a second transient component characteristic amount indicating a transient component of the block, a comparison result of the first and second stationary component characteristic amounts, and a first And means for generating an evaluation value indicating the matching degree between the reference block and the inspection block from the comparison result of the second transient component feature amount, and moving the inspection block within a predetermined search range. Based on the made evaluation value, a motion vector detecting apparatus characterized by having a means for detecting a motion vector corresponding to the position of the inspection block that best matches the reference block.

According to a fifth aspect of the present invention, the step of extracting the image data of the reference block from the first image data, the first steady component characteristic amount indicating the steady component of the block from the image data of the reference block, and the transient First showing ingredients
Of extracting the transient component characteristic amount of the inspection block, extracting the image data of the inspection block from the second image data, and the second stationary component characteristic amount and the transient component indicating the stationary component of the block from the image data of the inspection block. Showing the second
From the step of extracting the transient component feature amount of, and the comparison result of the first and second stationary component feature amounts and the comparison result of the first and second transient component feature amounts, the matching degree between the reference block and the inspection block is determined. And a step of detecting a motion vector corresponding to the position of the inspection block that most matches the reference block, based on the evaluation value generated by moving the inspection block within a predetermined search range. It is a motion vector detection method characterized by consisting of

[0028]

The local feature of the block uses not only the stationary component but also the transient component. Using both of these,
Since the matching degree between the reference block and the inspection block is checked, the correct result can be obtained. Further, all the pixels in the block and all the survey points of the search range are significantly reduced, and the amount of calculation can be significantly reduced.

[0029]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of one embodiment. In FIG. 1, 1 is an input terminal for image data of a current frame, 2 is an input terminal for image data of a reference frame, and 3 is a current frame for storing image data of the current frame. Memory, 4
Is a reference frame memory for storing image data of the reference frame. The controller 5 controls the write / read operation of the current frame memory 3 and the reference frame.
Further, the address moving circuit 6 provided in association with the reference frame memory 4 moves the inspection block within the reference frame by the controller 5.

In order to extract the image feature amount for each block from the current frame memory 3, the stationary component extraction circuit 7a is used.
And the transient component extraction circuit 8a is connected. The stationary component extraction circuit 7b and the transient component extraction circuit 8b are also connected to the reference frame memory 4. Here, the stationary component is
Average value, low frequency component, low order component of coefficient obtained by orthogonal transformation, maximum value, minimum value, etc.Transient component means standard deviation, high frequency component, coefficient obtained by orthogonal transformation. It means a high-order component, a dynamic range, a difference with respect to the average value, a maximum value of the difference with respect to the average value, and the like. The transient component indicates the degree of local activity of the image. As described above, the simplification of the conventional block matching method uses only the stationary component as the feature amount of the block.

The target of block matching is not limited to the above-mentioned current frame and reference frame (frames preceding or following the current frame in time). For example, when detecting a motion vector between two still images,
Alternatively, the present invention can be applied to the case where a motion vector is detected between images having different resolutions.

The difference detection circuit 12a detects the difference value from the stationary component extraction circuits 7a and 7b. The difference detection circuit 12b detects the difference value from the transient component extraction circuits 8a and 8b. Absolute value conversion circuits 14a and 14b are connected to the difference detection circuits 12a and 12b, respectively. The absolute value conversion circuits 14a to 14e convert the difference value into an absolute value.

A weighted average circuit 15 is connected to the absolute value circuits 14a and 14b. The weighting averaging circuit 15 is supplied with the weighting factors w1 and w2 from the controller 5. The evaluation value from the weighted averaging circuit 15 is supplied to the determination circuit 18. The judgment circuit 18 detects the minimum value among the evaluation values calculated within the search range. The motion vector corresponding to the detected position of the minimum evaluation value is the determination circuit 1
8 to the output terminal 27. The controller 5 controls the judgment circuit 18, and the judgment result of the judgment circuit 18 is given to the controller 5.

FIG. 2 shows a more specific structure of the above-mentioned embodiment. That is, the averaging circuits 31a and 31b are provided as the stationary component extracting circuits 7a and 7b, and the standard deviation (σ) generating circuit 32 is provided as the transient component extracting circuits 8a and 8b.
a and 32b are provided. Difference detection circuit 12a, 1
2b is realized by the subtractors 35a and 35b.

The weighted averaging circuit 15 comprises multipliers 36a and 36b for multiplying the weighting factors w1 and w2, respectively, and an adder 37 for adding the multiplication outputs. The determination circuit 18 detects the weighted average data, that is, the memory 44 that stores the evaluation value and the minimum value of the evaluation values stored in the memory 44, and outputs the position of the minimum value as the motion vector. And a detection circuit 47. The determination circuit 18 may be configured without the memory 44.

The motion vector detecting operation of the above-described embodiment of the present invention will be described. As an example, the size of the motion vector detection target block (reference block and inspection block) is (4 × 4). (4 x
The value of each pixel data in the block of 4) is set to x1 to x16.

The averaging circuit 31a and the standard deviation generating circuit 32a generate the average value m and the standard deviation σ calculated for each (4 × 4 pixel) block as described below. m = (x1 + x2 + ... + x16) / 16 σ = {Σ (xi−m) ² × 1/16} ^1/2 where Σ is the value of each pixel minus the average value of the small blocks Means that (4 × 4) 16 pixels are aggregated. The average value obtained as described above for four small blocks included in one reference block is m
And the standard deviation is σ.

Further, for one inspection block of the reference frame, the average value m'and the standard deviation σ '(' indicates the average value and the standard deviation for the inspection block). Then, the subtracters 35a and 35b calculate the difference between the corresponding average values and the difference between the standard deviations, and the difference value is supplied to the weighted averaging circuit via the absolute value circuits 14a and 14b.

The position of the reference block is (x, y), and the position of the inspection block is (x + Δx, y + Δy) (where Δ
x is a 4-pixel step, and Δy is a 4-line step, the above-mentioned characteristic amount is expressed by the following equation. Hm (Δx, Δy) = | m ′ (x + Δx, y + Δy) −m (x, y) | Hσ (Δx, Δy) = | σ ′ (x + Δx, y + Δy) −σ (x, y) |

The weighted averaging circuit consisting of the multipliers 36a and 36b and the adder 37 generates the following evaluation value which is the weighted average of the above-mentioned characteristic amount. H (Δx, Δy) = w1 × Hm (Δx, Δy) + w2 × Hσ (Δx, Δy)

Evaluation values obtained when Δx and Δy are changed, that is, when the inspection small block is moved in steps of 4 pixels and 4 lines are stored in the memory 44. The minimum value detection circuit 47 detects the minimum value among the evaluation values stored in the memory 44. Δx and Δy corresponding to the position of this minimum value are the detected motion vectors.

As can be seen from the above description of the embodiment, the (16 × 16) pixels in the block are reduced to the average value and standard deviation, that is, two elements. Also, search is 4
Since it is done in pixel steps, it is simplified to 1/16. From both of these points, the amount of calculation can be greatly reduced compared to the conventional full search.

Various specific configurations of the present invention are possible other than the configurations of the above-described one embodiment and other embodiments. For example, by software processing, evaluation value creation,
You may make it detect a motion vector.

Further, when obtaining the motion vector, the precision of the motion vector may be half-pel precision instead of one pixel.

[0045]

According to the present invention, a motion vector can be detected with a small amount of calculation. In particular, in the present invention, not only the stationary component of the original image data but also the transient component is used, so that the motion vector can be correctly detected. Therefore, the risk of erroneous detection can be reduced.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention.

FIG. 2 is a block diagram more specifically showing a configuration of an embodiment of the present invention.

FIG. 3 is a schematic diagram for explaining the motion vector detection of the present invention using an example of image data.

FIG. 4 is a schematic diagram for explaining the motion vector detection of the present invention using an example of image data.

FIG. 5 is a schematic diagram for explaining the motion vector detection of the present invention using another example of image data.

FIG. 6 is a schematic diagram for explaining the motion vector detection of the present invention using another example of image data.

FIG. 7 is a schematic diagram for explaining the motion vector detection of the present invention using another example of image data.

FIG. 8 is a block diagram showing an example of a motion compensation predictive coding device to which the present invention can be applied.

FIG. 9 is a schematic diagram for explaining a motion vector detection method by a conventional block matching method.

FIG. 10 is a schematic diagram for explaining a motion vector detection method by a conventional block matching method.

FIG. 11 is a schematic diagram for explaining a search range in a motion vector detection method by a conventional block matching method.

FIG. 12 is a block diagram of an example of a motion vector detection device using a conventional block matching method.

[Explanation of symbols]

 3 Current frame memory 4 Reference frame memory 7a, 7b Steady component extraction circuit 8a, 8b Transient component extraction circuit 15 Weighted averaging circuit 18 Judgment circuit

Claims (5)

[Claims] 1. A means for extracting image data of a reference block from first image data, and a first component showing a stationary component of the block from the image data of the reference block, and a first component showing a transient component. 1
Means for extracting the transient component feature amount of, the means for extracting the image data of the inspection block from the second image data, and the second stationary component indicating the stationary component of the block from the image data of the inspection block. The second which shows the characteristic quantity and the transient component
From the comparison result of the first and second stationary component feature quantities and the comparison result of the first and second transient component feature quantities, the reference block and the Means for generating an evaluation value indicating the matching degree of the inspection block, and the inspection that best matches the reference block based on the evaluation value generated by moving the inspection block within a predetermined search range A motion vector detecting device comprising: a unit for detecting a motion vector corresponding to the position of a block. 2. The motion vector detection device according to claim 1, wherein the stationary component feature amount is an average value of pixel data in a block, a maximum value or a minimum value of the pixel data, and a low frequency component of the pixel data. , Or a device that is a low-order component formed by orthogonal transformation of the pixel data. 3. The motion vector detection device according to claim 1, wherein the transient component feature amount is a standard deviation of pixel data in a block, a dynamic range of the pixel data, a high frequency component of the pixel data, or the above. A device that is a higher order component formed by orthogonal transformation of pixel data. 4. A first memory for storing first image data, for outputting image data of a reference block, and a first average value indicating a stationary component of the block from the image data of the reference block. From the image data of the inspection block, the calculating means for generating the first standard deviation indicating the transient component, the second memory for storing the second image data, the second memory for outputting the image data of the inspection block, Computation means for generating a second average value indicating a steady component of the block and a second standard deviation indicating a transient component, a comparison result of the first and second average values, and the first and second Means for generating an evaluation value indicating the degree of coincidence between the reference block and the inspection block by weighted averaging the comparison results of two standard deviations; and the inspection block within a predetermined search range. When moving,
A motion vector detecting device, comprising: a means for detecting a motion vector corresponding to the position of the inspection block that generates the minimum evaluation value. 5. A step of extracting image data of a reference block from the first image data, and a first stationary component characteristic amount indicating a stationary component of the block from the image data of the reference block and a first indicating a transient component.
To extract the image data of the inspection block from the second image data, and to extract the image data of the inspection block from the image data of the inspection block. Second showing ingredients
The step of extracting the transient component feature amount, the comparison result of the first and second stationary component feature amounts and the comparison result of the first and second transient component feature amounts, the reference block and the inspection block. And a position of the inspection block that most matches the reference block, based on the evaluation value generated by moving the inspection block within a predetermined search range. And a step of detecting a corresponding motion vector.