JP3277417B2 - Apparatus and method for detecting motion vector - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and a method for detecting a motion vector representing the direction and amount of motion of an image by a block matching method.

[0002]

2. Description of the Related Art One application of motion vectors is in motion compensation in predictive coding of digital image data. As an example, MPEG (Moving Picture Coding Experts Groove), an international standard for efficient coding of moving images, is used.
up) A method has been proposed. This MPEG method uses DC
This is a combination of T (Discrete Cosine Transform) and motion compensation prediction coding.

FIG. 8 shows an example of a motion compensation predictive coding apparatus. 8, digital video data from an input terminal 61 is supplied to a motion vector detection circuit 62 and a subtraction circuit 63. In the motion vector detection circuit 63, the current frame and the reference frame (for example, a temporally previous frame)
Is detected. This motion vector is supplied to the motion compensation circuit 64.

[0004] After the image stored in the frame memory 65 is motion-compensated by the motion compensation circuit 64 based on the motion vector, it is supplied to a subtraction circuit 63 and an addition circuit 66. The subtraction circuit 63 subtracts the video data of the current frame and the decoded video data of the previous frame from the motion compensation circuit 64 for each pixel. The difference data from the subtraction circuit 63 is DCT-transformed by the DCT circuit 67.
The coefficient data from the DCT circuit 67 is requantized by the quantization circuit 68. Output data of the quantization circuit 68 is taken out to an output terminal 69 and supplied to an inverse quantization circuit 70.

The inverse quantization circuit 70 and the inverse D connected thereto
The CT circuit 71 includes a DCT circuit 67 and a quantization circuit 6
A local decoding circuit for performing a process opposite to that of FIG. The decoded difference data from the inverse DCT circuit 71 is supplied to the adding circuit 66. The output data of the adding circuit 66 is supplied to the motion compensation circuit 64 via the frame memory 65.
The decoded data of the previous frame from the motion compensation circuit 64 is supplied to the addition circuit 66, thereby forming decoded data.
This decoded data is stored in the frame memory 65.

The motion vector detection circuit 62 detects a motion vector by a block matching method. In this method, a motion vector is obtained by moving an inspection block of a reference frame within a predetermined search range and detecting a block that most closely matches a reference block of the current frame.

In the block matching method, as shown in FIG. 9A, one image, for example, an image of one frame of horizontal H pixels and vertical V lines is composed of P pixels × Q as shown in FIG. 9B.
Subdivided into blocks of lines. In the example of FIG. 9B, P
= 5, Q = 5. c is the center pixel position of the block.

FIG. 10 shows the positional relationship between a reference block whose center is c and an inspection block whose center is c '. The reference block whose center pixel is c is a certain reference block of interest in the current frame, and it is assumed that the inspection block of the reference frame that matches the image is located at the position of the block centered on c ′ in the reference frame. I have. In the block matching method, a motion vector is detected by finding an inspection block that best matches a reference block within a search range. In the case of FIG. 10A, +1 pixel in the horizontal direction, +1 line in the vertical direction,
That is, a motion vector of (+1, +1) is detected. In FIG. 10B, a (+3, +3) motion vector is detected, and in FIG. 10C, a (+2, -1) motion vector is detected. The motion vector is obtained for each reference block of the current frame.

Assuming that the range for searching for a motion vector is ± S pixels in the horizontal direction and ± T lines in the vertical direction, the reference block is shifted ± S horizontally and ± T vertically from its center c. It needs to be compared to a test block with center c '. FIG. 11 shows that when the position of the center c of a certain reference block of the current frame is R, comparison with (2S + 1) × (2T + 1) check blocks of a reference frame to be compared is necessary. That is, FIG.
All the test blocks in which c 'is present at the position of the first square are comparison targets. FIG. 11 is an example in which S = 4 and T = 3. Evaluation values obtained by comparison within the search range (ie, sum of absolute values of frame differences, sum of squares of the frame differences,
Alternatively, the motion vector is detected by detecting the minimum value among the sums of the absolute values of the frame differences.
The search range in FIG. 11 is an area where the center of the test block is located, and the size of the search range including the entire test block is (2S + P) × (2T + Q).

FIG. 12 shows an example of the configuration of a conventional motion vector detecting device. In FIG. 12, reference numeral 81 denotes an input terminal for image data of the current frame, and this image data is stored in the current frame memory 83. Reference numeral 82 denotes an input terminal for image data of a reference frame, and this image data is stored in a reference frame memory 84.

The reading / writing of the current frame memory 83 and the reference frame memory 84 is controlled by a controller 85. From the current frame memory 83, the pixel data of the reference block of the current frame is read, and from the reference frame memory 84, the pixel data of the inspection block of the reference frame is read. An address moving circuit 86 is provided in association with the reference frame memory 84. As a result of the controller 85 controlling the address moving circuit 86, the center position of the test block is moved within the search range in one-pixel steps.

The output of the current frame memory 83 and the output of the reference frame memory 84 are supplied to a difference detection circuit 87,
A difference for each pixel is detected. The output of the difference detection circuit 87 is converted into an absolute value by an absolute value conversion circuit 88, and this absolute value is supplied to an accumulation circuit 89. An accumulating circuit 89 accumulates the absolute value difference generated in one block, and its output is determined by a judging circuit 9.
0 is supplied. The determination circuit 90 detects a motion vector from the sum of absolute values of differences generated when the inspection block is moved within the search range. That is, the position of the test block that generates the absolute value sum of the minimum difference is detected as a motion vector.

In the above-described conventional block matching method, it is necessary to perform a process for obtaining the sum of absolute values of frame differences between a reference block and an inspection block within a search range. In the examples of FIGS. 9, 10 and 11, it is necessary to accumulate (P × Q) absolute value differences (2S + 1) × (2T + 1) times. From this relationship, the calculation amount is (P × Q) × (2S
+1) × (2T + 1). Therefore, the above-described block matching method has a problem that the scale of hardware is large and the amount of calculation is enormous.

As a countermeasure, two types of methods have been proposed. One is a technique for reducing the number of elements in a block, and the other is a technique for simplifying a search.
As a method of reducing the number of elements, a method has been proposed in which a reference block and an inspection block are further divided into small blocks in each of the horizontal direction and the vertical direction, and a feature amount is extracted for each small block. That is,
The feature value of the horizontal small block and the feature value of the vertical small block are separately compared between the reference block and the inspection block, the absolute values of the comparison results are respectively accumulated, and the weighted average of the accumulation results is calculated. Used as a comparison result between blocks. The feature amount of a small block is, for example, an accumulation result of pixel data in the small block. This method can reduce the operations required for the total number of pixels in one block to the number of small blocks in the horizontal and vertical directions.

As a method of simplifying the search, when the inspection block is moved within the search range, the first step is to move the inspection block at intervals of several pixels, thereby detecting a rough motion vector, and There is known a method (two-step method) of finally obtaining a motion vector by moving an inspection block at intervals of one pixel in the vicinity of a detected position in a step. A three-step method with three steps is also conceivable. According to this method, the number of calculations corresponding to the total number of pixels in the search range required for the full search can be reduced to the number of times corresponding to the number of pixels around the motion vector detected in each step. .

Furthermore, a method employing a hierarchical structure has been proposed as a motion vector detection method intended to both reduce the number of elements and simplify search. One of them can be called a thinning method. That is, the number of pixels in the block is thinned out by subsampling (for example, 4
Pixel is thinned to one pixel), block matching is performed on a block composed of the pixels resulting from the thinning, and then the origin is moved to the position of the detected minimum value, for example, two pixels are thinned to one pixel. Block matching is performed on the block structure thus obtained, the origin is moved to the position of the detected minimum value, and finally a motion vector is detected by one pixel step block matching. As a result of the thinning, both the number of elements in the block and the number of operations in the search range are reduced.

Another use of a low-pass filter is to both reduce the number of elements and simplify the search. From the original image (referred to as the first layer) and the low-pass filter and sub-sampling from the first layer,
The second in which the number of pixels is thinned in half in the horizontal and vertical directions
Layer and the second layer by a low-pass filter and sub-sampling by half in the horizontal and vertical directions.
Is defined as a third-layer structure thinned out. Then, block matching is performed on the third hierarchy,
The origin is moved to the position of the detected minimum value to perform block matching on the second hierarchy, and the origin is moved to the position of the detected minimum value to perform block matching on the first hierarchy.

[0018]

The above-described various improvements in block matching can reduce the amount of computation, but have the disadvantage of causing erroneous detection of motion vectors. This is because the simplification results in a loss of the information amount of the original image.

More specifically, in the method of reducing the number of elements in a block, the feature amount of the small block is obtained by passing the image data of the small block through a low-pass filter. In the method of simplifying the search, when a rough motion vector is detected, there is a risk of erroneous detection due to low accuracy. In the method of reducing the number of elements and simplifying the search, erroneous detection may occur when a motion vector is detected based on a thinned image or an image passed through a low-pass filter.

Accordingly, it is an object of the present invention to provide a motion vector detecting apparatus and a detecting method in which an erroneous detection is prevented, in which the amount of calculation can be reduced and the hardware has only simple features.

The present invention focuses on the necessity of image activity in order to correctly detect a motion vector by block matching. The activity means a level change or an amount of a high frequency component. The conventional simplification technique has resulted in a loss of activity, and has caused a problem of false detection by performing block matching based on the image with lost activity. This point will be described below using specific numerical values for easy understanding.

FIG. 3A shows an example of pixel values included in an (8 × 8) pixel area. This value indicates the level (0 to 255) of each pixel of 8 bits. Here, (2 ×
FIG. 3B shows the result of calculating the average value as the steady component feature value for each region of 2 = 4 pixels), and FIG. 3C shows the result of calculating the standard deviation as the transient component feature value for each region. In FIG. 3B, when looking at the average value, adjacent average values “164” and “163” are extremely close values. However, as shown in FIG. 3C, the corresponding standard deviations are “0.5” and “7”, which are greatly different. This is because the degree of the level change is different between the two (2 × 2) areas.

Next, FIG. 4A shows a (8 × 8) region in which the leftmost column in FIG. 3A is removed and a new column is added to the rightmost column. For this region, the result of calculating the average value is shown in FIG. 4B, and the result of calculating the standard deviation is shown in FIG. 4C, as described above. 4B and 4C
Also, the average values of the positions corresponding to “164” and “163” are “164” and “162”, and the standard deviations are “2” and “10.8”. Also in this example, it can be seen that the difference of the standard deviation is larger than that of the average value.

FIG. 5 shows an example of data of a region of (16 × 16) pixels. FIG. 6 shows the result of calculating the average value for each (4 × 4) pixel, and FIG. 7 shows the result of calculating the standard deviation. As can be seen from FIGS. 6 and 7, also in this example, although the average value is the same as “145”, the standard deviation is different from “4.91” and “4.05”.

Thus, (2 × 2) or (4 ×
The local feature of the image of 4) can be obtained by using only the stationary component feature amount.
It cannot be sufficiently expressed, and can be accurately expressed only by taking into account the transient component feature amount. Since the block matching method is to check the degree of matching of an image between a reference block and an inspection block, it is reasonable to include a transient component as a feature value of the block as in the present invention. I have.

[0026]

According to a first aspect of the present invention, there is provided a means for extracting image data of a reference block from first image data, and a method for extracting a stationary component of the block from the image data of the reference block. A means for extracting a first transient component feature quantity indicating a steady component feature quantity and a transient component, a means for extracting image data of an inspection block from the second image data, and a means for extracting image data of the inspection block. Means for extracting a second steady-state component feature amount indicating a steady-state component of the block and a second transient component feature amount indicating a transient component; a comparison result between the first and second steady-state feature amounts; Means for generating an evaluation value indicating the degree of coincidence between the reference block and the test block from the comparison result of the second transient component feature quantity, and moving the test block within a predetermined search range. Based on the made evaluation value, a motion vector detecting apparatus characterized by having a means for detecting a motion vector corresponding to the position of the inspection block that best matches the reference block.

According to a fifth aspect of the present invention, the step of extracting the image data of the reference block from the first image data, the first stationary component feature quantity indicating the steady component of the block from the image data of the reference block and the transient 1st showing ingredients
Extracting the transient component characteristic amount of the test block, extracting the image data of the inspection block from the second image data, and extracting the second steady component characteristic amount and the transient component indicating the stationary component of the block from the image data of the inspection block. The second showing
Extracting the transient component feature amount of the reference block and the comparison result of the first and second steady-state component feature amounts and the comparison result of the first and second transient component feature amounts to determine the degree of coincidence between the reference block and the inspection block. Generating an evaluation value to be indicated, and detecting a motion vector corresponding to the position of the inspection block that most closely matches the reference block based on the evaluation value generated by moving the inspection block within a predetermined search range. And a motion vector detecting method characterized by the following.

[0028]

[Function] As a local feature of a block, not only a steady component but also a transient component is used. Using both,
Since the matching degree between the reference block and the check block is checked, a correct result can be obtained. Also, the number of all pixels in the block and all the search points in the search range are significantly reduced, and the amount of calculation can be significantly reduced.

[0029]

An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of one embodiment. In FIG. 1, reference numeral 1 denotes an input terminal for image data of a current frame, reference numeral 2 denotes an input terminal for image data of a reference frame, and reference numeral 3 denotes a current frame for storing image data of the current frame. Memory, 4
Is a reference frame memory for storing image data of the reference frame. The write / read operation of the current frame memory 3 and the reference frame is controlled by the controller 5.
Further, an address moving circuit 6 provided in association with the reference frame memory 4 causes the controller 5 to move the inspection block in the reference frame.

In order to extract an image feature amount for each block from the current frame memory 3, a steady component extraction circuit 7a
And a transient component extraction circuit 8a. The stationary component extraction circuit 7b and the transient component extraction circuit 8b are also connected to the reference frame memory 4. Here, the stationary component is
Average value, low frequency component, low order component of coefficient obtained by orthogonal transform, maximum value, minimum value, etc.Transient component is standard deviation, high frequency component, coefficient of orthogonal transform Higher order component, dynamic range, difference with respect to average value, maximum value of difference with respect to average value, and the like. The transient component indicates the degree of local activity in the image. As described above, the simplification of the conventional block matching method is to use only the stationary component as the feature amount of the block.

The target of block matching is not limited to the above-described current frame and reference frame (a frame temporally before or after the current frame). For example, when detecting a motion vector between two still images,
Alternatively, the present invention can be applied to a case where a motion vector is detected between images having different resolutions.

The difference detection circuit 12a detects a difference value from the steady component extraction circuits 7a and 7b. The difference detection circuit 12b detects a difference value from the transient component extraction circuits 8a and 8b. Absolute value conversion circuits 14a and 14b are connected to difference detection circuits 12a and 12b, respectively. The absolute value conversion circuits 14a to 14e convert the difference values into absolute values.

A weighted averaging circuit 15 is connected to the absolute value conversion circuits 14a and 14b. Weighting coefficients w1 and w2 are supplied from the controller 5 to the weighted averaging circuit 15. The evaluation value from the weighted averaging circuit 15 is supplied to the judgment circuit 18. The judging circuit 18 detects the minimum value among the evaluation values calculated within the search range. The motion vector corresponding to the position of the detected minimum evaluation value is determined by the determination circuit 1.
8 to the output terminal 27. The control of the judgment circuit 18 is performed by the controller 5, and the judgment result of the judgment circuit 18 is given to the controller 5.

FIG. 2 shows a more specific configuration of the above-described embodiment. That is, averaging circuits 31a and 31b are provided as the stationary component extraction circuits 7a and 7b, and a standard deviation (σ) generation circuit 32 is used as the transient component extraction circuits 8a and 8b.
a and 32b are provided. Difference detection circuits 12a, 1
2b is realized by the subtracters 35a and 35b.

The weighted averaging circuit 15 comprises multipliers 36a and 36b for multiplying weight coefficients w1 and w2, respectively, and an adder 37 for adding the multiplied outputs. The judging circuit 18 detects the weighted average data, that is, the memory 44 for storing the evaluation value and the minimum value among the evaluation values stored in the memory 44 and the minimum value for outputting the position of the minimum value as a motion vector. And a detection circuit 47. A configuration without the memory 44 is also possible as the determination circuit 18.

With respect to the embodiment of the present invention described above, the operation of detecting the motion vector will be described. As an example, the size of the motion vector detection target block (the reference block and the test block) is (4 × 4). (4x
The value of each pixel data in the block of 4) is x1 to x16.

The averaging circuit 31a and the standard deviation generating circuit 32a generate an average value m and a standard deviation σ calculated for each (4 × 4 pixel) block as described below. m = (x1 + x2 +... + x16) / 16 σ = {(xi-m) ² × 1/16} ^1/2 where Σ is the value obtained by subtracting the average value of the small block from the value of each pixel. Is calculated for 16 pixels of (4 × 4). The average value obtained as described above for the four small blocks included in one reference block is represented by m
And the standard deviation is σ.

Further, for one inspection block of the reference frame, an average value m 'and a standard deviation σ'('indicates the average value and the standard deviation of the inspection block) are obtained. Then, the subtracters 35a and 35b calculate the difference between the corresponding average value and the difference between the standard deviations, and supply the difference value to the weighted averaging circuit via the absolute value circuits 14a and 14b.

The position of the reference block is (x, y), and the position of the inspection block is (x + Δx, y + Δy) (where Δ
(x changes in four pixel steps and Δy changes in four line steps), and the above-described feature amount is expressed by the following equation. Hm (Δx, Δy) = | m ′ (x + Δx, y + Δy) −m (x, y) | Hσ (Δx, Δy) = | σ ′ (x + Δx, y + Δy) −σ (x, y) |

The weighted averaging circuit composed of the multipliers 36a and 36b and the adder 37 generates the following evaluation value obtained by weighted averaging of the above-mentioned feature values. H (Δx, Δy) = w1 × Hm (Δx, Δy) + w2 × Hσ (Δx, Δy)

Evaluation values obtained when Δx and Δy are changed, that is, when the inspection small block is moved by 4 pixel steps and 4 line steps, are stored in the memory 44. The minimum value detection circuit 47 detects the minimum value among the evaluation values stored in the memory 44. Δx and Δy corresponding to the position of the minimum value are detected motion vectors.

As can be seen from the above description of the embodiment, the (16 × 16) pixels in the block are reduced to the average value and the standard deviation, that is, two elements. Also search 4
Because it is done in pixel steps, it is simplified to 1/16. From these two points, the amount of calculation can be greatly reduced as compared with the conventional full search.

Various specific configurations of the present invention are possible in addition to the configurations of the above-described embodiment and other embodiments. For example, creation of evaluation values by software processing,
A motion vector may be detected.

Further, when obtaining the motion vector, the accuracy of the motion vector may be half-pel accuracy instead of one pixel.

[0045]

According to the present invention, a motion vector can be detected with a small amount of calculation. In particular, according to the present invention, not only a steady component but also a transient component of the original image data is used, so that a motion vector can be correctly detected. Therefore, the possibility of erroneous detection can be reduced.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 2 is a block diagram more specifically showing the configuration of an embodiment of the present invention.

FIG. 3 is a schematic diagram for explaining the motion vector detection of the present invention using an example of image data.

FIG. 4 is a schematic diagram for explaining motion vector detection according to the present invention using an example of image data.

FIG. 5 is a schematic diagram for explaining motion vector detection according to the present invention using another example of image data.

FIG. 6 is a schematic diagram for explaining motion vector detection according to the present invention using another example of image data.

FIG. 7 is a schematic diagram for explaining a motion vector detection of the present invention using another example of image data.

FIG. 8 is a block diagram illustrating an example of a motion-compensated prediction encoding apparatus to which the present invention can be applied.

FIG. 9 is a schematic diagram for explaining a motion vector detection method using a conventional block matching method.

FIG. 10 is a schematic diagram for explaining a motion vector detection method using a conventional block matching method.

FIG. 11 is a schematic diagram for explaining a search range in a conventional motion vector detection method using a block matching method.

FIG. 12 is a block diagram of an example of a motion vector detection device using a conventional block matching method.

[Explanation of symbols]

 3 Current frame memory 4 Reference frame memory 7a, 7b Steady component extraction circuit 8a, 8b Transient component extraction circuit 15 Weighted averaging circuit 18 Judgment circuit

Claims (5)

(57) [Claims] 1. A means for extracting image data of a reference block from first image data, and a first steady-state feature quantity indicating a steady-state component of the block from the image data of the reference block; 1
Means for extracting a transient component characteristic amount of the test block; means for extracting image data of the test block from the second image data; and a second steady component indicating a steady component of the block from the image data of the test block. 2nd showing feature value and transient component
Means for extracting the transient component feature quantity of the reference block and the comparison result of the first and second steady-state component feature quantities and the comparison result of the first and second transient component feature quantities. Means for generating an evaluation value indicating the degree of coincidence of the inspection block; and the inspection which best matches the reference block based on the evaluation value generated by moving the inspection block within a predetermined search range. A motion vector detecting device, comprising: means for detecting a motion vector corresponding to a position of a block. 2. The motion vector detecting device according to claim 1, wherein the stationary component feature quantity is an average value of pixel data in a block, a maximum value or a minimum value of the pixel data, and a low-frequency component of the pixel data. Or a low-order component formed by orthogonal transformation of the pixel data. 3. The motion vector detecting device according to claim 1, wherein the transient component feature quantity is a standard deviation of pixel data in a block, a dynamic range of the pixel data, a high frequency component of the pixel data, or the high frequency component of the pixel data. A device that is a higher-order component formed by orthogonal transformation of pixel data. 4. A first memory for storing first image data, for outputting image data of a reference block, and a first average value indicating a steady component of the block from the image data of the reference block. Calculating means for generating a first standard deviation indicating a transient component; a second memory for storing second image data and outputting image data of an inspection block; and Calculating means for generating a second average value indicating a stationary component of the block and a second standard deviation indicating a transient component; and comparing the first and second average values with the first and second average values. Means for generating an evaluation value indicating the degree of coincidence between the reference block and the test block by performing a weighted average of the comparison results of the two standard deviations; and determining the test block within a predetermined search range. When moving,
Means for detecting a motion vector corresponding to the position of the test block that generates the minimum evaluation value. 5. A step of extracting image data of a reference block from the first image data; and a first steady-state feature amount indicating a steady-state component of the block from the image data of the reference block;
Extracting a transient component feature of the test block; extracting image data of the test block from the second image data; and a second steady component feature and a transient component indicating a steady component of the block from the image data of the test block. Second showing the ingredients
Extracting the transient component feature amounts of the reference block and the inspection block from the comparison result of the first and second steady-state component feature amounts and the comparison result of the first and second transient component feature amounts. Generating an evaluation value indicating the degree of matching of the test block; and, based on the evaluation value generated by moving the inspection block within a predetermined search range, a position of the inspection block that best matches the reference block. Detecting a corresponding motion vector.