JP3277419B2 - Apparatus and method for detecting motion vector - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and a method for detecting a motion vector representing the direction and amount of motion of an image by a block matching method.

[0002]

2. Description of the Related Art One application of motion vectors is in motion compensation in predictive coding of digital image data. As an example, MPEG (Moving Picture Coding Experts Groove), an international standard for efficient coding of moving images, is used.
up) A method has been proposed. This MPEG method uses DC
This is a combination of T (Discrete Cosine Transform) and motion compensation prediction coding.

FIG. 8 shows an example of a motion compensation predictive coding apparatus. 8, digital video data from an input terminal 61 is supplied to a motion vector detection circuit 62 and a subtraction circuit 63. In the motion vector detection circuit 63, the current frame and the reference frame (for example, a temporally previous frame)
Is detected. This motion vector is supplied to the motion compensation circuit 64.

[0004] After the image stored in the frame memory 65 is motion-compensated by the motion compensation circuit 64 based on the motion vector, it is supplied to a subtraction circuit 63 and an addition circuit 66. The subtraction circuit 63 subtracts the video data of the current frame and the decoded video data of the previous frame from the motion compensation circuit 64 for each pixel. The difference data from the subtraction circuit 63 is DCT-transformed by the DCT circuit 67.
The coefficient data from the DCT circuit 67 is requantized by the quantization circuit 68. Output data of the quantization circuit 68 is taken out to an output terminal 69 and supplied to an inverse quantization circuit 70.

The inverse quantization circuit 70 and the inverse D connected thereto
The CT circuit 71 includes a DCT circuit 67 and a quantization circuit 6
A local decoding circuit for performing a process opposite to that of FIG. The decoded difference data from the inverse DCT circuit 71 is supplied to the adding circuit 66. The output data of the adding circuit 66 is supplied to the motion compensation circuit 64 via the frame memory 65.
The decoded data of the previous frame from the motion compensation circuit 64 is supplied to the addition circuit 66, thereby forming decoded data.
This decoded data is stored in the frame memory 65.

The motion vector detection circuit 62 detects a motion vector by a block matching method. In this method, a motion vector is obtained by moving an inspection block of a reference frame within a predetermined search range and detecting a block that most closely matches a reference block of the current frame.

In the block matching method, as shown in FIG. 9A, one image, for example, an image of one frame of horizontal H pixels and vertical V lines is composed of P pixels × Q as shown in FIG. 9B.
Subdivided into blocks of lines. In the example of FIG. 9B, P
= 5, Q = 5. c is the center pixel position of the block.

FIG. 10 shows the positional relationship between a reference block whose center is c and an inspection block whose center is c '. The reference block whose center pixel is c is a certain reference block of interest in the current frame, and it is assumed that the inspection block of the reference frame that matches the image is located at the position of the block centered on c ′ in the reference frame. I have. In the block matching method, a motion vector is detected by finding an inspection block that best matches a reference block within a search range. In the case of FIG. 10A, +1 pixel in the horizontal direction, +1 line in the vertical direction,
That is, a motion vector of (+1, +1) is detected. In FIG. 10B, a (+3, +3) motion vector is detected, and in FIG. 10C, a (+2, -1) motion vector is detected. The motion vector is obtained for each reference block of the current frame.

Assuming that the range for searching for a motion vector is ± S pixels in the horizontal direction and ± T lines in the vertical direction, the reference block is shifted ± S horizontally and ± T vertically from its center c. It needs to be compared to a test block with center c '. FIG. 11 shows that when the position of the center c of a certain reference block of the current frame is R, comparison with (2S + 1) × (2T + 1) check blocks of a reference frame to be compared is necessary. That is, FIG.
All the test blocks in which c 'is present at the position of the first square are comparison targets. FIG. 11 is an example in which S = 4 and T = 3. Evaluation values obtained by comparison within the search range (ie, sum of absolute values of frame differences, sum of squares of the frame differences,
Alternatively, the motion vector is detected by detecting the minimum value among the sums of the absolute values of the frame differences.
The search range in FIG. 11 is an area where the center of the test block is located, and the size of the search range including the entire test block is (2S + P) × (2T + Q).

FIG. 12 shows an example of the configuration of a conventional motion vector detecting device. In FIG. 12, reference numeral 81 denotes an input terminal for image data of the current frame, and this image data is stored in the current frame memory 83. Reference numeral 82 denotes an input terminal for image data of a reference frame, and this image data is stored in a reference frame memory 84.

The reading / writing of the current frame memory 83 and the reference frame memory 84 is controlled by a controller 85. From the current frame memory 83, the pixel data of the reference block of the current frame is read, and from the reference frame memory 84, the pixel data of the inspection block of the reference frame is read. An address moving circuit 86 is provided in association with the reference frame memory 84. As a result of the controller 85 controlling the address moving circuit 86, the center position of the test block is moved within the search range in one-pixel steps.

The output of the current frame memory 83 and the output of the reference frame memory 84 are supplied to a difference detection circuit 87,
A difference for each pixel is detected. The output of the difference detection circuit 87 is converted into an absolute value by an absolute value conversion circuit 88, and this absolute value is supplied to an accumulation circuit 89. An accumulating circuit 89 accumulates the absolute value difference generated in one block, and its output is determined by a judging circuit 9.
0 is supplied. The determination circuit 90 detects a motion vector from the sum of absolute values of differences generated when the inspection block is moved within the search range. That is, the position of the test block that generates the absolute value sum of the minimum difference is detected as a motion vector.

In the above-described conventional block matching method, it is necessary to perform a process for obtaining the sum of absolute values of frame differences between a reference block and an inspection block within a search range. In the examples of FIGS. 9, 10 and 11, it is necessary to accumulate (P × Q) absolute value differences (2S + 1) × (2T + 1) times. From this relationship, the calculation amount is (P × Q) × (2S
+1) × (2T + 1). Therefore, the above-described block matching method has a problem that the scale of hardware is large and the amount of calculation is enormous.

As a countermeasure, two types of methods have been proposed. One is a technique for reducing the number of elements in a block, and the other is a technique for simplifying a search.
As a method of reducing the number of elements, a method has been proposed in which a reference block and an inspection block are further divided into small blocks in each of the horizontal direction and the vertical direction, and a feature amount is extracted for each small block. That is,
The feature value of the horizontal small block and the feature value of the vertical small block are separately compared between the reference block and the inspection block, the absolute values of the comparison results are respectively accumulated, and the weighted average of the accumulation results is calculated. Used as a comparison result between blocks. The feature amount of a small block is, for example, an accumulation result of pixel data in the small block. This method can reduce the operations required for the total number of pixels in one block to the number of small blocks in the horizontal and vertical directions.

As a method of simplifying the search, when the inspection block is moved within the search range, the first step is to move the inspection block at intervals of several pixels, thereby detecting a rough motion vector, and There is known a method (two-step method) of finally obtaining a motion vector by moving an inspection block at intervals of one pixel in the vicinity of a detected position in a step. A three-step method with three steps is also conceivable. According to this method, the number of calculations corresponding to the total number of pixels in the search range required for the full search can be reduced to the number of times corresponding to the number of pixels around the motion vector detected in each step. .

Furthermore, a method employing a hierarchical structure has been proposed as a motion vector detection method intended to both reduce the number of elements and simplify search. One of them can be called a thinning method. That is, the number of pixels in the block is thinned out by subsampling (for example, 4
Pixel is thinned to one pixel), block matching is performed on a block composed of the pixels resulting from the thinning, and then the origin is moved to the position of the detected minimum value, for example, two pixels are thinned to one pixel. Block matching is performed on the block structure thus obtained, the origin is moved to the position of the detected minimum value, and finally a motion vector is detected by one pixel step block matching. As a result of the thinning, both the number of elements in the block and the number of operations in the search range are reduced.

Another use of a low-pass filter is to both reduce the number of elements and simplify the search. From the original image (referred to as the first layer) and the low-pass filter and sub-sampling from the first layer,
The second in which the number of pixels is thinned in half in the horizontal and vertical directions
Layer and the second layer by a low-pass filter and sub-sampling by half in the horizontal and vertical directions.
Is defined as a third-layer structure thinned out. Then, block matching is performed on the third hierarchy,
The origin is moved to the position of the detected minimum value to perform block matching on the second hierarchy, and the origin is moved to the position of the detected minimum value to perform block matching on the first hierarchy.

[0018]

The above-described various improvements in block matching can reduce the amount of computation, but have the disadvantage of causing erroneous detection of motion vectors. This is because the simplification results in a loss of the information amount of the original image.

More specifically, in the method of reducing the number of elements in a block, the feature amount of the small block is obtained by passing the image data of the small block through a low-pass filter. In the method of simplifying the search, when a rough motion vector is detected, there is a risk of erroneous detection due to low accuracy. In the method of reducing the number of elements and simplifying the search, erroneous detection may occur when a motion vector is detected based on a thinned image or an image passed through a low-pass filter.

Accordingly, it is an object of the present invention to provide a motion vector detecting apparatus and a detecting method in which an erroneous detection is prevented, in which the amount of calculation can be reduced and the hardware has only simple features.

The present invention focuses on the necessity of image activity in order to correctly detect a motion vector by block matching. The activity means a level change or an amount of a high frequency component. The conventional simplification technique has resulted in a loss of activity, and has caused a problem of false detection by performing block matching based on the image with lost activity. This point will be described below using specific numerical values for easy understanding.

FIG. 3A shows an example of pixel values included in an (8 × 8) pixel area. This value indicates the level (0 to 255) of each pixel of 8 bits. Here, (2 ×
FIG. 3B shows the result of calculating the average value as the steady component feature value for each region of 2 = 4 pixels), and FIG. 3C shows the result of calculating the standard deviation as the transient component feature value for each region. In FIG. 3B, when looking at the average value, adjacent average values “164” and “163” are extremely close values. However, as shown in FIG. 3C, the corresponding standard deviations are “0.5” and “7”, which are greatly different. This is because the degree of the level change is different between the two (2 × 2) areas.

Next, FIG. 4A shows a (8 × 8) region in which the leftmost column in FIG. 3A is removed and a new column is added to the rightmost column. For this region, the result of calculating the average value is shown in FIG. 4B, and the result of calculating the standard deviation is shown in FIG. 4C, as described above. 4B and 4C
Also, the average values of the positions corresponding to “164” and “163” are “164” and “162”, and the standard deviations are “2” and “10.8”. Also in this example, it can be seen that the difference of the standard deviation is larger than that of the average value.

FIG. 5 shows an example of data of a region of (16 × 16) pixels. FIG. 6 shows the result of calculating the average value for each (4 × 4) pixel, and FIG. 7 shows the result of calculating the standard deviation. As can be seen from FIGS. 6 and 7, also in this example, although the average value is the same as “145”, the standard deviation is different from “4.91” and “4.05”.

Thus, (2 × 2) or (4 ×
The local feature of the image of 4) can be obtained by using only the stationary component feature amount.
It cannot be sufficiently expressed, and can be accurately expressed only by taking into account the transient component feature amount. Since the block matching method is to check the degree of matching of an image between a reference block and an inspection block, it is reasonable to include a transient component as a feature value of the block as in the present invention. I have.

[0026]

According to a first aspect of the present invention, there is provided means for extracting image data of a first block obtained by dividing a reference block from first image data;
Means for extracting, from the image data of the first block, a first steady-state component feature amount indicating a steady-state component of the first block and a first transient component feature amount indicating a transient component, and second image data Means for extracting image data of a second block obtained by dividing an inspection block from a second block, and a second steady-state feature value and a transient component indicating a steady-state component of the second block from the image data of the second block Means for extracting a second transient component feature amount indicating the reference block and a comparison result of the first and second stationary component feature amounts and a comparison result of the first and second transient component feature amounts. Means for generating an evaluation value indicating the degree of coincidence of the inspection block; and a position of the inspection block that best matches the reference block, based on the evaluation value generated by moving the inspection block within a predetermined search range. versus A motion vector detecting apparatus characterized by having a means for detecting a motion vector.

According to a fifth aspect of the present invention, a step of extracting image data of a first block obtained by dividing a reference block from the first image data, and a step of extracting the first block from the image data of the first block Extracting a first steady-state component feature amount indicating a steady-state component and a first transient component feature amount indicating a transient component, and image data of a second block obtained by dividing an inspection block from the second image data And extracting a second steady-state component feature amount indicating a steady-state component of the second block and a second transient component feature amount indicating a transient component from the image data of the second block. Generating an evaluation value indicating the degree of coincidence between the reference block and the inspection block from the comparison result of the second stationary component feature amount and the comparison result of the first and second transient component feature amounts; Detecting a motion vector corresponding to the position of the test block that most closely matches the reference block based on the evaluation value generated by moving the test block within the fixed search range. This is a vector detection method.

[0028]

The motion detection target block is further divided into small blocks, and both a steady component and a transient component are obtained as a feature amount for each small block. Then, the stationary component characteristic amount and the transient component characteristic amount are compared between the small block of the reference block and the small block of the reference block, and a motion vector is detected based on the comparison result. By also seeing the transient component, erroneous detection can be prevented.
Further, the amount of calculation can be greatly reduced.

[0029]

An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of one embodiment. In FIG. 1, reference numeral 1 denotes an input terminal for image data of a current frame, reference numeral 2 denotes an input terminal for image data of a reference frame, and reference numeral 3 denotes a current frame for storing image data of the current frame. Memory, 4
Is a reference frame memory for storing image data of the reference frame. The write / read operation of the current frame memory 3 and the reference frame is controlled by the controller 5.
Further, an address moving circuit 6 provided in association with the reference frame memory 4 causes the controller 5 to move the inspection block in the reference frame.

The output of the current frame memory 3 is connected to a small block circuit 11a. This small block
The reference block for motion vector detection is further divided. In order to extract the feature amount of the image for each small block, a steady component extracting circuit 7a and a transient component extracting circuit 8a are connected to the small block forming circuit 11a.

The reference frame memory 4 is also connected to a small block circuit 11b, a steady component extraction circuit 7b, and a transient component extraction circuit 8b. Here, the stationary component is an average value, a low-frequency component, a low-order component of a coefficient obtained by orthogonal transformation, a maximum value, a minimum value, and the like, and the transient component is a standard deviation, a high-frequency component, This means higher order components of coefficients obtained by orthogonal transformation, dynamic range, difference from the average value, maximum value of the difference from the average value, and the like. The transient component indicates the degree of local activity in the image. As described above, the simplification of the conventional block matching method is to use only the stationary component as the feature amount of the block.

The target of block matching is not limited to the above-described current frame and reference frame (the frame temporally before or after the current frame). For example, when detecting a motion vector between two still images,
Alternatively, the present invention can be applied to a case where a motion vector is detected between images having different resolutions.

The difference detection circuit 12a detects a difference value from the steady component extraction circuits 7a and 7b. The difference detection circuit 12b detects a difference value from the transient component extraction circuits 8a and 8b. Absolute value conversion circuits 14a and 14b are connected to difference detection circuits 12a and 12b, respectively. The absolute value conversion circuits 14a to 14e convert the difference values into absolute values.

Accumulation circuits 16a and 16b are connected to the absolute value conversion circuits 14a and 14b, respectively.
The outputs of a and 16b are supplied to the weighted averaging circuit 15.
Weighting coefficients w1 and w2 are supplied from the controller 5 to the weighted averaging circuit 15. Weighted averaging circuit 1
The evaluation value from 5 is supplied to the judgment circuit 18. The judging circuit 18 detects the minimum value among the evaluation values calculated within the search range. A motion vector corresponding to the position of the detected minimum evaluation value is generated at the output terminal 27 from the judgment circuit 18. The control of the judgment circuit 18 is performed by the controller 5, and the judgment result of the judgment circuit 18 is given to the controller 5.

FIG. 2 shows a more specific configuration of the above-described embodiment. That is, averaging circuits 31a and 31b are provided as the stationary component extraction circuits 7a and 7b, and a standard deviation (σ) generation circuit 32 is used as the transient component extraction circuits 8a and 8b.
a and 32b are provided. Difference detection circuits 12a, 1
2b is realized by the subtracters 35a and 35b.

The accumulators 16a, 16b are provided with adders 38a,
38b and registers 39a and 39b inserted in the feedback loop. Weighted averaging circuit 15
Is composed of multipliers 36a and 36b for multiplying weight coefficients w1 and w2, respectively, and an adder 37 for adding the multiplied outputs. The judging circuit 18 detects the weighted average data, that is, the memory 44 for storing the evaluation value and the minimum value among the evaluation values stored in the memory 44 and the minimum value for outputting the position of the minimum value as a motion vector. And a detection circuit 47. A configuration without the memory 44 is also possible as the determination circuit 18.

The operation of detecting the motion vector of the embodiment of the present invention will be described. As an example, the size of the motion vector detection target block is set to (16 ×
16). Smaller blocks are converted to (4 ×
This is a process of dividing into 4 small blocks of the size of 4). The value of each pixel data in the (4 × 4) block is x1
To x16.

The averaging circuit 31a and the standard deviation generating circuit 32a generate an average value m and a standard deviation σ calculated for each (4 × 4 pixel) small block, as described below. m = (x1 + x2 +... + x16) / 16 σ = {(xi-m) ² × 1/16} ^1/2 where Σ is the value obtained by subtracting the average value of the small block from the value of each pixel. Is calculated for 16 pixels of (4 × 4). The average value obtained as described above for the four small blocks included in one reference block is represented by m
1 to m4 and the standard deviation is σ1 to σ4

For one inspection block of the reference frame, the average values m1 'to m4' and the standard deviation σ1 '
~ Σ4 '(' indicates the average value and standard deviation for the inspection block). Then, in the subtracters 35a and 35b, the difference between the corresponding average value and the difference between the standard deviations are calculated, and the difference value is calculated by the absolute value circuits 14a and 14b.
Is supplied to the accumulator through. Accordingly, the accumulator including the adder 38a and the register 39a outputs an average value feature amount of Hm = | m1-m1 '+ m2-m2' + m3-m3 '+ m4-m4' |. The accumulator including the adder 38b and the register 39b outputs a standard deviation feature amount of Hσ = | σ1-σ1 ′ + σ2-σ2 ′ + σ3-σ3 ′ + σ4-σ4 ′ |.

The position of the reference block is (x, y), and the position of the inspection block is (x + Δx, y + Δy) (where Δ
(x changes in four pixel steps and Δy changes in four line steps), and the above-described feature amount is expressed by the following equation. Hm (Δx, Δy) = Σ | mi ′ (x + Δx, y + Δy) −mi (x, y) | Hσ (Δx, Δy) = Σ | σi ′ (x + Δx, y + Δy) −σi (x, y) |

The weighted averaging circuit composed of the multipliers 36a and 36b and the adder 37 generates the following evaluation value obtained by weighting and averaging the above-mentioned feature amounts. H (Δx, Δy) = w1 × Hm (Δx, Δy) + w2 × Hσ (Δx, Δy)

Evaluation values obtained when Δx and Δy are changed, that is, when the inspection small block is moved by 4 pixel steps and 4 line steps, are stored in the memory 44. The minimum value detection circuit 47 detects the minimum value among the evaluation values stored in the memory 44. Δx and Δy corresponding to the position of the minimum value are detected motion vectors.

As can be seen from the above description of the embodiment, the (16 × 16) pixels in the block are reduced to the average value and standard deviation in four small blocks, that is, eight elements. Also, since the search is also performed in 4-pixel steps, 1
/ 16. From these two points, the amount of calculation can be greatly reduced as compared with the conventional full search.

Various specific configurations of the present invention are possible in addition to the configurations of the above-described embodiment and other embodiments. For example, creation of evaluation values by software processing,
A motion vector may be detected.

Further, the formation of the small blocks is not limited to the above example. For example, a block may be divided at equal intervals in the horizontal and / or vertical directions to form a striped small block. Furthermore, small blocks may overlap. Furthermore, when obtaining the motion vector, the accuracy of the motion vector may be set to half-pel accuracy instead of one pixel.

[0046]

According to the present invention, since the feature amount can be calculated for each small block and the search can be performed using the block size, the motion vector can be detected with a small amount of calculation. In particular, according to the present invention, not only a steady component but also a transient component of the original image data is used, so that the possibility of erroneous detection can be reduced.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 2 is a block diagram more specifically showing the configuration of an embodiment of the present invention.

FIG. 3 is a schematic diagram for explaining the motion vector detection of the present invention using an example of image data.

FIG. 4 is a schematic diagram for explaining motion vector detection according to the present invention using an example of image data.

FIG. 5 is a schematic diagram for explaining motion vector detection according to the present invention using another example of image data.

FIG. 6 is a schematic diagram for explaining motion vector detection according to the present invention using another example of image data.

FIG. 7 is a schematic diagram for explaining a motion vector detection of the present invention using another example of image data.

FIG. 8 is a block diagram illustrating an example of a motion-compensated prediction encoding apparatus to which the present invention can be applied.

FIG. 9 is a schematic diagram for explaining a motion vector detection method using a conventional block matching method.

FIG. 10 is a schematic diagram for explaining a motion vector detection method using a conventional block matching method.

FIG. 11 is a schematic diagram for explaining a search range in a conventional motion vector detection method using a block matching method.

FIG. 12 is a block diagram of an example of a motion vector detection device using a conventional block matching method.

[Explanation of symbols]

 3 Current frame memory 4 Reference frame memory 7a, 7b Steady-state component extraction circuit 8a, 8b Transient component extraction circuit 16a, 16b Accumulation circuit of absolute value of difference 15 Weighted averaging circuit 18 Judgment circuit

Claims (5)

(57) [Claims] A means for extracting image data of a first block obtained by dividing a reference block from the first image data; and determining a stationary component of the first block from the image data of the first block. Means for extracting a first steady-state component characteristic amount shown and a first transient component characteristic amount showing a transient component, and a second part obtained by dividing an inspection block from the second image data.
Means for extracting the image data of the second block, a second steady-state component feature indicating the steady-state component of the second block and a second transient component feature indicating the transient component of the second block from the image data of the second block From the comparison result of the first and second steady-state component features and the comparison result of the first and second transient component features from the reference block and the inspection block. Means for generating an evaluation value indicating the position of the inspection block that best matches the reference block based on the evaluation value generated by moving the inspection block within a predetermined search range. Means for detecting a motion vector to be detected. 2. The motion vector detecting device according to claim 1, wherein the first and second stationary component feature amounts are an average value of pixel data in a block, a maximum value or a minimum value of the pixel data, A device which is a low-frequency component of pixel data or a low-order component formed by orthogonal transformation of the pixel data. 3. The motion vector detecting device according to claim 1, wherein the first and second transient component features are a standard deviation of pixel data in a block, a dynamic range of the pixel data, and a dynamic range of the pixel data. A device that is a high-frequency component or a higher-order component formed by orthogonal transformation of the pixel data. 4. A first memory for storing first image data, for outputting image data of a first block obtained by dividing a reference block, and a first memory for outputting image data of the first block. A first average value indicating a stationary component and a first average value indicating a transient component of one block
A second memory for storing second image data and outputting second image data obtained by dividing the inspection block; and a second memory for storing the second image data. From the image data, a second average value indicating a stationary component of the second block and a second average value indicating a transient component of the second block
Calculating means for generating a standard deviation of the first and second average values, and a value obtained by accumulating the comparison result of the first and second average values and a value obtained by accumulating the comparison result of the first and second standard deviations. Means for generating an evaluation value indicating the degree of coincidence between the reference block and the inspection block by performing weighted averaging; and when the inspection block is moved within a predetermined search range,
Means for detecting a motion vector corresponding to the position of the test block that generates the minimum evaluation value. 5. A step of extracting image data of a first block obtained by dividing a reference block from the first image data; and extracting a stationary component of the first block from the image data of the first block. Extracting a first transient component feature indicating a steady component feature and a transient component, and a second test block obtained by dividing an inspection block from the second image data.
Extracting the image data of the second block, and extracting the second stationary component characteristic amount indicating the stationary component of the second block and the second transient component characteristic amount indicating the transient component from the image data of the second block. An evaluation value indicating a degree of coincidence between the reference block and the inspection block from the comparison result of the first and second steady-state component feature amounts and the comparison result of the first and second transient component feature amounts. And detecting a motion vector corresponding to the position of the test block that best matches the reference block based on the evaluation value generated by moving the test block within a predetermined search range. And a step for detecting a motion vector.