DE69517673T2 - Hierarchical motion vector estimation with several block sizes - Google Patents

Description

Field of the invention

The present invention relates to a method for determining motion vectors, and more particularly to an improved method for determining motion vectors from two consecutive video images using a hierarchical motion estimation method.

Description of the state of the art

In many electronic/electrical applications, such as high definition television and video telephone systems, an image signal must be transmitted in digital form. When the image signal is represented in a digital form, a significant amount of digital data is inevitably involved. However, since the available frequency bandwidth of a conventional transmission channel is limited, in order to transmit the image signal through it, it is inevitable to compress or reduce the amount of transmission data. Among the numerous video compression techniques, a motion compensated interframe coding technique is known as one of the more efficient compression techniques. It uses temporal redundancies of the video signals between two consecutive video frames to compress the signals.

In the motion compensated inter-frame coding method, current image data is predicted from the previous image data based on an estimate of motion between the current and previous frames. Such motion estimation can be described using two-dimensional motion vectors representing the pixel displacement between the previous and current frames. One motion vector estimation method proposed in the prior art is a block matching algorithm (see, e.g., BJR Jain et al., "Displacement Measurement and Its Application in Interframe Image Coding," IEEE Transactions on Communications COM-29, No. 12 (December 1981)).

According to the block matching algorithm, a current image is divided into several equally sized search blocks. The size of the search blocks is typically between 8 x 8 and 8 x 32 pixels. To determine a motion vector for a search block in the current image, a similarity calculation is performed between the search block in the current image and each of several equally sized candidate blocks contained in a generally larger search area in a previous image. Error functions such as mean absolute error values or mean square error values are used to perform a similarity measurement between the search block of the current image and each candidate block in the search area. A motion vector is by definition the displacement between the search block and a block with the highest match, i.e. a candidate block that results in a minimal "error value" or difference.

However, in this block matching motion estimation method using a search block with a fixed size, if the size of the search block is relatively large, the same motion vector may be assigned to all pixels in the search block, but they do not move in the same way. This lowers the overall image quality.

JP-A-5328334 describes an apparatus for determining a motion vector between corresponding blocks of a current and a previous video image using a search algorithm in which maximum and minimum error value differences between a candidate block and search blocks of a search area are derived. The difference between them is compared with a threshold value. If the difference exceeds the threshold, the vector corresponding to the search block showing the minimum error value is output as the best estimated motion vector. If not, the search area is enlarged.

EP-A-0538667 describes an encoder in which several motion compensators work in parallel. The block sizes for which the compensators determine the corresponding motion vectors are organized hierarchically. The motion compensator that results in the smallest amount of compressed data is selected. This has the additional effect that fewer motion vectors need to be coded if the best result was achieved with a large block.

Summary of the invention

It is therefore a primary aim of the invention to provide an improved method with which motion vectors can be determined in an accurate manner from two consecutive video images using a hierarchical motion estimation method.

According to the present invention, there is provided a method for determining a motion vector according to claim 1, and an apparatus according to claim 7. Preferred embodiments of the invention are defined in the dependent claims.

Short description of the drawing

The above and other objects and features of the present invention will become apparent from the following description of preferred embodiments taken together with the accompanying drawings, in which:

Fig. 1 is a block diagram of the inventive apparatus for determining motion vectors between a current and a previous video signal frame;

Fig. 2 shows a detailed block diagram illustrating an error value generator shown in Fig. 1; and

Fig. 3 shows exemplary hierarchical search blocks.

Detailed description of the preferred embodiments

Referring to Fig. 1, a block diagram of the inventive apparatus for determining a motion vector for each of a plurality of processing blocks in a current image using a multi-tiered hierarchical motion estimation method is shown. Each processing block of size P · Q, e.g. 2 · 2 pixels, comprises hierarchical search blocks in a current image.

The data of the current image is supplied as a digital video input signal to an image memory 50 and to a block forming section 100 to generate hierarchical search blocks. The block forming section 100 serves to generate the hierarchical search blocks for each processing block, with the processing block located at the center of the search blocks. A search block of a hierarchy is larger than a search block of a lower hierarchy by 2 N pixels in the vertical and horizontal directions, where N is an integer greater than 0, e.g. 1, and a search block of a lowest hierarchy is equal to or larger than the processing block.

As shown in Fig. 1, the block forming section 100 includes a number M, e.g. 4, block formatters, e.g. 110-170. In the block formatters 110, 130, 150 and 170, the hierarchical search blocks are generated for each processing block, and the sizes of the search blocks vary in 2 N pixel steps from 2 N to 2 N (M-1) in the horizontal and vertical directions, where M is an integer greater than 1. More specifically, a first block formatter 110 generates search blocks of a highest hierarchy with a size of H x V pixels, a second block formatter 130 generates search blocks of a lower hierarchy with a size of (H-2 N) · (V-2 N), e.g. (H-2) · (V-2) pixels, a third block formatter 150 produces search blocks of a still lower hierarchy with a size of (H-4 N) · (V-4 N), e.g. (H-4) · (V-4) pixels, and a final block formatter 170 produces search blocks of a lowest hierarchy with a size of (H-2N (M-1)) · (V-2N (M-1)), e.g. (H-2 (M- 1)) · (V-2(M-1)) pixels.

Figure 3 illustrates by way of example the hierarchical search blocks generated by the block formatters 110-170, which hierarchical search blocks are used in a preferred embodiment of the present invention. For purposes of illustration, H and V have been assumed to be the same number 16 for each pixel search block in the current image.

As shown in Fig. 3, the block formatter 110 forms a search block 10 with the size of H x V, i.e., 16 x 16 pixels, the search block 10 is constructed using a processing block 5 as a reference, and the processing block 5 is located at the center of the search block 10. Similarly, the block formatter 130 constructs a smaller search block 12 with the size (H-2) x (V-2), i.e., 14 x 14 pixels, the block formatter 150 constructs an even smaller search block 14 with the size (H-V) x (V-4), i.e., 12 x 12 pixels, and the last block formatter 170 creates a smallest search block 16 of the lowest search block hierarchy with the size (H-8) x (V-8), i.e., 16 x 16 pixels. i.e. 8 x 8 pixels using the processing block 5 as a reference, wherein the processing block 5 is located at the center of the corresponding smaller search blocks 12, 14 and 16.

Referring again to Fig. 1, the hierarchical search blocks generated by the block formatters 110-170 are fed to a motion estimation unit 200 which includes a plurality of error function calculators 101-270. Data of the previous frame from the frame store 50 is also fed to the motion estimation unit 200. In each of the motion estimation units 200 The error function calculator 210-270 included processes the previous frame data from the frame memory 50 and the current frame data to calculate and estimate a motion vector representing the displacement between a search block of the current frame and a candidate block, and at which the minimum error function results.

The error function calculators 210, 230, 250 and 270 are formed of substantially identical elements, with each element serving the same function. Accordingly, the operation of each error function calculator 210, 230, 250 or 270 is substantially identical. For this reason, the operation of the error function calculator 210 will be described in detail with reference to Figure 2, which shows a detailed block diagram of the error function calculator 210.

As shown in Fig. 2, the previous image data supplied from the image memory 50 shown in Fig. 1 is supplied to a search area forming section 211. The search area forming section 211 defines search areas of the previous image having the same size, the size of each search area being generally larger than that of the search block.

After the search area is determined in the search area forming section 211, the search area data is supplied to a plurality of block forming sections, e.g. 212-124. In each block forming section, a candidate block of H · V, i.e. 16 · 16 pixels, is generated from the search area. The relative displacements of the candidate blocks from the location of the search block in the current image are output as displacement vectors, e.g. 612-614 of the candidate blocks. All possible candidate blocks of size H · V pixels are formed within the determined search area, and corresponding displacement vectors are obtained for each candidate block.

The pixel data of each candidate block is output from each candidate block forming section 212-216 to the respective block matching section 215-217. The data of the search block of size H x V pixels is also supplied from the block formatter 110 shown in Fig. 1 to each block matching section 215-217. In each block matching section, an error function, e.g., average absolute error values between the search block from the block formatter 110 and the candidate blocks from the corresponding candidate block forming sections 212-214, is calculated, which corresponds to the average of the absolute difference between the pixels in the search block and the candidate block. A comparison, e.g., of the light intensity or brightness between corresponding pixels in the search block and each candidate block is made to obtain an error function for each candidate block. The error function indicates the degree of similarity between the search block and the respective candidate block.

All error functions from the block matching sections 215-217 are fed to a minimum value detector 218. The minimum value detector 218 compares the error functions and selects the error function that has a smallest value.

The minimum value detector 218 outputs the signal indicating the block corresponding to the minimum error function to a selector 219. The displacement vectors 612-614 obtained from the block sections 212-214 are also supplied to the selector 219. The selector 219 then selects the displacement vector of the candidate block corresponding to the minimum error function as the motion vector for the search block of size H · V pixels. The determined motion vector is output as a first hierarchical motion vector MV1 to a motion vector selector 400 and a selector 332 shown in Fig. 1. The minimum error function selected by the minimum value detector 218 is as a first hierarchical error function E1 to a low value detector 312 and a comparator 322 shown in Fig. 1.

Similarly, each of the error function generators 230-270 shown in Fig. 1 compares a corresponding search block with each candidate block contained in its corresponding search area of the previous image using an error function such as an average absolute error value. Each error function generator 230-270 then generates error functions for each search block at each hierarchical level. The error functions at each hierarchical level are processed to generate the minimum error function for each search block at each hierarchy. The displacement vector corresponding to the minimum error function is selected as the motion vector of the search block. The minimum error functions from each error function generator 230-270 are fed to low value detectors 312-316 as second hierarchical, third hierarchical and last hierarchical error functions E2-E4, respectively. Each motion vector for each search block in each hierarchy is fed to selectors 332-336 as second hierarchical, third hierarchical and last hierarchical motion vectors MV2-MV4, respectively.

The low value detector 312 outputs a smaller error function among its two inputs, ie, the first and second hierarchical error functions E1, E2 to the comparator 324 and a next low value detector 314, and generates a control signal indicating to the selector 332 the hierarchical motion vector corresponding to the smaller error function. The next low value detector 314 outputs a smaller error function among its two inputs, ie, the output from the low value detector 312 and the third hierarchical error function E3 to the comparator 326 and the next stage low value detector, and generates a control signal indicating the hierarchical motion vector. corresponding to the smaller error function, which is output therefrom to the selector 334. The last low detector 316 outputs the smaller error function among its two inputs, ie, the output from the previous stage low detector and the last hierarchical error function E4 to the comparator 328, and generates a control signal indicative of the hierarchical motion vector corresponding to the smaller error function, which is output therefrom to the selector 336.

Each comparator 322-328 compares its two inputs, i.e., a predetermined threshold and the hierarchical error function supplied to it, and generates a logic high or logic low select signal S1-S4 to the motion vector selector 400. That is, each comparator generates a logic high select signal if the predetermined threshold is greater than the corresponding hierarchical error function. Otherwise, it generates a logic low select signal to the motion vector selector 400.

The comparator 322 compares the predetermined threshold with the first hierarchical error function E1 and generates a logic high select signal if the predetermined threshold is greater than E1. Otherwise, it generates a logic low select signal for the motion vector selector 400. Similarly, each comparator 324-328 compares the predetermined threshold with the output of the corresponding low detector 312-316 and generates a logic high select signal if the predetermined threshold is greater than the output of the corresponding low detector. Otherwise, it generates a logic low select signal for the motion vector selector 400.

According to the present invention, when S1 is logic high, the selection signals S2-S4 become logic high. When S1 is logic low and S2 is logic high, S3-S4 logically high. In other words, when a select signal corresponding to a hierarchy is logically high, the select signal(s) corresponding to the lower hierarchy will be logically high.

Each selector 332-336 selects one of the two hierarchical motion vectors input to it in response to the control signals provided by the respective low level detector 312-316. That is, the selector 332 selects one of its two inputs, i.e., the first and second hierarchical motion vectors MV1, MV2 in response to the control signal provided by the low level detector 312, and supplies the selected motion vector to the selector 334 and the motion vector selector 400. The selector 334 selects one of its two inputs, i.e., the output from the selector 332 and the third hierarchical motion vector MV3 in response to the control signal provided by the low level detector 314, and supplies the selected motion vector to the motion vector selector 400 and a next stage selector. The last selector 336 selects one of its two inputs, i.e., the output from a previous stage selector and the last hierarchical motion vector MV4 in response to the control signal generated by the last low level detector 316, and supplies the selected motion vector to the motion vector selector 400.

The selection signals S1-S4 represent inputs to the motion vector selector 400 so that it selects one of the motion vectors supplied to it as the motion vector for the processing block.

Legend Fig.1

Input Digital Video Signal = digital video input signal

Frame Memory = Image Memory

Block Formatter = Block Formatter

Error Function Generator = Error Function Generator

Predetermined Threshold = predetermined threshold

Small Value Detector = Low Value Detector

Comparator = Comparator

Selector - Selector

Motion Vector Selector = motion vector selector,

Selected Motion Vector = selected motion vector

Fig. 2

From Frame Memory =from frame memory

Search Area Formation = search area formation

Block Formation = Block formation

From Block Formatter = from block formatter

Block Matching = Block Match

Minimum Value Detector = minimum value detector

Selector = Selector

Claims (10)

1. Method for determining a motion vector for each processing block (5) in a current image by using a multi-stage, hierarchical motion estimation in which the current image and a previous image are used, wherein the current image is divided into a number of processing blocks (5) of identical size, each processing block (5) is mapped to M hierarchical search blocks (10, 12, 14, 16), where M is an integer greater than 1, and each hierarchical search block (10, 12, 14, 16) arranged at the center contains the corresponding processing block (5), wherein the size of a search block (10, 12, 14, 16) of a higher hierarchy is larger than that of a search block (10, 12, 14, 16) of a lower hierarchy, and wherein the previous image is divided into several search areas which correspond to the hierarchical search blocks (10, 12, 14, 16), each search area having a plurality of candidate blocks, and each candidate block having a size identical to that of the corresponding hierarchical search block (10, 12, 14, 16), characterized in that the method comprises the steps: (a) performing a motion estimation for each hierarchical search block (10, 12, 14, 16) of a processing block (5) with respect to a corresponding hierarchical search area to thereby provide motion vectors and minimum error functions or values corresponding to the hierarchical search blocks (10, 12, 14, 16), each motion vector representing a pixel movement between each hierarchical search block (10, 12, 14, 16) and a candidate block, in which a corresponding hierarchy results in a minimum error value; (b) comparing the minimum error values with a predetermined threshold value; (c) determining a search block (10, 12, 14, 16) which has a smallest minimum error value among the hierarchical search blocks (10, 12, 14, 16); and (d) if minimum error values of one or more search blocks (10, 12, 14, 16) are smaller than the predetermined threshold value, determining a motion vector corresponding to a search block (10, 12, 14, 16) of a highest hierarchy among the one or more search blocks (10, 12, 14, 16) as the motion vector of the processing block (5), and if none of the minimum error values is smaller than the predetermined threshold value, determining a motion vector corresponding to the search block (10, 12, 14, 16) determined in step (c) as the motion vector of the processing block (5). 2. Method according to claim 1, characterized in that if the minimum error value of only one search block (10, 12, 14, 16) is smaller than the predetermined threshold value, a motion vector corresponding to the one search block (10, 12, 14, 16) is determined as the motion vector of the processing block (5). 3. Method according to claim 1 or 2, characterized in that the sizes of the hierarchical search blocks (10, 12, 14, 16) are varied in 2N pixel steps from 2N to 2N(M-1) in the horizontal and vertical direction of the processing block (5), where N is an integer greater than zero. 4. Method according to claim 3, characterized in that N is equal to 1. 5. Method according to one of the preceding claims, characterized in that the size of each processing block (5) is 2 x 2 pixels. 6. Method according to one of the preceding claims, characterized in that the size of a search block (10, 12, 14, 16) of the lowest hierarchy is equal to that of the processing block (5). 7. Device for determining a set of motion vectors for a set of processing blocks (5) in a current image by using a multi-stage hierarchical motion estimation using the current image and a previous image, the current image being divided into the set of processing blocks (5) of identical size, characterized in that the device comprises: means for sequentially generating each processing block (5) as a target processing block (5); Means (100) for providing M hierarchical search blocks (10, 12, 14, 16) for the target processing block (5), each hierarchical search block (10, 12, 14, 16) containing the target processing block (5) at the center, the size of a search block (10, 12, 14, 16) of one hierarchy being larger than that of a search block (10, 12, 14, 16) of a lower hierarchy, and M being an integer greater than 1; Means (211) for providing M search areas corresponding to the hierarchical search blocks (10, 12, 14, 16) and based on the previous image, each search area having a plurality of candidate blocks, and each candidate block having a size identical to that of the corresponding hierarchical search block (10, 12, 14, 16); Means for performing motion estimation for each hierarchical search block (10, 12, 14, 16) of the target processing block (5) with respect to a corresponding hierarchical search area, thereby providing motion vectors and minimum error functions or values corresponding to the hierarchical search blocks (10, 12, 14, 16), each motion vector representing a pixel motion between each hierarchical search block (10, 12, 14, 16) and a candidate block, which results in a minimum error value for a corresponding hierarchy; Means (322-328) for comparing minimum error values with a predetermined threshold value; Means (312-316) for comparing minimum error values with one another in order to determine among the hierarchical search blocks a search block that has a smallest minimum error value; Means for determining, when only one search block (10, 12, 14, 16) has a minimum error value smaller than the predetermined threshold value, a motion vector corresponding to the one search block (10, 12, 14, 16) as the motion vector of the target processing block (5); and for determining, when one or more search blocks (10, 12, 14, 16) has a minimum error value smaller than the predetermined threshold value, a motion vector corresponding to a search block (10, 12, 14, 16) of a highest hierarchy among the one or more search blocks (10, 12, 14, 16) as the motion vector of the target processing block (5); and then, if no minimum error value is smaller than the predetermined threshold value, to determine a motion vector corresponding to the search block (10, 12, 14, 16) having a minimum error value that has the smallest value as the motion vector of the target processing block (5). 8. Device according to claim 7, characterized in that the sizes of the hierarchical search blocks (10, 12, 14, 16) are varied in 2N pixel steps from 2N to 2N(M-1) in the horizontal and vertical direction of the target processing block (5), where N is an integer greater than zero. 9. Device according to claim 7 or 8, characterized in that the size of each processing block (5) is 2 x 2 pixels. 10. Device according to one of claims 7 to 9, characterized in that the size of a search block (10, 12, 14, 16) of the lowest hierarchy is equal to that of the target processing block (5).