AU750312B2 - Method and apparatus for improved video data motion estimation - Google Patents

Description

I

S&F Ref: 504626

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

S

*5 Name and Address of Applicant: Actual Inventor(s): Address for Service: Invention Title: Canon Kabushiki Kaisha 3-30-2, Shimomaruko Ohta-ku Tokyo 146 Japan Delphine Anh Dao Le Spruson Ferguson St Martins Tower 31 Market Street Sydney NSW 2000 Method and Apparatus for Improved Video Data Motion Estimation ASSOCIATED PROVISIONAL APPLICATION DETAILS [33] Country [31] Applic. No(s) AU PQ0901 [32] Application Date 10 Jun 1999 The following statement is a full description of this invention, including the best method of performing it known to me/us:- Ip Australa Documents received 1-1W fj tc- U 2 D on: N 5815c METHOD AND APPARATUS FOR IMPROVED VIDEO DATA MOTION ESTIMATION Field of Invention The present invention relates to motion estimation in compressed video data and, in particular, to motion-compensated encoded data.

The invention has been developed primarily for producing a field of motion vectors based on motion compensated encoded data, and will be described herein with reference to this specific application. However, it will be appreciated that the invention is not limited to use in this field.

Background Image motion plays an important role in computer vision and scene understanding. Motion vector analysis has been applied to many fields over the last few decades, including object tracking, autonomous navigation, surveillance and virtual reality. More recently, motion information has played an important role in video indexing, contributing to video segmentation and shot classification.

S• Some of these applications require motion information to be estimated as a motion field, where a per-pixel vector field density is required. It is difficult to find a typical motion estimation method that is both efficient and accurate at the same time.

Summary of the Invention It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

According to one aspect of the invention, there is provided a method of generating a field of motion vectors based on motion compensated video data, said method comprising the steps of: providing a first frame and a second frame of said 25 motion compensated video data, wherein the first and second frames each comprise a plurality of macroblocks and each said macroblock of the first frame has an associated S" prediction motion vector referenced to a corresponding macroblock of the second frame; assigning said prediction motion vectors as initial motion vectors of said field of motion vectors associated with the first frame, wherein each said macroblock of the first frame comprises a plurality of first blocks of one or more pixels, where each said first block has an associated said initial motion vector, and wherein the associated initial motion vectors of said first blocks within any one said macroblock are the same as the S prediction motion vector of that macroblock; and modifying said field of initial motion 504626_amendments_01 .doc vectors by performing a relaxation procedure thereon to minimise a predetermined energy function so as to provide said field of motion vectors.

According to another aspect of the invention, there is provided apparatus for generating a field of motion vectors based on motion compensated video data, said apparatus comprising: means for providing a first frame and a second frame of said motion compensated video data, wherein the first and second frames each comprise a plurality of macroblocks and each said macroblock of the first frame has an associated prediction motion vector referenced to a corresponding macroblock of the second frame; means for assigning said prediction motion vectors as initial motion vectors of said field of motion vectors associated with the first frame, wherein each said macroblock of the first frame comprises a plurality of first blocks of one or more pixels, where each said first block has an associated said initial motion vector, and wherein the associated initial motion vectors of said first blocks within any one said macroblock are the same as the prediction motion vector of that macroblock; and means for modifying said field of initial motion vectors by performing a relaxation procedure thereon to minimise a predetermined S• energy function so as to provide said field of motion vectors.

According to another aspect of the invention, there is provided a computer program for generating a field of motion vectors based on motion compensated video data, said computer program comprising: means for providing a first frame and a second frame of said motion compensated video data, wherein the first and second frames each comprise a plurality of macroblocks and each said macroblock of the first frame has an associated prediction motion vector referenced to a corresponding macroblock of the second frame; means for assigning said prediction motion vectors as initial motion vectors of said field of motion vectors associated with the first frame, wherein each said 25 macroblock of the first frame comprises a plurality of first blocks of one or more pixels, where each said first block has an associated said initial motion vector, and wherein the associated initial motion vectors of said first blocks within any one said macroblock are the same as the prediction motion vector of that macroblock; and means for modifying said field of initial motion vectors by performing a relaxation procedure thereon to minimise a predetermined energy function so as to provide said field of motion vectors.

Other aspects of the invention are also disclosed.

Brief Description of Drawings Preferred embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which: 504626_amendments_01 .doc -3- Fig. 1 shows a general-purpose computer for implementing preferred embodiments of the invention; Fig. 2 shows a flow chart setting out steps involved in implementing the preferred embodiment of the invention; Fig. 3 shows a frame divided into a plurality of macroblocks; Fig. 4 shows a macroblock from the frame in Fig. 3, divided into sub-blocks; Fig. 5 shows the sub-blocks from Fig. 4 with energy-minimised vectors applied to each of the sub-blocks; and Fig. 6 shows a sub-block from Fig. 5 subdivided into further sub-blocks.

C

*o **o

C.

504626 amendments 01.doc -4- Detailed Description Recent compression standards for moving images, like H.261 and MPEG, have adopted a macroblock approach for motion compensation. A target macroblock in a frame to be encoded is matched with a most similar displaced macroblock in a previous (or consecutive) frame, called a reference image. The position of the best matching

S

S

*o* 504626 amendments 01.doc macroblock, or prediction macroblock, is indicated by a motion vector that describes a displacement from the target macroblock to the prediction macroblock. The motion vector information is encoded and transmitted along with compressed image frames. In the H.261 and MPEG standards, the macroblock size is chosen to be 16x16 pixels, representing a tradeoff between motion-compensated compression and the cost of transmitting the motion vectors.

Using the preferred embodiment of the invention, it is possible to take advantage of the motion information embedded in the compressed data in order to compute a relatively dense and more accurate motion field.

Preferred embodiments of the invention are implemented on a conventional general-purpose (host) computer system, such as the computer system 40 shown in Fig. 1.

The application program discussed above (and thereby the preferred embodiments of the invention described with reference to the other drawings) is implemented as software executed on the computer system The computer system 40 includes a computer module 41, input devices such as a keyboard 42 and mouse 43, and output devices including a printer 56 and a display device 57. A modulator-demodulator ("modem") transceiver device 52 is used by the computer module 41 for communication via a telecommunications network 61, such as a telephone line, Local Area Network or Wide Area Network The computer module 41 comprises an 1/O Interface 10 for coupling the computer module 41 to the computer Network 61. The modem 52 can also be used to access the Internet or other telecommunications networks.

The computer module 41 includes a number of functional components, including (in the embodiment shown): a processor unit a memory unit 46 including semiconductor Random Access Memory ("RAM") and Read Only Memory and input/output interfaces including a video interface 47; an 1/0 interface 48 for the keyboard 42 and mouse 43; 504626.doc -6a storage device 49, including a hard disk drive 53 and a floppy disk drive 54; and a CD-ROM drive 55 for use as a non-volatile source of data.

The components 45 to 49 and 53 to 55 of the computer module 41 communicate via an interconnecting bus 50 in a manner that results in conventional operation of the computer system 40, as known to those in the relevant art. Examples of computers on which the embodiments can be practised include IBM-PC's and compatibles, and Sun Sparcstations.

Typically, the application program of the preferred embodiment is resident on a hard disk drive 53 and read and controlled using the processor 45. Intermediate storage o. of the program and the print list and any data fetched from the network may be S-accomplished using the semiconductor memory 46, possibly in concert with the hard disk drive 53. In some instances, the application program may be supplied to the user encoded on a CD-ROM or floppy disk, or alternatively could be read by the user from the network 15 via the modem device 52.

Depending upon the implementation, it may be desirable to use a hardwarebased graphics accelerator to improve performance of the system as a whole.

A flowchart 200 in Fig. 2 sets out the steps involved in implementing the preferred embodiment of the invention. In an initial step 201, decoded frames from an MPEG stream are provided. At least a first frame 300 (Fig. 3) and a second frame (not shown) are provided. Each frame 300 consists of macroblocks 301, which, in the case of frames originally MPEG encoded, are 16x16 pixels in size.

Each frame is based on either an I frame having DCT encoded macroblocks, or an intermediate B or P frame. Where the frame 300 is based on a B or P frame, each macroblock is based on a best match macroblock from a preceding or subsequent frame in the originally encoded series of video frames. The best matching macroblock vector is referred to by means of a motion vector 302. The motion vector 302 represents a relative coordinate offset, the resultant position of which represents the best match macroblock in a preceding or subsequent frame (the second frame referred to in this specification).

504626.doc In step 202, a motion field is generated covering the whole of the frame 300, as shown in Fig. 4. For this step, each of the macroblocks 301 is considered a sub-block associated with the first frame. Each sub-block (macroblock) 301 is defined by a plurality of pixels 303, which represent, in this case, the desired resolution of the motion field.

Within each sub-block (macroblock) 301, each pixel 303 is initialised with the vector 302 having a value equal to that of the corresponding sub-block (macroblock) 301, as shown in Fig. 3.

In step 203, a level index k is initialised with a value N, the purpose of which will become apparent from the following paragraphs. It will be appreciated that the precise point in the method at which k is initialised is not critical. For example, k could be initialised at the very beginning of the procedure prior to step 201.

A relaxation at level k is then performed in step 204. Relaxation in the present context refers to the iterative implementation of an energy function E to each of the pixels 303, until E is minimised. The implementation of the energy function is discussed in 5 greater detail below.

In step 205 an assessment is made as to whether k=0. If the answer is affirmative, the method ends at step 206. If not, the method proceeds to step 207, in which k is decremented. Decrementing k amounts to dividing the sub-block 301 into further sub-blocks 304 (Fig. 6).

The method then returns to step 204, and repeats the relaxation for the new value ofk. In effect, changing the value ofk reduces the size of the sub-block within which the motion vectors are kept the same as each other. Steps 204, 205 and 207 are then repeated until k=0 and the process ends. At that point, each pixel has associated with it an individual motion vector that is not constrained to be the same as the motion vectors associated with any of its neighbouring pixels.

As discussed above, encoded video data have a motion vector per macroblock, those vectors being defined hereinafter as Macroblock Motion Vectors (MMV's). Scene understanding or computer vision tasks which use motion information often require a 504626.doc dense and accurate motion vector field, sometimes to the level of one motion vector per pixel; motion-based segmentation is one such case.

When using an iterative method, using macroblock motion vectors to provide an initial guess, in accordance with the preferred embodiment, can improve the speed and accuracy of convergence. This is particularly appropriate if the refined motion estimation method is a hierarchical approach which performs a coarse-to-fine motion estimation, for example, the multiscale relaxation algorithm proposed by Heitz et al. Heitz, P. Perez and P. Bouthemy, "Multiscale minimisation of global energy functions in some visual recovery problems", CVGIP: Image Understanding, vol.59, pp. 125-134, 1994].

In implementation, the preferred embodiment minimizes a global energy function, in which the energy function models desirable properties of the motion field. Such S"properties can include, for example, intensity invariance under motion, or smoothness of the motion field. These two exemplary constraints can be applied by formulating the energy as the weighted sum of two terms. An energy function E can then be written as: 2 +a jj 112 seS <s,p>EC wherein: S is the set of all pixels s in a frame; U is the set of motion vectors u(s); I is the image intensity at time t; It+, is the image intensity of the subsequent frame; c is a weighting factor; is a pair of neighbouring pixels; and C is the set of all neighbouring pixels.

It will be appreciated by those skilled in the art that any other suitable constraint or constraints can also be utilised in an energy function. Multiple constraints are usually dealt with by applying a weighting factor to one or more of the constraints within the energy function. This allows manipulation of the relative contribution of each constraint to the value E of the energy function.

504626.doc -9- Preferably the weighting factor ax is optimised for each image. Otherwise it can be arbitrarily set. For instance, it can be set to a high value, e.g. 100 which tends to result in a smooth motion vector field.

The energy function is minimised over nested subspaces of the original space of possible solutions, these subspaces consisting of solutions constrained at different scales. For example, scale 0 corresponds to the original configuration space, whilst scale k k k corresponds to configurations which are constant over blocks of size 2kx2 k Letting N be the predetermined number of scales, and following a coarse-to-fine strategy, the preferred energy minimisation method starts at level N. A relaxation is performed in order to find a motion field that is constant over blocks of size 2 Nx 2 N and minimises the global energy function E. At level N-l, each parent block of level N is S•divided into 4 child blocks (sub-blocks), and the motion field is then constrained to be constant over the child blocks (sub-blocks) of size 2N' X2N-'. The estimate obtained at level N defines a crude solution which can be used as initial guess at level N-1. The energy minimisation is performed again at level N-l, and so on, until level 0 is reached.

In the preferred embodiment, at each level, the energy minimisation is performed using an iterative deterministic relaxation process known as Iterated Conditional Modes This optimisation method converges relatively quickly but towards the closest local minimum. For this reason, the initial "guess" provided by the motion vectors associated with the encoded macroblock data is an important aspect of this embodiment in that it leads to fast and reliable results.

If the coarsest chosen scale corresponds directly to the block size used for motion-compensated coding (eg N=4 for MPEG), then the MMV can directly be used as the initial values for the multiscale motion estimation algorithm. It will be appreciated, on the other hand, that the scale chosen need not correspond directly with such a block size. Further, it is not strictly necessary that square blocks be used, nor that the block division process be exponential in nature.

504626.doc Further, when bidirectional temporal prediction is used as in MPEG, two MMVs may be available for one block. That being the case, the two available MMVs can be combined or averaged to obtain a unique motion vector for each block.

Another potential use of the MMV is to provide an indicator of the motion magnitude, thereby enabling a reduction of the search space required for the refined motion estimation. In this aspect, for a given macroblock, only a subset of the motion vector space centered around the compressed motion vector needs to be explored. It will be appreciated by those skilled in the art that the range of the search space is an important factor in the time complexity of the algorithm, especially for relaxation methods like the 10 ICM algorithm. For example, the boundaries of the search space can be defined .oeooi depending on the range spanned by the MMVs. For instance, the search space can be of *the order of the largest possible magnitude of the MMVs. This can be done on a global or local basis. Thus the complexity of the calculation of the minimisation of the energy

C

function can be reduced.

Both aspects of the preferred embodiment (ie, the use of an initial "guess" and restriction of the search space) can considerably reduce the processing time of the consequent motion estimation procedure, by accelerating the convergence of the motion estimation algorithm and yielding a better final estimate.

The foregoing describes only one embodiment/embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the spirit or scope of the invention. The present embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive.

In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including" and not "consisting only of'. Variations of the word comprising, such as "comprise" and "comprises" have corresponding meanings.

504626.doc

Claims (12)

1. A method of generating a field of motion vectors based on motion compensated video data, said method comprising the steps of: providing a first frame and a second frame of said motion compensated video data, wherein the first and second frames each comprise a plurality of macroblocks and each said macroblock of the first frame has an associated prediction motion vector referenced to a corresponding macroblock of the second frame; assigning said prediction motion vectors as initial motion vectors of said field of motion vectors associated with the first frame, wherein each said macroblock of the first frame comprises a plurality of first blocks of one or more pixels, where each said first block has an associated said initial motion vector, and wherein the associated initial motion vectors of said first blocks within any one said macroblock are the same as the prediction motion vector of that macroblock; and 15 modifying said field of initial motion vectors by performing a relaxation ooooo procedure thereon to minimise a predetermined energy function so as to provide said field of motion vectors.

2. A method as claimed in claim 1, wherein the modifying step is performed iteratively.

3. A method as claimed in claim 2, wherein the modifying step comprises the sub- steps of: performing the relaxation procedure over the macroblocks to determine 25 a said field of motion vectors constant over the corresponding macroblocks; ooo dividing said macroblocks into sub-blocks; performing the relaxation procedure over the sub-blocks to determine a said field of motion vectors constant over the corresponding sub-blocks; dividing the sub-blocks into further sub-blocks; repeating steps and until the sub-blocks reach the size of the first blocks; and performing the relaxation procedure over the first blocks to determine a -3 final said field of motion vectors. 504626_amendmentsOl.doc -12-

4. A method according to claim 1, wherein the relaxation procedure includes the step of minimising a global energy function E(U) based on one or more variable features associated with the video data.

5. A method according to claim 4, wherein: 2 a 12 seS <s,p>eC and: S is the set of all pixels s in a frame; U is the set of motion vectors u(s); I is the image intensity at time t; It+1 is the image intensity of the subsequent frame; ac is a weighting factor; is a pair of neighbouring pixels; and C is the set of all neighbouring pixels.

6. A method according to claim 3, wherein the macroblocks (and thereby the initial sub-blocks) are of size 2 Nx 2 N and steps c(2) and c(4) include the step of dividing the macroblocks or sub-blocks into four further sub-blocks of equal size.

7. A method according to claim 1, wherein the first blocks are a single pixel in size. S: 8. A method as claimed in claim 1, wherein the relaxation procedure has a search space, the range of which is of the order of the largest magnitude of the prediction vectors o on a local or global basis. S9. Apparatus for generating a field of motion vectors based on motion compensated video data, said apparatus comprising: means for providing a first frame and a second frame of said motion compensated video data, wherein the first and second frames each comprise a plurality of macroblocks and each said macroblock of the first frame has an associated prediction motion vector referenced to a corresponding macroblock of the second frame; means for assigning said prediction motion vectors as initial motion vectors of said field of motion vectors associated with the first frame, wherein each said macroblock 504626_amendments_01 .doc

13- of the first frame comprises a plurality of first blocks of one or more pixels, where each said first block has an associated said initial motion vector, and wherein the associated initial motion vectors of said first blocks within any one said macroblock are the same as the prediction motion vector of that macroblock; and means for modifying said field of initial motion vectors by performing a relaxation procedure thereon to minimise a predetermined energy function so as to provide said field of motion vectors. Apparatus as claimed in claim 9, wherein the modifying means is performed iteratively. 11. Apparatus as claimed in claim 10, wherein the modifying means comprises: means for performing the relaxation procedure over the macroblocks to determine a said field of motion vectors constant over the corresponding macroblocks; means for dividing said macroblocks into sub-blocks; S: means for performing the relaxation procedure over the sub-blocks to determine a said field of motion vectors constant over the corresponding sub-blocks; means for dividing the sub-blocks into further sub-blocks; means for repeating the operations of the means for performing the relaxation procedure over the sub-blocks and the means for dividing the sub-blocks until the sub- blocks reach the size of the first blocks; and means for performing the relaxation procedure over the first blocks to determine a final said field of motion vectors. 25 12. Apparatus according to claim 9, wherein the relaxation procedure includes means for minimising a global energy function E(U) based on one or more variable features associated with the video data. 13. Apparatus according to claim 12, wherein: E(U) +a C 12 sES <s,p>EC and: S is the set of all pixels s in a frame; U is the set of motion vectors u(s); 504626_amendments_0Oldoc -14- I is the image intensity at time t; It,+ is the image intensity of the subsequent frame; cc is a weighting factor; is a pair of neighbouring pixels; and C is the set of all neighbouring pixels.

14. Apparatus according claim 11, wherein the macroblocks (and thereby the initial sub-blocks) are of size 2N x 2 N, and the means for dividing said macroblocks and the means for dividing the sub-blocks include means for dividing the macroblocks or sub- blocks into four further sub-blocks of equal size respectively. Apparatus according to claim 9, wherein the first blocks are a single pixel in size. 15 16. Apparatus according to claim 9, wherein the relaxation procedure has a search o o space, the range of which is of the order of the largest magnitude of the prediction vectors on a local or global basis.

17. A computer program for generating a field of motion vectors based on motion compensated video data, said computer program comprising: means for providing a first frame and a second frame of said motion compensated video data, wherein the first and second frames each comprise a plurality of S. macroblocks and each said macroblock of the first frame has an associated prediction S motion vector referenced to a corresponding macroblock of the second frame; means for assigning said prediction motion vectors as initial motion vectors of said field of motion vectors associated with the first frame, wherein each said macroblock of the first frame comprises a plurality of first blocks of one or more pixels, where each said first block has an associated said initial motion vector, and wherein associated initial motion vectors of said first blocks within any one said macroblock are the same as the prediction motion vector of that macroblock; and means for modifying said field of initial motion vectors by performing a relaxation procedure thereon to minimise a predetermined energy function so as to provide said field of motion vectors. 504626 amendments_01 .doc

18. A method of producing a field of motion vectors, the method substantially as described herein with reference to the accompanying drawings.

19. Apparatus for producing a field of motion vectors, the apparatus substantially as described herein with reference to the accompanying drawings. A computer readable medium comprising a computer program for producing a field of motion vectors, the computer program substantially as described herein with reference to the accompanying drawings. DATED this twenty-first Day of May, 2002 Canon Kabushiki Kaisha Patent Attorneys for the Applicant SPRUSON FERGUSON **S *•g 504626 amendments_01.doc