EP0293644A1 - Method for determining movement vector fields from digital image sequences - Google Patents

Abstract

This method determines, from in each case two successive frames, a movement vector field which allocates to each picture element of one frame a picture element of the other frame, the allocation in each case being defined by a movement vector which reproduces the relative displacement of the picture elements with respect to one another and in which in each case all picture elements in a square or rectangular block of picture elements are given the same movement vector. The movement vectors are determined by minimising a composite target function which, on the one hand, takes into consideration the differences between the luminance values of the mutually allocated picture elements of the two given frames and, on the other hand, evaluates the differences between movement vectors, the coordinates of which are adjacent, namely so-called neighbour movement vectors, with the aid of a smoothness measure. This target function is minimised in such a manner that first the movement vectors minimising the target function are determined with the restriction that the movement vectors in larger blocks than those ultimately required are constant and that then each of these blocks (16x16) is subdivided into smaller, preferably equally-large, blocks until the required block size (4x4) is reached. <IMAGE>

Description

The present invention relates to a method for determining motion vector fields from digital image sequences, which calculates a motion vector field from each two successive images, which associates each pixel of an image with an image point of the other image, the assignment being defined in each case by a motion vector, which is the reproduces relative displacement of the pixels relative to each other, and each pixel in a square or rectangular block of pixels receives the same motion vector.

For different applications, e.g. B. image data compression or machine vision (robot, scene analysis), it is necessary to automatically detect the existing in a digital image sequence shifts in image content from image to image, which result from object movements or camera movements. These shifts in the local image content can be represented by motion vector fields which, for. B. for each pixel of an image indicate how much the image content has shifted at this point compared to the previous image.

The motion vector fields are e.g. B. used in image data compression for the purpose of transmitting digital images with low data rates to predict the next, not yet transmitted image from already transmitted images. The better this prediction is, the smaller the data rate that is required for the transmission of the new image, or with the better image quality, the new image can be transmitted at a predetermined data rate.

Another application of the motion vector fields is the reconstruction of missing images from an image sequence that was undersampled in time for the purpose of data compression. That means that e.g. B. only every third image of the sequence is available and the two missing images between two existing images (the "reference point images") are to be interpolated as "movement-correctly" as possible, so that the movements of objects in the reconstructed scene run as smoothly as in the original . For this purpose, motion vector fields are required which indicate for each pixel of an image to be interpolated which image point or which image points are to be used in the two associated reference point images for the reconstruction of the relevant pixel.

In any case, a motion vector is assigned to each pixel of an image or also to a group of adjacent pixels in the motion vector fields, which motion vector describes the local motion by two components, namely the horizontal and the vertical motion component.

A problem with the determination of such motion vector fields is that the movements present in an image sequence are generally dependent on the location of the pixels, so that several different motion vectors can occur in a small image section, in particular at the edges of moving objects. Therefore, to determine a motion vector for a certain pixel, only this pixel itself should actually be considered. On the other hand, a motion vector cannot be determined from a single pixel, if only because the motion vector contains two components and each individual pixel defines only one equation for these two unknowns, cf. BBKP Horn, BG Schunck: "Determining optical flow", Artificial Intelligence 17, pp. 185-203, 1981. But is also in a small environment around the pixel the content of the picture is often structured so little that the movement at the location of the pixel in question cannot be clearly determined. This results in the difficulty that, on the one hand, only small surroundings are to be used for calculating a movement vector in areas with movement vectors that are strongly dependent on the location, and on the other hand, large areas are required in areas with not clearly structured image contents in order to be able to clearly recognize the movement. It is therefore necessary to vary the size of the respective environment, and it must also have previous knowledge, namely z. B. the assumption of a certain smoothness of the motion vector field can be used in order to obtain motion vectors that are useful for the above applications even with noisy images and in little differentiated image sections.

• 1) Block-Matching-Verfahren,
• 2) Differentielle Verfahren,
• 3) Verfahren, die mit markanten Punkten arbeiten.

So far, three different approaches to motion vector estimation have been investigated. BHG Musmann, P. Pirsch, H.-J. Grallert: "Advances in Picture Coding", Proc. IEEE 73 (1985) 4, pp. 523-548

• 1) block matching method,
• 2) differential methods,
• 3) Procedures that work with distinctive points.

The mode of operation of these methods is briefly explained below in the event that the shift of the image contents with respect to the previous image (image A) is to be determined for an image (for example image B) of an image sequence.

To 1):

In block matching methods, the image for which motion vectors are to be determined is divided into square or rectangular blocks of constant size, ie with a predetermined number of pixels (often 16x16 or 8x8), cf. BCM Lin, SC Kwatra: "Motion compensated interframe color image coding ", Proc. Int. Conf. on Communications, 1984, vol. 1, pp. 516-520 and H. Brusewitz, P. Weiss:" A videoconference system at 384 kbit / s ". Picture Coding Symposium, Tokyo , Abstracts p. 212, 1986. The same motion vector is determined for all pixels in a block, namely on the assumption that approximately the motion in the small image section that corresponds to a block is constant.

Here, the motion vector for a BLock in image B is determined by picking out the block in image A that contains the pixels shifted by the motion vector from image A for a large number of possible motion vectors in a predetermined value range, and that from the plurality of these blocks, the one is selected that has the least difference to the given block in image B. The difference between two blocks in Figures A and B is determined by a suitable distance, namely z. B. the sum of the squares (L2 norm) or the sum of the amounts (L1 norm) of the pixel differences, expressed. The motion vector in which the two blocks from image A and image B have the smallest distance is adopted in the motion vector field sought.

The problem here - as stated at the beginning - is the selection of the appropriate block size: If the blocks are too large, the motion vector field becomes too coarse and inaccurate because the assumption of constant movement in the individual blocks is no longer correct; if the blocks are too small, the image content is often too undifferentiated to allow the correct object movement to be recognized. In the publication G. Kummerfeldt, F. May, W. Wolf: "Coding television signals at 320 and 64 kbit / s", Image Coding, M. Kunt, TS Huang (editor), Proc. SPIE 594, pp. 119-128, 1985, an attempt is made to retroactively combine and smooth the motion vectors of several blocks that are classified as belonging to an object solve the incorrectly estimated vectors in blocks with ambivalent image content. The results show, however, that this approach only improves the overall movement of the image content with very simple to describe overall movements of the image content, such as "camera zoom: without additional object movements - in this case for use in motion-adaptive prediction in image data compression.

to 2):

In the differential method, cf. B. P. Robert, C. Cafforio, F. Rocca: "Time / space recursions for differential motion estimation," Image Coding, M. Kunt, T.S. Huang (Editor), Proc. SPIE 594, pp. 175-185, 1986, the assumption of a constant movement for a block of neighboring pixels is dropped and instead a separate motion vector is determined for each pixel. For this purpose, certain model parameters are calculated for each pixel, which describe the local course of the image signal in the vicinity of the pixel, and from these parameters and from the difference in the image contents of images A and B at the location of the pixel under consideration, the underlying movement is d. H. the shift of the image contents against each other, closed. Since this estimate of the movement is initially only an approximate solution, the procedure is continued iteratively until there is no further improvement in the movement vector.

A problem with this method is that the description of the image content as model parameters is only valid within narrow limits and z. B. fails with large shifts between image A and image B. Furthermore, the environment of the image point is used to calculate the model parameters for a pixel, again the uniformity of the movement in this environment is presupposed, so that the choice of the size of this environment poses the same problems as the choice of block size in block matching processes. The fact that the surroundings of the pixels overlap each other results in vector fields that change only slightly from one pixel to the next and therefore do not correctly represent jumps in the movement that occur at object boundaries.

In the publication B.K.P. Horn, B.G. Schunck: "Determining optical flow", Artificial Intelligence 17, pp. 185-203, 1981, the question is discussed as to how it can be ensured with these differential methods that even in homogeneous image areas which do not permit unambiguous motion detection, meaningful motion vector fields largely corresponding to reality can be determined. It is proposed to include in the objective function to be minimized a term that expresses the smoothness of the motion vector field that arises. Because of the design of this part of the target function, which measures the smoothness of the motion vector field - the quadratic norm of the so-called "Laplacian" of the vector field is measured, above all because this results in an analytical, mathematically easy-to-use function - problems of the method at object boundaries arise: jumps in the Motion vector fields are incorrectly suppressed.

To 3):

In the third type of method an attempt is made to circumvent the problem that the actual object movement is often not clearly recognizable from the local image content by first searching for striking points ("gray value corners") or also lines (brightness edges) in the image and a motion vector is determined only for these points or along the lines, cf. BR Lenz: "Estimation of 2-D general motion parameters in TV scenes", Proc. 7th Int. Conf. Pattern Rec., Montreal, Canada, July 30-Aug. 2, 1984. Vol. 1, pp. 546-548 and CJ Radford: "Optical flow fields in Houch transform space", Pattern Recognition Letters 4, pp. 293-303, 1986. Then, for the rest of the pixels, the motion vector field must be interpolated from the given motion vectors using suitable means. Problematic here are the reliable finding of the distinctive points or lines, for which the motion vectors are initially determined, and the segmentation of the image into areas with uniform movement, which can be determined from the given vectors of the distinctive points or lines by interpolation. Because of the difficulty of these subtasks, these methods are practically only suitable for image sequences with rigid bodies, such as, for. B. vehicles, but not for processing scenes with moving people, as they often occur in image data compression.

Smoothing operators have also been developed for these methods, cf. B. H.-H Nagel, W. Enkelmann: "An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences". IEEE Trans. PAMI-8/5, pp. 565-593, Sept. 1986, again based on the quadratic norm of a smoothness function derived from the motion vector field. Since this creates the known problems at object boundaries, it was proposed in this work to set up a "directed smoothness requirement" which should only smooth the motion vector field perpendicular to the gradient of the brightness in the image in question. However, the resulting process is very complex.

The present invention has for its object to provide a novel method of the type mentioned, by means of which motion vector fields can be determined from a given image sequence, special measures being taken to ensure that the motion vector fields reproduce the motion actually present in the image as far as possible.

The object is achieved by a method for determining motion vector fields from digital image sequences, which determines a motion vector field from two successive images, which assigns a pixel of the other image to each pixel of one image, the assignment being defined in each case by a motion vector which defines the relative Displacement of the pixels against each other, and where each pixel in a square or rectangular block of pixels receive the same motion vector, solved, which is characterized according to the invention in that the determination of the motion vectors is carried out by minimizing a composite objective function, which on the one hand the differences of the luminance values of the pixels assigned to one another of the two given images is taken into account, and secondly the differences between motion vectors whose coordinates are adjacent, namely so-called neighboring motion vectors , evaluated with the aid of a smoothness measure, and that the minimization of this objective function is carried out in such a way that first the motion vectors minimizing the objective function are determined with the restriction that the motion vectors are constant in larger than the finally aimed blocks, and then each of these Blocks are subdivided into smaller, preferably equally large, blocks until the desired block size is reached, the objective function being minimized by varying the motion vectors after each reduction of the blocks.

The method according to the invention is based on the principle of block matchine explained above, see HG Musmann et al, as stated above, ie a motion vector is generated for each block of pixels by evaluating a target function for various possible vectors and searching for that Motion vector that delivers the optimum of the target function is determined.

Advantageous developments of the invention are characterized by the features specified in the subclaims.

• 1) Um ein Bewegungsvektorfeld mit hoher Auflösung zu erhalten, das auch an Objektgrenzen die tatsächliche Bewegung mit hoher Genauigkeit beschreibt, wird das Bild in kleine Blöcke von z. B. 4*4 Bildpunkten (Pixel) unterteilt, für die jeweils ein Bewegungsvektor ermittelt wird. Jedoch wird zum Zwecke der Überwindung der Nachteile der kleinen Blöcke (möglicherweise ambivalenter, nicht genügend charakteristischer Bildinhalt) zunächst in einem ersten Schritt des Verfahrens eine Bewegungs­vektor-Ermittlung für erheblich größere Blöcke (z. B. 16*16 oder 32*32 Bildpunkte) durchgeführt. Diese großen Blöcke werden in weiteren Schritten des Verfahrens in kleinere Blöcke unter­teilt, wofür jeweils ein eigener Bewegungsvektor ermittelt wird, wobei die Bewegungsvektoren der großen Blöcke als Aus­gangspunkt und Hilfsmittel dienen. (ABNEHMENDE BLOCKGRÖSSE).
• 2) Anstelle der üblichen Zielfunktionen von Block-Matching-­Verfahren, die nur die Differenzen der Bildpunktwerte der aufeinanderfolgenden Bilder zum Ausdruck bringen, werden in dem erfindungsgemäßen Verfahren Zielfunktionen verwendet, die durch geeignete ZUSATZTERME auch die "GLATTHEIT" des Bewegungsvektor­feldes berücksichtigen. Insbesondere wird im ersten Schritt des Verfahrens bei der Initialisierung des Bewegungsvektorfeldes mit großer Blockgröße zunächst die Länge der einzelnen Be­wegungsvektoren mit in die minimierende Zielfunktion ein­bezogen. In den Folgeschritten des Verfahrens werden dann die Unterschiede "benachbarter" Bewegungsvektoren, d. h. der Bewegungsvektoren benachbarter Blöcke, in die Zielfunktion mit hineingenommen. Durch diese Art der Zielfunktion kann eine Glättung des Bewegungsvektorfeldes und eine Unterdrücking von Ermittlungsfehlern, die als "Ausreißer" hervortreten, bewirkt werden, und zwar auch bei der Ermittlung der Bewegung von Ob­jekten, die ihre Form ändern können, wie z. B. Personen.
• 3)Die bei anderen Verfahren (differentiellen sowie mit mar­kanten Punkten arbeitenden Verfahren) im Zusammenhang mit Glattheitsmaßen auftauchenden Probleme an Objektkanten, wo Sprünge im Bewegungsvektorfeld möglich sein müssen, werden in dem erfindungsgemäßen Block-Matching-Verfahren dadurch ver­mieden, daß das Glattheitsmaß, welches die Unterschiede be­nachbarter Bewegungsvektoren mißt, nicht auf der quadratischen Norm, sondern auf der BETRAGSNORM der Differenzen beruht. Der Effekt ist ähnlich einer MEDIAN-Filterung des Bewegungsvektor­feldes, bei der auch ausgeprägte Sprünge im Bewegungsvektorfeld erhalten bleiben und nur sog. "Ausreißer" unterdrückt werden. Dieses Glattheitsmaß ermöglicht daher auch das korrekte Er­mitteln von Bewegungsvektorfeldern an Objektkanten.
• 4) Speziell in der Bildcodierung zum Zwecke der Bilddaten­kompression kann das Verfahren in der Weise angewendet werden, daß sendseitig zunächst nur mit größeren Blöcken gearbeitet wird und die zugehörigen Bewegungsvektoren zum Empfänger übertragen werden. Diese Bewegungsvektoren werden zur be­wegungskompensierenden Prädiktion von Bildern im Sender (Coder) und Empfänger (Decoder) verwendet. Zusätzlich werden auf der Empfängerseite mit Hilfe der empfangenen BIlder iterativ die noch fehlenden Verfeinerungschritte für das Bewegungsvektor­feld durchgeführt, bis die gewünschte kleinste Blockgröße erreicht ist. DIeses Bewegungsvektorfeld kann dann dazu ver­wendet werden, fehlende BIlder der Bildsequenz, die beim Sender zwecks Datenreduktion übersprungen wurden, bewegungsrichtig zu INTERPOLIEREN.

To overcome the problems discussed above, the following new principles have been incorporated into the motion vector detection method according to the invention:

• 1) In order to obtain a motion vector field with high resolution, which also describes the actual motion with high accuracy at object boundaries, the image is divided into small blocks of e.g. B. 4 * 4 pixels (pixels), for each of which a motion vector is determined. However, in order to overcome the disadvantages of the small blocks (possibly ambivalent, not sufficiently characteristic picture content), a motion vector determination for considerably larger blocks (e.g. 16 * 16 or 32 * 32 pixels) is carried out in a first step of the method . In further steps of the method, these large blocks are subdivided into smaller blocks, for each of which a separate motion vector is determined, the motion vectors of the large blocks serving as a starting point and aids. (DECREASING BLOCK SIZE).
• 2) Instead of the usual target functions of block matching methods, which only express the differences in the pixel values of the successive images, target functions are used in the method according to the invention, which also take into account the "SMOOTHNESS" of the motion vector field by means of suitable ADDITIONAL TERMS. In particular, in the first step of the method, when the motion vector field is initialized with a large block size, the length of the individual motion vectors is first included in the minimizing target function. In the subsequent steps of the method, the differences between "adjacent" motion vectors, ie the motion vectors of adjacent blocks, are then included in the target function. This type of objective function can smooth the motion vector field and suppress Detection errors that appear as "outliers" are caused, including when determining the movement of objects that can change their shape, such as. B. People.
• 3) The problems that occur with other methods (differential methods as well as methods with distinctive points) in connection with smoothness measures at object edges, where jumps in the motion vector field must be possible, are avoided in the block matching method according to the invention in that the smoothness measure which the Measures differences of neighboring motion vectors, not based on the quadratic norm, but on the AMOUNT of the differences. The effect is similar to MEDIAN filtering of the motion vector field, in which pronounced jumps in the motion vector field are retained and only so-called "outliers" are suppressed. This smoothness measure therefore also enables the correct determination of motion vector fields at object edges.
• 4) Especially in image coding for the purpose of image data compression, the method can be used in such a way that on the transmission side only larger blocks are initially used and the associated motion vectors are transmitted to the receiver. These motion vectors are used for motion-compensating prediction of images in the transmitter (coder) and receiver (decoder). In addition, the missing refinement steps for the motion vector field are carried out iteratively on the receiver side with the aid of the received images until the desired smallest block size is reached. This motion vector field can then be used to INTERPOLATE missing images of the image sequence that were skipped at the transmitter for the purpose of data reduction.

• Fig. 1 zeigt eine schematische Darstellung des Ablaufs des Bewegungsvektor-Ermittlungsverfahrens für eine endgültige Blockgröße 4*4 und eine Startblockgröße 16*16.
• Fig. 2 zeigt eine schematische Darstellung der Art und Weise einer durchzuführenden Blockunterteilung.
• Fig. 3 zeigt die Darstellung eines Bewegungsvektors X16(m,n) mit den Bewegungsvektoren der vier Nachbar­blöcke.
• Fig. 4 zeigt ein Flußdiagramm für den Gesamtablauf des Verfahrens.
• Fig. 5 zeigt ein erstes Unter-Flußdiagramm, das den Ablauf der Initialisierung des Verfahrens verdeutlicht.
• Fig. 6 zeigt ein zweites Unter-Flußdiagramm, das den Ablauf der Iterationsschritte innerhalb des Verfahrens verdeut­licht.
• Fig. 7 zeigt ein drittes Unter-Flußdiagramm, das den Ablauf eines Optimierungsvorgangs innerhalb des Verfahrens verdeutlicht.
• Fig. 8 zeigt eine blockschaltbildartige Darstellung des Bewegungsvektor-Ermittlungsverfahrens.
• Fig. 9 zeigt ein Blockschaltbild betreffend eine Bildsequenzen-­Übertragungsanordnung, die Einrichtungen zur Durch­führung des erfindungsgemäßen Verfahrens enthält.

The invention is described in detail below with reference to several figures.

• 1 shows a schematic representation of the sequence of the motion vector determination method for a final block size 4 * 4 and a starting block size 16 * 16.
• Fig. 2 shows a schematic representation of the manner of a block division to be carried out.
• 3 shows the representation of a motion vector X16 (m, n) with the motion vectors of the four neighboring blocks.
• 4 shows a flow chart for the overall sequence of the method.
• FIG. 5 shows a first sub-flow diagram which illustrates the process of initializing the method.
• FIG. 6 shows a second sub-flow diagram which clarifies the sequence of the iteration steps within the method.
• FIG. 7 shows a third sub-flow diagram which clarifies the course of an optimization process within the method.
• FIG. 8 shows a block diagram representation of the motion vector determination method.
• FIG. 9 shows a block diagram relating to an image sequence transmission arrangement which contains devices for carrying out the method according to the invention.

The method is described below on the basis of a preferred exemplary embodiment for the case in which a motion vector field is determined from two images - image A and image B - one for blocks of 4 * 4 pixels (pixels) uniform motion vector field is defined. The process begins with larger blocks that contain several small blocks. In the present example, blocks with the size 16 * 16 are started.

The entire process sequence then follows a scheme as shown in FIG. 1.

The process steps are described below.

Block matching 16 * 16 (INIT16)

The two successive images A and B of a sequence are given, which consist of pixels
a (k, h), k = 1 ... I, h = 1 ... J (1.1a)
and
b (i, j), i = 1 ... I, j = 1 ... J (1.1b)
exist, where i and k is the row index, j and h is the column index.

The aim of the entire method is to determine a motion vector field X (i, j), i = 1 ... I, j = 1 ... J. (1.1c)

Image B is now in blocks
B (m, n), with m = 1 ... M, n = 1 ... N (1.2)
subdivided, each containing 16 * 16 pixels b (i, j), see FIG. 2.

For each block B (m, n) there is now a provisional motion vector that serves as an aid
X16 (m, n) = [x16 (m, n), y16 (m, n)] (1.3)
determined, which consists of two components, namely the horizontal displacement x16 (m, n) and the vertical displacement y16 (m, n).

This motion vector X16 (m, n) assigns a pixel a (k, h) from image A to each pixel b (i, j) in block B (m, n), specifically through the linkage
k = i + x16 (m, n) (1.4a) and
h = j + y16 (m, n), (1.4b)
ie the coordinates [k, h] of the pixel a (k, h), which is assigned to the pixel b (i, j), result from adding the displacement or motion vector X16 (m, n) to the coordinated [i, j].

To calculate X16 (m, n), the "Displaced Frame Difference" d () is used for each pixel b (i, j) in block B (m, n) and for each motion vector [r, s] in question. i, j, r, s) defines:
d (i, j, r, s) = b (i, j) - a (i + r, j + s), (1.5)
that is, the difference from the corresponding image point shifted by a motion vector [r, s] from image A.

Then the absolute amounts of the differences d (i, j, r, s) within the block B (M, n) are summed up, as a result of which the L1 norm (amount norm) D16 (m, n, r, s) becomes the "displaced frame" DIfference "for block B (m, n) and the motion vector [r, s] results in:

In addition to this sum D (m, n, r, s), a first measure for smoothing the motion vector fields is added "Penalty Term" (Penalty) P16 (r, s) added, which evaluates the length of the motion vector [r, s]:
D16 ′ (m, n, r, s) = D16 (m, n, r, s) + P16 (r, s) (1.7a)
in which
P16 (r, s) = 256 * β * (abs (r) + abs (s)) (1.7b)
is.

The "penalty term" thus consists of the L1 norm of the motion vector, multiplied by a control parameter β and by the number of pixels in a block, namely 256. The parameter β can be used to determine how much the length of the motion vector is in the target function D16 ′ (m, n, r, s) is received. (A typical value for β that has proven itself in simulation experiments is β = 1.0).

The minimum of D16 ′ (m, n, r, s) is then determined by varying [r, s] in a predetermined value range S, as a result of which the motion vector X16 (m, n) = [x16 (m, n) , y16 (m, n)] gives:

A square value range is usually chosen for S, e.g. B. the set of all motion vectors [r, s] for which the maximum amount of the two components r and s does not exceed an upper limit.

By adding the "penalty term", which represents a modified smoothness measure, to D (m, n, r, s) it is achieved that in homogeneous image areas or on straight edges where the movement is not clearly from the local image content can be determined (no clear minimum of D (m, n, r, s), short motion vectors are preferred. The probability of "outliers" occurring in the motion vector field is thus already reduced.

For β = 1.0 z. For example, a motion vector [r, s] = [0.1] results in an average displaced frame difference d (i, j, r, s) that is at least 1.0 smaller, so that it corresponds to the zero vector [r, s] = [0.0] is preferred. The same applies to larger motion vectors.

Since a full search is usually too time-consuming, the value range S is best limited to a number of sample values [r, s] in a fixed grid (e.g. lattice constant 4) and then in the vicinity of the Optimums continued to be searched ("Three Step Search", see, for example, BHG Musmann, P. Pirsch, H.-J. Grallert: "Advances in Picture Coding", Proc. IEEE 73 (1985) 4, pp. 523-548. In In this case, the accuracy of the method can be increased in that the search for the optimal motion vector is carried out partly in low-pass filtered images, which in turn can be combined with undersampling of the image in order to reduce expenditure.

This first step of the motion vector determination method therefore represents a known block matching method, which, however, is introduced by introducing the "penalty term" P16 (r, s) according to Eq. 1.7a, b was modified for the purpose of smoothing the motion vector fields.

Iterative improvement of the motion vector field at block size 16 by relaxation (ITER 16)

After the provisional motion vectors X16 (m, n) have been determined for all blocks B (m, n), as described above, this motion vector field is improved iteratively (Relaxation). For this purpose, a new "penalty term" or a new smoothness measure P16 ′ (m, n, r, s) is now defined, with the aid of which the deviation of the motion vectors X16 (m, n) from their respective four neighboring motion vectors X16 ( m-1, n), X16 (m + 1, n), X16 (m, n-1) and X16 (m, n + 1) is measured, see FIG. 3.

The smoothness measure P16 ′ (m, n, r, s) is defined by:
P16 ′ (m, n, r, s) = abs (r - x16 (m-1, n)) + abs (s - y16 (m-1, n)) +
+ abs (r - x16 (m + 1, n)) + abs (s - y16 (m + 1, n)) +
+ abs (r - x16 (m, n-1)) + abs (s - y16 (m, n-1)) +
+ abs (r - x16 (m, n + 1)) + abs (s - y16 (m, n + 1)) *
* α * 16 (2.1)

Here, r and s are the components of a motion vector [r, s] which is to be used as the new motion vector X16 (m, n) for block B (m, n).

The smoothness measure P16 '(m, n, r, s) thus represents the sum of the absolute norms (L1 norms) of the four difference vectors between X16 (m, n) and its neighbors, multiplied by a control parameter "α" and the edge length of one Blocks, namely 16. With the control parameter "α", the degree of smoothness of the motion vector field can be controlled in the relaxation step. The L1 standard was chosen because it preserves edges in the motion vector field as they occur at object edges, in contrast to the "quadritic standard" (L2 standard), which favors continuous transitions.

From the smoothness measure P16 ′ (m, n, r, s) and the “displaced frame difference” D16 (m, n, r, s) (Eq. 1.6) a new target function D16˝ (m, n, r, s ) educated:
D16˝ (m, n, r, s) = D (m, n, r, s) + P16 ′ (m, n, r, s) (2.2)

The motion vector field is smoothed by that first of all according to Eq. 1.8 (without neighborhood relationships) determined motion vector field, and starting from this default step by step, a new, optimal motion vector X16 (m, n) is determined for one block after another, by means of
X16 (m, n) = [x16 (m, n), y16 (m, n)] (2.3a)
so that

In order to limit the search effort (and as a further measure for smoothing the vectors), the value range S (m, n) is designed adaptively for each BLock in such a way that the search is all the more extensive, the different the motion vectors X16 (m, n) = x16 (m, n), y16 (m, n) and their four neighboring motion vectors are:
S (m, n) = [rmin ... rmax, smin ... smax] (2.4a)
With
rmin = min (x16 (m, n), x16 (m-1, n), x16 (m + 1, n), x16 (m, n-1), x16 (m, n + 1))
rmax = max (x16 (m, n), x16 (m-1, n), x16 (m + 1, n), x16 (m, n-1), x16 (m, n + 1))
smin = min (y16 (m, n), y16 (m-1, n), y16 (m + 1, n), y16 (m, n-1), y16 (m, n + 1))
smax = max (y16 (m, n), y16 (m-1, n), y16 (m + 1, n), y16 (m, n-1), y16 (m, n + 1)) (2.4b )

Only those motion vectors [r, s] are examined for which the following applies:
rmin ≦ r ≦ rmax and smin ≦ s ≦ smax.

In areas with a constant motion vector even across the block boundaries, rmin = x16 (m, n) = rmax and smin = y16 (m, n) = smax, so that the value range is at one point shrinks and no search effort arises. An improvement is only sought if there are jumps and other divergences and "outliers" in the motion vector field.

After looking for an improvement of the vector X16 (m, n) once for all blocks B (m, n), the process must be repeated for all those blocks B (m, n) for which at least one of the four neighbors is Motion vectors X16 (m-1, n), etc. changed in the previous pass. This results in an iterative process that has to be continued until none of the motion vectors X16 (m, n) can be improved, while keeping the four neighbors constant.

As a rule, about 5 to 10 iterations are sufficient, depending on the degree of movement in the image, whereby it must be taken into account that all blocks really only have to be checked in the first iteration and then only those in their vicinity during the last run Changes have occurred.

The fact that the smoothness measure P16 '(m, n, r, s) Eq. 2.1) The L1 norm of neighboring vectors and not the L2 norm, on the other hand, means that edges are preserved in this smoothing process, similar to MEDIAN filtering. In fact, the median of a set of numbers is the least that minimizes the sum of the amounts of the differences, that is, the sum of the L1 norms. The previously described smoothing by minimizing D16˝ (m, n, r, s) can therefore also be interpreted as a generalized median filtering of the motion vectors, which takes into account the "displaced frame differences".

mit d(i,j,x(i,j),y(i,j)) entsprechend Gl (1.5), und

A local optimum of the overall target function Z is now reached, which is obtained by summing D16˝ (m, n, r, s) over all blocks B (m, n) in the image, ie for Z:
Z = Z1 + αZ2, (2.5)
in which

with d (i, j, x (i, j), y (i, j)) according to Eq (1.5), and

Here, the quantities x (i, j) and y (i, j) etc. are the components of the motion vectors X (i, j) which result from the motion vector field X16 (m, n) in that all pixels b ( i, j) in block B (m, n) the same motion vector X16 (m, n) is assigned:
X (i, j) = [x (i, j), y (i, j)] = X16 (m, n) if b (i, j) in B (m, n). (2.8)

The motion vector field is further optimized in the following - d. H. the objective function Z is further minimized by halving the size of the blocks in which the motion vector field is assumed to be uniform.

Block division from 16 * 16 to 8 * 8 pixels (L = L / 2)

From the given motion vector field [X16 (m, n)] for the blocks of 16, a new field [X8 (p, q)] for blocks of 8 B8 (p, q) is created, which consists of the motion vectors
X8 (p, q) = [x8 (p, q), y8 (p, q)] (3.1)
consists. For this purpose, all 16-blocks are divided into four 8-blocks, and each of the sub-blocks is initially assigned the same motion vector, namely that of the 16-block. This new motion vector field serves as the default for the next relaxation step.

Iterative improvement of the vector field at block size 8 by relaxation (ITER 8)

This process step corresponds exactly to the relaxation step for block size 16, as described above, but with the change that "16" is to be replaced by "8" in the equations for the target function. So there is a target function D8˝ (p, q, rs) corresponding to D16˝ (m, n, r, s) according to Eq. 2.2 minimizes which is a smoothness measure P8 '(p, q, r, s) as in Eq. 2.1 contains. The same value as for block size 16 can be used for α (see Eq. 2.1).

Block division from 8 * 8 to 4 * 4 pixels (L = L / 2)

As with the block division from 16 * 16 to 8 * 8 pixels, the motion vectors that were determined for blocks of size 8 * 8 are now divided into four blocks with 4 * 4 pixels.

Iterative improvement of the vector field at block size 4 by relaxation (ITER 4)

This process step corresponds exactly to the relaxation steps for block sizes 16 and 8.

The process steps "block subdivision" and "relaxation" can be continued up to block size 1 * 1 pixel; however, for many applications a resolution of the motion vector field with one motion vector per 4 * 4 pixels is sufficient.

General block diagram

FIG. 8 shows a functional arrangement with which the motion vector determination method described above can in principle be carried out.

Starting from two input images A and B, the final motion vector field is determined in a sequence of variations of the motion vector field, taking into account the respective values of the target function Z.

Motion vector determination for inserting intermediate images (image interpolation)

The method described above can also be used to determine motion vector fields for image interpolation. For this, only the sizes that measure the "displaced frame differences", namely D16 (m, n, r, s) (Eq. 1.6) and the corresponding sizes for the blocks of sizes 8 * 8 and 4 * 4, something to be modified.

mit
    d′(i,j,r,s) = b(i-r/2,j-s/2) - a(i+r/2,j+s/2)      (7.2)
Der Bewegungsvektor [r,s] wird also nun nicht vollständig auf das Bild A angewendet, sondern zur Hälfte auf Bild A und mit umgekehrtem Vorzeichen auf Bild B, so daß insgesamt Bild A und Bild B wieder um den gesamten Bewegungsvektor [r,s] gegen­einander versetzt sind.If e.g. B. Exactly one intermediate image should be inserted between the given images A and B by motion-adaptive interpolation in such a way that moving objects in the interpolated image have just moved by half the shift from image A to image B, instead of D16 ( m, n, r, s) in Eq. 1.6 the new size D16i (m, n, r, s), where i stands for interpolation:

With
d ′ (i, j, r, s) = b (ir / 2, js / 2) - a (i + r / 2, j + s / 2) (7.2)
The motion vector [r, s] is now not completely applied to image A, but half to image A and with the opposite sign to image B, so that image A and image B again around the entire motion vector [r, s] are offset from each other.

The same applies to interpolation by factors higher than 2, i.e. H. if two or more images are to be inserted between the given images A and B. In general, the shift [r * r, t * s] is applied to image A and the shift [(t-1) * r, (t-1) * s] to image B, with 0 <t <1.

If the shift does not lead to integer pixel coordinates, rounding is required.

Distribution of the motion vector determination between transmitter (coder) and receiver (decoder) for moving picture coding

• 1) Bewegungskompensierende Prädiktion bei Sender und Empfänger
• 2) Bewegungsadaptive Interpolation fehlender Bilder beim Empfänger.

In a moving picture coding method, the motion vector determination is used for two purposes:

• 1) Movement-compensating prediction for sender and receiver
• 2) Motion adaptive interpolation of missing images at the receiver.

In this case, a movement vector determination is necessary on the transmission side in order to determine the movement vectors for the movement-compensating prediction. Since these motion vectors have to be transmitted, the motion vector field cannot be refined here at will. However, these motion vectors can also be used for the prediction in addition to the prediction for motion-compensating interpolation if a refined motion vector field is obtained beforehand from the transferred motion vector field and the transmitted images present on the receiver.

For use in image sequence coding, the previously described multi-stage process (see FIG. 1) can therefore be used in such a way that the process steps "initialization with block size 16 * 16" and "iterations with block size 16" are carried out on the transmission side and the remaining process steps ("block subdivision" and "iterations" for blocks of 8 and 4) at the receiver, see FIG. 9.

Simulation experiments have shown that the motion vector field obtained on the transmitter side for motion-compensating prediction is actually suitable as a specification for a receiver-side refinement for the purpose of interpolation, the "Displaced Frame Difference" D16 (m, n, r, s) from Eq. 1.6 and on the receiver side the function D16i (m, n, r, s) (Eq. 7.1, 7.2) adapted to the interpolation or its equivalents are used for smaller blocks.

The sequence of the motion vector determination method explained above is illustrated by the flow diagrams shown in FIGS. 4 to 7.

Fig. 4 shows the overall flow chart of the method, in which it is shown that initially with the maximum block size L = L _max an initialization (INIT) of the vector field takes place and that subsequently for all block sizes from L _max to L _min - where the block edge length L is halved - an iterative improvement (ITER) takes place. The course of the iterative improvement (ITER) for a given block size L is shown in FIG. 5. The motion vector which minimizes the objective function with modified smoothness measure is determined here for all blocks B (m, n).

The iterative improvement (ITER) follows the flow chart according to FIG. 6. There is a logging field FINISHED (m, n), which indicates for each block B (m, n) - with the respective block size L - whether the block is still closed edit is - ie

DONE (m, n) = 0 - or whether it is already in a local optimum (i.e. minimum) of the target function - d. H. DONE (m, n) = 1. First, the DONE field is set to zero for all blocks. In the following loop, the minimum of the target function in a certain value range (OPTI) is searched for all blocks that are not yet "DONE", see FIG. 7. DONE (m, n) = 1 for all these blocks set. If the motion vector of the block has changed during the minimum search, the DONE field for the neighboring blocks is set so that they can be processed again. When all blocks have the value FINISHED (m, n) = 1, the iteration sequence is ended.

The process of the optimization step (OPTI), which is run through in ITER, shows. Fig. 7.

In summary, it should be noted that the division of the blocks is preferably carried out in each case by halving the edge lengths of the blocks. When an intermediate image is inserted between the two images, two pixels, namely one from the first and one from the second image, are assigned to each pixel of this intermediate image. The differences in the luminance values of corresponding pixels of the two successive images within a block of pixels are evaluated by means of the sum of the absolute amounts of the differences in the luminance values and used as summands in the target function to be minimized, these summands forming a first component of the target function. The differences in the luminance values of corresponding pixels of the two successive images within a block of pixels can also be evaluated by means of the sum of the squares of the differences in the luminance values and used as summands in the target function to be minimized, these summands forming a first component of the target function.

The inventive method also provides that the differences between neighboring motion vectors by the Be norms of these differences are expressed, the sum of these norms forming a second component of the objective function and used as the smoothness measure. At least one of the two components is multiplied by a weighting factor, and the corresponding products form the target function by summation, the target function preferably having the form Z = Z1 + α Z2, Z1 being the first component, Z2 the second component and α the weighting factor .

In the preferred embodiment, only those four neighboring motion vectors are used as neighbors of each motion vector, the coordinates of which are horizontally and vertically adjacent to the coordinates of the relevant motion vector, i. that is, their coordinates differ by (0.1), (0, -1), (1.0) or (-1.0) from the coordinates of the relevant motion vector.

When initializing the motion vector field with large blocks - preferably 16-16 pixels (pixels) - namely as long as a motion vector has not been calculated for each block at least once, the smoothness measure is modified in such a way that instead of the differences between neighboring motion vectors, the amount norms of motion vectors to be optimized - multiplied by a weighting factor - are used in the target function by means of the smoothness measure.

In each stage of the block subdivision, ie at the beginning at the maximum block size and then after each block subdivision, which is preferably carried out up to a block size of 4x4 pixels (pixels), each individual motion vector is optimized in turn by variation in a relevant value range , until a smaller value of the objective function can no longer be found in this way for any motion vector.

The range of values of the motion vectors within which the individual motion vectors are varied in order to minimize the objective function is made dependent on the values of the motion vectors in the motion vector field already calculated, so that the range of values for the optimization of the motion vector field is small if neighboring Motion vectors are the same or similar, and is only larger if neighboring motion vectors have large differences between them.

A logging system has the effect that only those motion vectors are optimized again with a view to a possible reduction in the value of the target function, the neighboring motion vectors of which have changed since the last optimization of the motion vector in question, so that the smoothness measure has also changed, where appropriate the logging system contains a logging field with a storage space per block for storing control information for the optimization process.

When optimizing a particular motion vector, it is not absolutely necessary to take into account each motion vector of the relevant value range, but only a subset of the motion vectors according to a predetermined scheme.

In addition to the difference between neighboring motion vectors in the motion vector field to be calculated, the difference between the motion vectors of the current motion vector field and those of the motion vector field calculated immediately beforehand from a previous pair of successive images can also be determined and used in the smoothness measure, for the purpose of smoothing the motion vector field in the time axis direction, d. H. for matching successive motion vector fields.

The method according to the invention can also be used in cases in which there are two successive means Data compression of images transmitted from a coder via a channel to a decoder, motion vector fields are determined for the purpose of inserting intermediate images, in such a way that corresponding motion vector fields that have already been transmitted are used to initialize the method.

The appendix shows a complete program listing with 42 pages (p 1 ... p 42) covering the entire process sequence according to the invention for a preferred programming example in the programming language FORTRAN.

Claims (15)

1. A method for determining motion vector fields from digital image sequences, which determines a motion vector field from two successive images, which assigns a pixel of the other image to each pixel of one image, the assignment being defined in each case by a motion vector which determines the relative displacement of the pixels reproduces each other, and wherein each pixel in a square or rectangular block of pixels receives the same motion vector, characterized in that the determination of the motion vectors (X (m, n)) is carried out by minimizing a composite objective function (Z), which for one takes into account the differences in the luminance values of the mutually assigned pixels of the two given images, the other takes into account the differences between motion vectors whose coordinates are adjacent, namely so-called neighboring motion vectors, using a smoothness measure (P), and d The minimization of this objective function is carried out in such a way that first the motion vectors minimizing the objective function are determined with the restriction that the motion vectors are constant in larger than the blocks ultimately desired, and then each of these blocks (16x16) in smaller, preferably blocks of equal size are subdivided until the desired block size (4x4) is reached, the objective function being minimized by varying the motion vectors after each reduction of the blocks. 2. The method according to claim 1, characterized in that the division of the blocks is carried out in each case by halving the edge lengths of the blocks. 3. The method according to claim 1, characterized in that when inserting an intermediate image between the two images, each pixel of this intermediate image two Pixels, namely one from the first and one from the second image, can be assigned. 4. The method according to claim 1, characterized in that the differences in the luminance values of corresponding pixels of the two successive images within a block of pixels are evaluated by means of the sum of the absolute amounts of the differences in the luminance values and used as summands in the target function to be minimized, these Summands form a first component of the objective function. 5. The method according to claim 1, characterized in that the differences in the luminance values of mutually corresponding pixels of the two successive images within a block of pixels are evaluated by means of the sum of the squares of the differences in the luminance values and used as summands in the target function to be minimized, these Summands form a first component of the objective function. 6. The method according to claim 4 or 5, characterized in that the differences between neighboring motion vectors are expressed by the magnitude norms of these differences, the sum of these magnitude norms forming a second component of the objective function and being used as the smoothness measure. 7. The method according to claim 6, characterized in that at least one of the two components is multiplied by a weighting factor and that the corresponding products form the target function by summation, the target function preferably having the form Z = Z1 + α Z2, where Z1 is the first Component, Z2 are the second component and α is the weighting factor. 8. The method according to claim 6, characterized in that only those four neighboring motion vectors are used as neighbors of each motion vector, the coordinates of which are horizontally and vertically in proximity to the coordinates of the relevant motion vector, ie that their coordinates are around (0, 1), (0, -1), (1,0) or (-1,0) from the coordinates of the relevant motion vector. 9. The method according to claim 6, characterized in that during the initialization of the motion vector field with large blocks - preferably 16x16 pixels (pixels) - namely as long as a motion vector has not been calculated for each block at least once, the smoothness measure is modified so that instead of the differences between neighboring motion vectors, the amount norms of the motion vectors to be optimized - multiplied by a weighting factor - are used in the target function by means of the smoothness measure. 10. The method according to claim 2, characterized in that in each stage of the block subdivision, ie at the beginning at the maximum block size and then after each block subdivision, which is preferably carried out up to a block size of 4x4 pixels (pixels), in order each individual motion vector is optimized by variation in a relevant value range until a smaller value of the target function can no longer be found in this way for any motion vector. 11. The method according to any one of the preceding claims, characterized in that the value range of the motion vectors, within which the individual motion vectors are varied in order to minimize the objective function, is made dependent on which values the motion vector gates in the motion vector field already calculated, so that the value range for the optimization of the motion vector field is small if neighboring motion vectors are the same or similar, and is only larger if neighboring motion vectors have large differences between them. 12. The method according to any one of the preceding claims, characterized in that a logging system causes only those motion vectors to be optimized again with a view to a possible reduction in the value of the target function whose neighboring motion vectors have changed since the last optimization of the relevant motion vector have, so that the degree of smoothness may also have changed, the logging system containing a logging field with one storage space per block for storing control information for the optimization process. 13. The method according to claim 10 or 11, characterized in that when optimizing a particular motion vector not every motion vector of the relevant value range, but only a subset of the motion vectors according to a predetermined scheme is taken into account. 14. The method according to claim 1 or 6, characterized in that in addition to the difference between neighboring motion vectors in the motion vector field to be calculated, the difference between the motion vectors of the current motion vector field and those of the motion vector field calculated immediately beforehand from a preceding pair of successive images is determined and is used in the smoothness measure, namely for the purpose of smoothing the motion vector field in the time axis direction, ie for aligning successive motion vector fields. 15. The method according to the preceding claims, wherein motion vector fields are determined from two successive images transmitted by data compression from a coder via a channel to a decoder for the purpose of inserting intermediate images, characterized in that corresponding motion vector fields are used to initialize the method, that have previously been broadcast.

Images