WO2001045422A1

WO2001045422A1 - Estimation of movements in video images

Info

Publication number: WO2001045422A1
Application number: PCT/EP2000/011710
Authority: WO
Inventors: Gerard De Haan; Rimmert B. Wittebrood
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 1999-12-17
Filing date: 2000-11-23
Publication date: 2001-06-21
Also published as: EP1153513A1; WO2001045420A1; WO2001045421A1; JP2003517795A; KR20010102216A

Abstract

The invention relates to an object-oriented method for the interpolation between video images. Two or more models of movement are used for describing the displacement of image objects between two video images. The models of movement are determined by parameter sets. Displacement vectors can be calculated on the basis of said parameter sets. One of the models of movement is required for considering the static, that is fixed, image components. According to the invention, when determining the parameter sets for the models of movement the method is restricted to interesting image areas for saving time when calculating. Image components having two video images that are significantly different from one another are selected.

Description

Motion estimation in video images

The invention relates to a method for motion estimation in video image data, wherein parameter sets of two or more motion models are first determined on the basis of a first and a second video image, and image objects are then assigned to the motion models. The invention further relates to a device for performing the method, a device according to the method for displaying video images and a computer program product for motion estimation.

Advances in multimedia technology have led to the development of a wide variety of video formats and playback standards. These differ among other things in the frame rate, i.e. in the number of individual images per unit of time. If a video sequence is to be played back on a PC or TV screen, it is necessary to adapt it to the frame rate of the playback device. Interfaces suitable for this work with conversion processes of different complexity. The simplest method is to repeat or omit frames of the video sequence depending on the desired frame rate. However, undesirable artifacts occur when displaying video data treated in this way. Depending on the ratio of the underlying frame rates, undesirable display errors occur. The playback appears stagnant and uneven, so that the movements shown in the video sequence appear unnatural. More complex methods carry out interpolation between successive video images, an algorithm for motion estimation being used which detects the displacements of individual image elements from one frame to the next and generates image data therefrom which are temporally between the individual frames of the video sequence. The application of such

Methods in the field of home terminals require that the underlying algorithms deliver a high-quality frame rate conversion and only require a small amount of computation, since the performance of digital signal processing electronics from home devices is limited. In addition to image rate conversion, methods for motion estimation of the type mentioned at the outset are also suitable for coding and compression during the transmission of video data, as well as for depth estimation in 3D image data processing and for disparity estimates for stereo images. WO 99/16251 proposes such a method. It is a powerful, object-oriented method for motion estimation, in which two or more motion models are used to describe the displacement of image objects between a current and a previous video image. The motion models are determined by parameter sets, from which displacement vectors can be calculated. One of the motion models is required to take into account those parts of the image that are static. The associated displacement vector is therefore the zero vector. The parameter sets of the other motion models are determined by evaluating the adjustment errors of the motion models when describing the displacement of the image objects between successive video images. It is then necessary for the interpolation to segment the image data and to assign suitable motion models to individual image objects. The result of the segmentation are separate objects, that is to say parts of the image which perform a similar or comparable shift from the previous to the current video image. The known method for motion estimation represents a powerful alternative to the otherwise usual block-oriented method, since the number of objects that can move independently of one another in normal video sequences is small and therefore only a correspondingly small number of motion models has to be processed. This results in a small number of arithmetic operations, which makes the method universally applicable even in the home.

The basic object of the present invention is to further improve the previously known motion estimation method while at the same time further reducing the complexity.

An important step in motion estimation is the determination of the parameter sets for the motion models. In the previously known method, the parameter sets are combined into vectors. For each movement model, a parameter set is vectored from a set of candidate according to a selection criterion selected. The selection criterion is the assessment of an adjustment error. This is calculated as the sum of absolute differences between individual motion-compensated pixel intensities between the current and previous video image, a displacement vector according to one of the candidate vectors being used for the compensation. A major problem here is that it is not clear from the start which image area which movement model with which parameter set is to be assigned. The previously known method proceeds by first applying the above selection criterion to the same image areas with all motion models. The most suitable parameter sets are then selected without an assignment having already been made. According to the above task, another

Reduction of complexity in a motion estimation method of the type mentioned at the outset achieved in that only parts of the image area are taken into account when determining the parameter sets. One problem here is to select appropriate parts of the image area in a suitable manner so that shifts between the video images are recorded as completely as possible. According to the invention, only those parts of the image area are taken into account for the determination of the parameter sets in which the first and the second video image differ significantly.

Such differences are a clear indication that there is a shift at the relevant points. This makes it very easy to select parts of the image for determining the parameter sets, without first having to be more precise

Movement data must be known. In addition, it is avoided that when the parameter sets for the motion models are ascertained, still parts of the image are processed in which motion compensation is meaningless and therefore unnecessary. The parameter determination only has to take place in the non-stationary parts of the image. Because the selection criterion is only applied to parts of the image area, the number of arithmetic operations required is greatly reduced and the entire motion estimation is thus accelerated. Due to the fact that only a few moving objects are shown in typical video sequences, it is normally sufficient to limit yourself to a corresponding number of "interesting" points in the video image when determining the parameter sets.

The "areas of interest" are expediently determined by evaluating deviations between the video images block by block, taking into account those blocks for determining the parameter sets in which the deviation value exceeds a predetermined threshold value. The image area is So divided into individual blocks, the size of which must be such that the parameter sets can be determined using individual blocks. The deviations between the current and previous video image can be determined, for example, by forming the absolute differences in the pixel intensities in each case within the individual blocks. The result is a positive number, so that comparison with a predetermined threshold value makes it easy to determine whether or not there is movement in the associated image area. When determining the parameter sets, the method according to the invention is then limited to those blocks in which a certain difference between the two video images can be recognized on the basis of the pixel intensities. This procedure has the additional advantage that the threshold value can be determined on the basis that the number of image areas taken into account for the determination of the parameter sets is limited to a predeterminable value. Since the entire process must be performed in real time for the motion estimation, it must be ensured that the number of arithmetic operations remains below a specified maximum value. It is thus possible, when the method according to the invention is used repeatedly, to regulate the threshold value such that the data processing time remains uncritical.

In practice it has been shown that it can be advantageous to take into account those parts of the image area for the determination of the parameter sets in which movements of previous video data of a sequence of video images were determined. This results in a higher temporal consistency in the movement compensation.

A device for motion estimation in video image data, with a digital image memory in which a first and a second video image can be stored, and with means for determining parameter sets of two or more motion models according to one is suitable for carrying out the method according to the invention

Selection criterion. The device according to the invention has means for the block-wise evaluation of the deviations between the current and previous video image and for the selection of those blocks for the application of the selection criterion in which the deviation value exceeds a predefinable threshold value. Devices of this type are used, for example, as subcomponents in television and video devices. The digital image memory of the device according to the invention does not necessarily have to have sufficient capacity to record the first and the second video image simultaneously. The storage of the respective images in succession is sufficient for the method according to the invention. Devices for displaying video images, such as television sets, monitors, etc., with a digital image memory in which video image data can be stored, with electronic means for processing the image data stored in the image memory and for displaying video images on a display device, the means for processing the image data have means for determining parameter sets of two or more motion models according to a selection criterion can advantageously benefit from the method according to the invention if the means for processing the image data also include means for evaluating the deviations between the current and previous video image in blocks and for selecting those blocks Have application of the selection criterion in which the deviation value exceeds a predefinable threshold. Conventional digital television sets and monitors can be operated in a simple manner by the method according to the invention, which improves the quality of the image display. Display devices in the above sense are, for example, the cathode ray tubes or dot matrix displays that are common in television sets and monitors. Other display devices for the visual output of digital image data are also conceivable.

According to the invention, a computer program product is suitable for the interpolation between pairs of video image data sets, which receives an input as a first and a second video image, calculates parameter sets of two or more motion models based thereon, and outputs motion data which describe the displacement of the image objects from the previous to the current image, the image data of the two video images being compared with one another and only those parts of the image area being taken into account in the calculation of the parameter sets in which there are significant differences between the two video images. The computer program can be made available on various data carriers, such as floppy disks, CD-ROMs or the like, but also for transmission via computer networks (e.g. the Internet).

Exemplary embodiments of the invention are explained below, also with reference to the figures. 1 shows a selection of interesting image areas;

Fig. 2 block diagram of a motion estimation method according to the invention.

Fig. 3 block diagram of a device according to the invention for displaying video images. When determining the parameter sets for the motion models according to the

Invention, a selection criterion is applied to selected image areas. The

The selection criterion consists, for example, in the evaluation of an adaptation error ε.

This is the sum of absolute differences between individual motion-compensated

Pixel intensities between a current and a previous video image are calculated as follows: ε ( _σ , n) = ∑W _o (-r> | F. (, n) - F. (* - C ₀ (x, π), n -l) | x I (.n)

There is a summation via image coordinates x, in one set

I (n) of selected image areas are included. The absolute differences between the pixel intensities in the current or previous video image are summed up at these image coordinates. F _s (x, n) is the pixel intensity at the image coordinate x in a video image with a reduced image raster. It has been shown that the use of an image with a reduced resolution (sub-sampled image) is completely sufficient for the parameter determination. This advantageously leads to a considerable reduction in the computing effort. The number n of the

Frame and thus the point in time within the video sequence. C ₀ (x, n) denotes the displacement vector at the image coordinate x for the image pair n as the current and n - \ as the previous video image according to the motion model with the index o. W ₀ (x) represents a weighting factor with which it is taken into account which motion model o the image coordinate x was assigned to in the previous sequence of image data of the video sequence. This enables a link to be established between parameter determination and image segmentation, which has advantages with regard to the temporal consistency of the motion estimation and the performance of the method.

The following motion model can be used to calculate displacement vectors based on four parameters:

(s _x (o, ή) + x ^■ d _x (o, n) C ₀ (x,) = sXo, n) + y - d., (o, n)

It is a simple linear first-order model that can be used to describe translations and scalings. The model is determined by the

Parameter set P _g (n) = s _x (o, n), d _x (o, n), s (o, n), d (o, n) f. The parameter set is determined in such a way that the above adaptation error for the corresponding motion model is one assumes minimum value. In the motion estimation method according to the invention, at least two motion models are always used, one of which always has the zero vector as a parameter set, so that by means of this motion model with the

Displacement vector C ₀ (x, n) = 0 describes the stationary image areas. The next step in motion estimation according to the invention is

Image segmentation, i.e. the assignment of image areas to the motion models. To do this, the entire image area is first divided into blocks. In practice, square blocks of 8 by 8 pixels have proven their worth. For all image coordinates within the

Blocks at position X then xe B (X). An adaptation error to a motion model o is in turn calculated for each block: ε ₀ (X, n) = ∑ \ F _s (x + (l -) C ₀ (x, n), n) - F _s (x - aC ₀ (x, n), n - ϊ) \ x≡B (X)

The point in time at which the segmentation should be valid is determined by a. In the simplest case, block X is assigned the motion model o for which ε _o (X, n) is minimal. The assignment is then stored in the segmentation mask M (X, ή).

According to the invention, the motion estimation method is limited when determining the parameters of the motion models to "interesting" image areas, which is given by the set I (ή). It is advantageous to fill the set I (n) with the blocks that have a poor match with the corresponding blocks in a previous picture. This can be done, for example, according to the following rule:

I (n) = {x \ ε ₀ (X, nl) ≥ τ]

T is a predeterminable threshold value which defines the measure of the deviation between two successive images, from which a parameter determination is carried out in the relevant image area.

FIG. 1 shows a video image 1, on which a motorcyclist 2 is shown who is driving on a road 3. The motorcyclist moves from left to right in the image section shown. The background, including street 3, is at rest. The figure shows the selection of "interesting" image areas, which are shown as white blocks 4 are shown. The white blocks 4 are assigned a movement model that describes the movement of the motorcyclist 2. The image background is stationary and is assigned to another corresponding movement model. FIG. 2 schematically shows the sequence of the motion estimation method according to the invention. Starting from a previous video image 6, a current video image 7 and a threshold value 8, the image areas of interest for determining the parameter sets are determined in a first step 9 using the method described above. For a plurality of motion models, all these image areas are provided with weighting factors 10 and then further processed in a method step 11, in which the parameters of the motion models are determined according to a selection criterion. Starting from the completely determined motion models, the entire image area is then divided into blocks in a step 12 and the displacement vectors corresponding to the individual motion models are calculated for each block. The segmentation of the image area then takes place in FIG. 13, the blocks being assigned to the movement models. In a segmentation mask 14 that arises, the assignments that are used in the weighting 10 for the next pair of images are stored.

Figure 3 shows schematically the structure of a digitally operating display device, which is e.g. can be a television set or a video monitor. The device is supplied with a video signal 20, which is stored and processed in a digital image processing unit 21. For this purpose, the image processing unit has an image memory 22, a processor 23 and a program memory 24. If necessary, these elements can also, at least partially, be combined in a discrete component. A program stored in the program memory 24 runs on the processor 23 and controls the image processing according to the invention. An image display unit 25 receives image data 26 processed by the image processing unit 21 and uses this to generate a signal 27 for controlling a cathode ray tube 28, via which the video images are visually output.

Claims

CLAIMS:

1. Method for estimating motion in video image data, in which, based on a first and a second video image (6, 7), parameter sets of two or more motion models are determined (11), characterized in that only those parts (4, 5) of the image area ( 1) are taken into account for the determination of the parameter sets (9), in which the first and the second video image differ significantly.

2. The method according to claim 1, characterized in that deviations between the current and previous video image are evaluated block by block, taking into account those blocks for the determination of the parameter sets in which the deviation value exceeds a predetermined threshold value.

3. The method according to claim 2, characterized in that the threshold value is determined according to the proviso that the number of image areas taken into account for the determination of the parameter sets is limited to a predeterminable value.

4. The method according to claim 1, characterized in that those parts of the image area are taken into account for the determination of the parameter sets in which movements of a sequence of video images were determined in the case of earlier video image data.

5. Device for motion estimation in video image data, with a digital image memory in which a first and a second video image can be stored, and with means for determining parameter sets of two or more motion models according to a selection criterion, characterized by means for the block-wise evaluation of the

Deviations between the current and previous video image and for the selection of those blocks for the application of the selection criterion in which the deviation value exceeds a predefinable threshold value.

6. Apparatus for displaying video images, in particular a television or monitor, with a digital image memory (22) in which video image data can be stored, with electronic means (21, 25) for processing the image data stored in the image memory and for displaying video images on one Display device (28), the means (21) for processing the image data having means for determining parameter sets of two or more motion models according to a selection criterion, characterized in that the means (21) for processing the image data furthermore means for evaluating the deviations in blocks have between the current and previous video image and for the selection of those blocks for applying the selection criterion in which the deviation value exceeds a predefinable threshold value.

7. Computer program product for motion estimation in video image data, which receives as input a first and a second video image, based on which parameter sets of two or more motion models are calculated and output motion data which describe the displacement of the image objects from the previous to the current image, characterized in that the image data of the two video images are compared with one another and only those parts of the image area are taken into account in the calculation of the parameter sets in which there are significant differences between the two video images.