EP0970586A1

EP0970586A1 - Method and array for computer-assisted assessment of the movement of an element of an image to be coded

Info

Publication number: EP0970586A1
Application number: EP98925399A
Authority: EP
Inventors: Jürgen PANDEL; Albert Salai
Original assignee: Siemens AG
Current assignee: Siemens AG
Priority date: 1997-03-26
Filing date: 1998-03-16
Publication date: 2000-01-12
Also published as: DE19712785C1; WO1998043434A1; CN1244992A; JP2002501697A

Abstract

In a first step (401) for assessing movement, the frequency distribution of movement vectors and/or the components of movement vectors of an image are determined. At least one search position in a reference image is determined (402) depending on the frequency distribution. A reference image is determined in each search position for an element of an image to be coded. An error margin for said search position is also determined (403). Similarities between the element of the image to be coded and the reference image are described on the basis of the error margin.

Description

description

Method and arrangement for computer-based motion estimation of an element of an image to be encoded

The invention relates to the computer-based motion estimation of an element of an image to be encoded.

For efficient compression of moving picture sequences, a reliable motion estimation is required for the picture coding. Various types of image coding are known. A distinction is usually made between two types of image coding, the so-called block-based image coding and the so-called object-based image coding.

Methods for block-based image coding are known, for example, from documents [1], [2], [3].

Methods for object-based image coding are known, for example, from document [4].

A method for image coding is known from [5], in which motion vectors are determined for an image and from the motion vectors an optimal motion vector is selected from the motion vectors with regard to the actual motion of an image block.

A further block-based image coding method is known from [6], in which a statistical distribution of the differential image signal is determined. Depending on the statistical distribution, a decision is made as to whether an image block to be encoded is encoded or not.

A motion estimation is carried out in both the block-based and the object-based image coding using conventional methods. When estimating motion, an element of an image to be coded is attempted to determine whether a before encoded image contains an area that matches the area to be encoded so well that it is sufficient with sufficient picture quality to encode only a reference to the already encoded area instead of encoding the entire element to be encoded. Since the respective element is usually shifted between successive images, the reference is made in the form of a so-called motion vector, which describes the shift of the area from the previous image to the element m of the image to be coded.

There are a variety of different search strategies for motion vector estimation. The so-called "block matching method" is usually used for block-based image compression methods. It is based on the fact that the picture block to be coded is compared with blocks of the same size of a previous picture. The previous image is referred to below as the reference image. One of the reference picture blocks is in the same position as the picture block to be coded, the other reference blocks are shifted in relation to this. With a large search area in the horizontal and vertical directions, there are a large number of search positions, so that with a full search (“fill search”), a corresponding number of block comparisons (“matchmgs”) must also be carried out. In general, the sum of the absolute differences in the coding information, which is assigned in each case to m individual pixels, is used as the criterion for the goodness of agreement between the image block to be coded and the reference block.

In this context, coding information is understood to mean, for example, luminance information or chrominance information associated with the respective pixel.

The so-called spiral search method is also known as a search strategy. In the spiral search, all search positions are again processed, but in a spiral, ie starting from the so-called zero shift, ie from the same position as the block to be coded. The search positions are selected on a spiral curve around the zero shift, the search positions being always further away from the zero shift.

In the method for motion estimation, at the end of the method, the motion vector is assigned to that search position in which the sum of the absolute differences in the coding information of the image block to be coded and the corresponding image block m is minimal in the reference image.

The invention is based on the problem of specifying a method for motion estimation with which the motion characteristics of elements of digitized images in a moving image sequence are better taken into account in the context of image coding than is possible with known methods.

The problem is solved by the method according to claim 1.

The method uses components of motion vectors or motion vectors of previously processed image elements, i.e. a frequency distribution was determined for picture elements for which a motion estimation had already been carried out. Depending on the frequency distribution, at least one search position is determined. A reference picture element is determined at the search position. An error measure is determined for the original picture element, the error measure being used to describe the similarity between the original picture element and the reference picture element. The location of the reference picture element determines the search position.

The method described in the following is described for simpler illustration using an image block BB as an image element which has, for example, 8x8 pixels. However, it is without limitation of generality without can also be used for macro blocks, which usually consist of 4 or 6 image blocks. Any other elementary units, ie picture elements of the respective underlying coding method, for example rectangles or triangles, etc. of any shape and size or, in the case of object-based picture coding, picture objects of any shape or arbitrarily shaped parts of picture objects can also be taken into account in the process. Thus, a picture element is to be understood as an elementary unit of any shape and size, in which the picture B is divided and for which the respective coding method takes place.

With this method, a search strategy for a new motion vector is proposed for the first time, which is adapted to the vector statistics of motion vectors already found or components of motion vectors in the same image or to the vector statistics of the images in the past. The picture coding is thus better adapted to the movement characteristics of the moving picture sequence, which reduces the computation effort required for the movement estimation.

In the arrangement for carrying out the method, an image memory for storing the digitized images and a processor unit are provided with which the individual method steps of the method are carried out.

The arrangement also has the above-mentioned advantages of the method compared to the known method for motion estimation.

Advantageous developments of the invention result from the dependent claims.

In a further development of the method, it is advantageous when determining the error measure, which can be formed by a sequence of accumulations of difference values, which Abort determination of the error measure with respect to a reference picture element if the value of the error measure with respect to the respectively examined reference picture element is greater than a predeterminable threshold value.

This procedure avoids unnecessary additional computing operations, which leads to a saving in the computing power required for the arrangement when carrying out the method.

This development can be further improved by the fact that the threshold value is of variable design and the threshold value is assigned the value of the error measure of the image element considered to be optimal in the previous method. In this way, a further reduction in required arithmetic operations is achieved.

Especially in connection with this further development of the method, the frequency distribution of the motion vectors or the components of motion vectors is taken into account very advantageously, since in the event that frequently occurring motion vectors are used to determine search positions in which the reference picture elements are preferably at an early stage Phase of the method are compared, a very good error measure and thus a very small threshold value is determined statistically very early, which, when the error measure is further determined with respect to further reference picture elements in the context of motion estimation, leads to the accumulations of the differences in the coding information in the further procedures can be terminated early. This saves a considerable amount of computing time compared to known methods for estimating movement.

In the figures, an exemplary embodiment of the invention is shown, which will be explained in more detail below. 1 shows a computer arrangement with two computers, one

Camera and a transmission medium; 2 shows a sequence of digitized images which are stored in a memory of a computer;

3a to 3c frequency distributions of components of motion vectors or of motion vectors of picture elements of a digitized image; 4 shows a flow chart in which the individual method steps of the method are shown.

1 shows a camera K with which images are recorded. The camera K can be, for example, any analog camera K that records images of a scene and either digitizes these images in the camera K or also transmits them analogously to a computer R1, which then either processes the digitized images or digitizes the analog images Images are converted and the digitized images are processed.

The computer R1 has a processor unit P, with which the method step of motion estimation or motion compensation described in the following and possibly further method steps, for example for image coding, are carried out. The processor unit P is coupled, for example, via a bus BU to a memory SP, in which the image data are stored.

It is envisaged that the computer R1 will perform the image coding and, after the compressed image data has been transmitted via a transmission medium UM to another computer R2, the other computer R2 will perform the image decoding. The further computer R2 has, for example, the same structure as the first computer R1, that is to say the memory SP, which is coupled to the processor unit P via the bus BU. The digitized images or the reconstructed images can either be displayed on a first screen BS1, which is coupled to the first computer R1, or on a second screen BS2, which is coupled to the second computer R2.

The method for motion estimation can be used both in the context of so-called block-based image coding methods and in the context of object-based image coding methods.

However, for the sake of simplicity, only the procedure for a block-based image coding method is shown below.

A sequence of digitized images ZVB, EVB, OB is shown symbolically in FIG. 2 and is stored in the memory SP.

This representation is merely a symbolic representation, since in most image coding methods it is not the case that several successive images are completely stored in the memory SP. This representation therefore only serves to illustrate the method.

The aim of the motion estimation is to carry out an image coding for an original image OB of the image sequence.

An attempt is made for an element OBE of the original picture OB, an already coded, i.e. to find processed image element BBE, which is contained in a first preceding image EVB, which is sufficiently similar to the element OBE of the original image OB. The processed picture element BBE is already coded and thus already transmitted.

In this context, sufficiently similar means that it is sufficient with only a slight reduction in the image quality is to insert the processed picture element BBE with a possible shift, which takes place between the processed picture element BBE m the first preceding picture EVB and the element OBE of the original picture OB, in the picture decoding m the picture to be decoded, with which a complex coding of the element OBE des Origmalbildes OB no longer required

The displacement of the processed picture element BBE between the first preceding picture EVB and the respective element OBE of the original picture OB is referred to as the motion vector BV.

In conventional block-based image coding methods, a motion vector BV is determined for each image block of an image or even just a so-called macro block, which has, for example, 4 or 6 image blocks, and is assigned to the image block or macro block.

The order that is searched for the element OBE of the orig. Picture OB m in the first previous picture EVB for a picture element that matches the element OBE of the orig. The image blocks with which the element OBE of the original image OB is compared are referred to as reference image elements RBE.

2 shows processed image elements BBE, each of which has been assigned a motion vector BV.

Before carrying out the motion estimation for the element OBE of the original image OB or for the entire original image OB, a frequency distribution of motion vectors BV of image elements BBE that have already been processed is determined. Any number of picture elements BBE that have already been processed can be within the original picture OB or within a Any number of previous pictures, for example the first previous picture EVB or also a second previous picture ZVB or further previous pictures are taken into account in the picture sequence.

In the frequency distribution, the number of movement vectors BV that occur in each case, as sketched, for example, in FIG. 3c, is accumulated. The motion vector BV usually has a first component BV _X and a second component BVy in the 2-dimensional space. Both components together form the motion vector BV.

A frequency distribution F of the motion vectors BV is shown as an example in FIG. 3c. It is the number for each component of the motion vector BV that occurs

ABV _{xy is} shown, with which the frequency of the occurrence of the respective motion vector BV is described. This results in a 2-dimensional area F in a 3-dimensional space, which is spanned by the first component BV _X , the second component BV _y and the number ABV _X y.

In Fig. 3c it is shown as an example that the motion vector (2,1) occurred 6 times in the processed image elements BBE considered, which were used to determine the frequency distribution. The motion vector (6,6) has occurred 5 times in this example.

In a further step of the method, a search position for the element OBE in the original image OB with respect to at least one reference image element RBE in the first preceding image EVB is determined depending on the frequency distribution F. At the search position, an error measure is also determined for the original picture element OBE. This is done, for example, by comparing the coding information in the first preceding picture EVB at the search position with the reference picture element RBE which contains the search position. mation, which contains the reference picture element RBE or the element OBE of the original picture OB, respectively.

The error measure takes place, for example, by forming the difference between the coding information of the individual pixels of the element OBE and the reference pixel RBE. Here, for example, the sum of the quadratic differences is used.

In a further development of the method, several search positions can be determined and thus also several difference picture elements RBE, each of which contains at least one search position.

The formation of the error measure is carried out in each case for a reference picture element RBE and the element OB of the original picture OB. That reference picture element RBE is selected and used in the context of the motion estimation as the reference picture element RBE most similar to the element OBE, which of the reference picture elements RBE taken into account, the element OB of the original picture OB has the greatest correspondence with the element OB of the original picture OB with regard to the error measure.

The order in which the individual reference picture elements RBE are examined depends on the frequency distribution of the motion vectors BV.

This means, for example, that the comparison of the element OBE of the original picture OB with the reference picture elements RBE is started at the search position, which results from the fact that, starting from the position of the element OBE of the original picture OB, the position around the most frequently occurring motion vector in the frequency distribution is shifted. This results in the search position in the first preceding picture EVB, in which, according to the statistics, ie the semantics of the picture content, it is most likely that one of the Element OBE very similar reference pixel RBE m which it is ^¬ most previous image EVB.

A predeterminable threshold value is also taken into account as part of this method.

Each time the element OBE of the original picture OB is compared with a reference picture element RBE, the error measure is determined as long, i.e. the individual differences in the coding information are added up until the value of the error measure exceeds the threshold value.

In Fig. 4 the method is shown summarized its individual method steps in a flow chart.

In a first step (401), a frequency distribution of motion vectors BV and / or components of motion vectors BV _X , BVy of processed image elements BBE of previous images EVB, ZVB, ... is determined.

In a further step (402), at least one search position in a reference image EVB is determined as a function of the frequency distribution.

For the element OBE of the original picture OB, an error measure of the coding information of the element OBE of the original picture OB with respect to a reference picture element RBE at the search position is determined (403) as part of the motion estimation, the error measure being used to determine the similarity between the element OBE and the reference picture element RBE is described.

Some variants of the exemplary embodiment described above are described below:

The camera K can, for example, also be a digital camera K, with which directly digitized images B are recorded and fed to the computer R1 for further processing. The computer R1 can also be designed as an independent arrangement that carries out the method steps described, for example as an independent computer card that is installed in a computer.

Even if only the procedure for a block-based image coding method was shown above, the method can, however, also be used without any problems for object-based image coding methods. In the case of object-based image coding methods, it is only necessary that image objects of approximately the same shape and size are compared with one another as part of the motion estimation, since otherwise the result of the motion estimation could possibly be incorrect.

In a variant of the method, it is also provided to determine only a frequency distribution for the individual components BV _X and BVy of the motion vector BV.

Such a frequency distribution is shown by way of example in FIG. 3a for the first component BV _X. FIG. 3a shows that the first component BV _X with a value 4 occurred 8 times in the processed picture elements BBE taken into account. The first component BV _X with a value of 1 has occurred twice, for example.

Such a frequency distribution for the second components BVy is shown in FIG. 3b. The order in which the individual reference picture elements RBE are examined is selected in this case depending on the frequency distribution of the components BV _X and BVy of the motion vectors BV.

The error measure can also be determined, for example, by summing the absolute difference between the pixels of the element OBE and the reference picture element RBE. Further options for forming the error measure are sufficiently ongoing and can be used without limitation within the scope of this procedure.

Furthermore, it is provided in a variant to adaptively design the threshold value, i.e. in each case to set the threshold value to a new value if the error measure has been taken into account after taking into account all pixels in the respective elements OBE, RBE and the error measure is less than the previous threshold value. In this case the value of the error measure is assigned to the value of the threshold value. In this way, the threshold value is carried out to the “optimal” value of the error measure when the method is carried out iteratively for a plurality of reference picture elements RBE.

If the examination of a reference image element RBE is stopped in each case when the value of the error measure for the respective comparison exceeds the value of the threshold value, a considerable saving in computing time is achieved by taking the frequency distribution into account, since it is statistically guaranteed that a very early on very similar image element RWE is determined and thus a threshold value with a small value is determined very early.

The following publications have been cited in this document:

[1] Ming Liou, Overview of the px64 kbit / s Video Coding Standard, Communications of the ACM, Vol. 34, No. 4, pp. 60-63, April 1991

[2] G. Wallace, The JPEG Still Picture Compression Standard, Communications of the ACM, vol. 34, no. 4, ^' pp. 31-44, April 1991

[3] D. Le Gall, MPEG: A Video Compression Standard for Multimedia Applications, Communications of the ACM, vol. 34, no. 4, pp. 47-58, April 1991

[4] S. Hofmeier, multimedia on the go, Funkschau, No. 7, pp. 75 - 77, March 15, 1996

[5] US 5,028,996

[6] US 5,565,921

Claims

claims

1. Method for computer-aided motion estimation of an element (OBE) of an image to be encoded, with any number of pixels (BP), for image coding of digitized images,

a frequency distribution (HV) of at least one component of motion vectors and / or of motion vectors of processed image elements (BBE) for which motion estimation has already been carried out is determined,

- in which at least one search position (SP) is determined depending on the frequency distribution (HV),

- in which an error measure is determined for the original picture element (OBE) at the search position (SP), - in which the error measure describes the similarity between the element (OBE) and a reference picture element (RBE), and

- in which the search position (SP) is given by the reference picture element (RBE).

2. The method according to claim 1,

- in which several search positions (SP) are determined,

- in which several reference picture elements (RBE) are determined, each containing at least one search position (SP), - in which the method is carried out for each element (OBE) of the picture to be coded and one reference picture element (RBE).

3. The method according to claim 2, wherein the reference picture elements (RBE) are processed in a sequence that results from the frequency distribution (HV).

4. The method according to claim 3, wherein the reference picture elements (RBE) are processed in the order of decreasing frequency of the component of motion vectors and / or the motion vectors.

5. The method according to any one of claims 1 to 4,

coding information is assigned to the pixels and the error measure is formed by comparing the coding information of the element (OBE) with coding information of the reference picture element (RBE),

6. The method of claim 5, wherein the error measure by a sum of differences of

Coding information of the element (OBE) with coding information of the reference picture element (RBE) is determined.

7. The method according to any one of claims 2 to 6, wherein the determination of the error measure for em reference picture element (RBE) is terminated if the value of the error measure is greater than an threshold value.

8. The method according to claim 7, wherein the threshold value is specified at the beginning of the method.

9. The method according to claim 7 or 8,

- in which the threshold value is designed to be variable, and - in which the value of the error measure is assigned to the threshold value if the error measure for the respective reference picture element (RBE) is smaller than the threshold value.

10. The method according to any one of claims 1 to 9, in which a block-based image coding is used for image coding.

11. The method according to any one of claims 1 to 9, in which an ect-based image coding is used for image coding.

12. Arrangement for performing the method according to one of claims 1 to 11,

with a processor unit with which method steps of the method are carried out,

- With an image memory, which is coupled to the processor unit (PE) for storing digitized images.

13. The arrangement according to claim 12, with a camera (K) coupled to the image memory (BS).