US20100271554A1

US20100271554A1 - Method And Apparatus For Motion Estimation In Video Image Data

Info

Publication number: US20100271554A1
Application number: US12/677,507
Authority: US
Inventors: Volker Blume
Original assignee: Individual
Current assignee: Entropic Communications LLC
Priority date: 2007-09-10
Filing date: 2008-08-05
Publication date: 2010-10-28
Also published as: CN101803362A; EP2188990A2; WO2009034489A2; WO2009034486A3; CN101803363A; WO2009034492A3; WO2009034488A2; US20100238355A1; CN101836441A; CN101836441B; US20110205438A1; CN101803361A; US20100277644A1; US9036082B2; WO2009034493A2; CN101803363B; EP2206342A2; US8526502B2; EP2206341A2; CN101796813A

Abstract

The present invention is related to a method and an apparatus for motion estimation using a vector histogram providing a ranking of most and less used vectors and motion compensation in video image data and especially for line based field rate up-conversion motion estimation and motion compensation of video image data. The present invention is further related to TV-set, a computer program product and a data carrier comprising a computer program.

Description

FIELD OF THE INVENTION

The present invention is related to a method and an apparatus for motion estimation in video image data and especially for field rate up-conversion motion estimation video image data. The present invention is further related to a TV-set, a computer program product and a data carrier comprising a computer program.

TECHNICAL BACKGROUND OF THE INVENTION

The present invention relates to a motion estimation and motion compensation device and more particularly to a motion estimation and motion compensation device that estimates motion vectors and performs motion compensated predictions of an interlaced sequence of chrominance sub-sampled video frames.
Hereinafter, the present invention and its underlying problem is described with regard to the processing of a video signal for line-based motion estimation and motion compensation within a video processing apparatus such as a microprocessor or microcontroller having line memory devices, whereas, it should be noted, that the present invention is not restricted to this application, but can also be used for other video processing apparatus.
The market introduction of high-end TV-sets based on 100 Hz Cathodic Ray Tubes (CRT) required the development of reliable Field Rate Up-conversion (FRU) techniques to remove artefacts within a picture such as large area flickers and line flickers. Standard FRU methods, which interpolate the missing image fields to be displayed on the CRT without performing an estimation and compensation of the motion of moving objects in successive image fields are satisfactory in many applications, especially with regard to a better quality of the image and with regard to the reduction of the above-mentioned artefacts. However, many pictures contain moving objects, like persons, subtitles and the like, which cause so-called motion judders.
This problem is better understood by referring to FIG. 1, wherein the motion trajectory of the moving objects (white squares) in the original image fields (i.e. transmitted and received image fields) is supposed to be straight-lined. If the missing fields result from interpolation by means of the above mentioned standard FRU methods (i.e. without motion estimation and compensation), the motion of the moving object in the interpolated fields (dark grey squares) is not at a position as expected by the observer (dotted squares). Such artefacts are visible and induce a blurring effect especially of fast moving objects. These blurring effects typically reduce the quality of the displayed images significantly.
In order to avoid such blurring effects and to reduce artefacts several methods for motion estimation and motion compensation—or shortly MEMC—are proposed. This MEMC provides the detecting of a moving part or object within the received image fields and then the interpolation of the missing fields according to the estimated motion by incorporating the missing object or part in an estimated field.
FIG. 2 shows schematically the change of the position of a moving object between two successive image fields. Between two successive received image fields, the moving objects will have changed their position, e.g. object MO which is in the previous field T in position A is then in the current field T+1 then in position B. This means, that a motion exists from the previous field T to the current field T+1. This motion of an object in successive image fields can be represented by a so-called motion vector. The motion vector AB represents the motion of the object MO from position A in the previous field T to position B in the current field T+1. This motion vector AB typically has a horizontal and a vertical vector component. Starting from point A in the previous field T and applying this motion vector AB to the object MO the object MO is then translated in position B in the current field T+1. The missing position I of the object MO in the missing field T+½ that has to be interpolated must be calculated by the interpolation of the previous field T and the current field T+1 taken account of the respective positions A, B of the moving object MO. If the object MO does not change its position between the previous field and the current field, e.g. if A and B are the same positions, position I in the missing field is obtained by the translation of A with a motion vector |AB|/2. In this manner the missing field T+½ is interpolated with a moving object in the right position with the consequence that blurring effects are effectively avoided.
MEMC implementations operating in the way sketched up above are described for example by Gerard de Haan in EP 765 571 B1 and U.S. Pat. No. 6,034,734.
Theoretically, for each pixel of a frame or field a corresponding motion vector has to be calculated. However, calculating motion vectors for the huge amount of the pixels within a field or frame a enormous amount of calculations is needed with the consequence that the memory requirements are increasing significantly. Especially with modern Plasma- and LCD-TVs which typically have an increased resolution this goes along with an corresponding increase of the number of pixels and thus the number of motion vectors to be calculated. To reduce this enormous calculation and memory effort there is an increasing need to provide MEMC implementations which are able to only use a reduced set of motion vectors.
To reduce the amount of motion vectors needed there exist several different approaches:
A first approach employs a so-called block-based MEMC. This approach assumes that the dimension of an object in the image is always larger than that of a single pixel. Therefore, the image field is divided into several image blocks. For MEMC only a single motion vector is calculated for each one of these blocks which leads to a significant reduction of used motion vectors. This approach is for example described in EP 874 523 A1.
A second approach employs a so-called line-based MEMC. In this approach the algorithms is based on a reduced set of video input data of a single line of a field or part of this line. However, in this line based MEMC there is so far no method known in the art for an effective reduction of motion vectors.
It is clear, that for the most video applications the different pixels within a field or a frame or of a line of this field/frame is not changing from one field/frame to the next one. Therefore, for these pixels typically a zero motion vector is applied, that is a motion vector having the zero-magnitude and a zero-angle. In many applications those zero motion vectors build by far the most used motion vectors within a field/frame. On the other hand, there are also a huge number of pixels having an identical motion vector especially those pixels assigned to the same moving object in the picture.
The present invention is, therefore, based on the object to provide a possibility to more efficiently use motion vectors within a motion estimation process. The present invention is further based on the object to reduce the memory requirement and/or the computational requirements in motion estimation implementations.

SUMMARY OF THE INVENTION

In accordance with the present invention, a method comprising the features of claim 1 and/or an apparatus comprising the features of claim 15 and/or a TV-set comprising the features of claim 23 and/or a computer program product comprising the features of claim 24 and/or a data carrier comprising the features of claim 25 is/are provided.
Accordingly, it is provided:

- A method for motion estimation in video image data, especially for field rate up-conversion in consecutive frames of a motion picture, comprising the steps of: providing a video signal comprising video image data of a video line or part of the video line of the picture; performing the motion estimation by detecting and analysing the video image data and by deriving motion vectors depending on the detected motion; calculating a histogram for motion vectors to derive most and less used motion vectors in a current frame or field.
- An apparatus for motion estimation in video image data, especially field rate up-conversion in consecutive frames or fields of a motion picture, wherein the apparatus is configured to perform a method according to the present invention.
- A TV-set comprising: an analogue or digital input terminal to provide a video input signal; a device to generate a video signal out of the video input signal comprising video image data of a video line or part of the video line of the picture; an apparatus to perform a line-based motion estimation and a motion compensation according to the present invention and to provide a motion compensated image output signal; a screen to display a motion compensated picture using the motion compensated image output signal.
- A computer program product comprising a code, said code being configured to implement a method according to the present invention.
- A data carrier comprising a computer program product according to the present invention.

During the process of motion estimation several motion vectors are calculated which are suitable for being used in a subsequent motion compensation process. The calculation of the motion vector might be performed for every pixel of a frame or a field, or alternatively for only some of these pixels, e.g. several selected pixels within a line or part of a line. It is also possible that this motion vector is assigned to a predefined block or section of a frame or a field.
One basic idea of the present invention is the provision of a motion vector histogram which contains information which of the calculated motion vectors is used mostly and which is even rarely used in a current frame or field of a picture. This information stored in the motion vector histogram enables a significant and effective motion estimation process and thus also an efficient motion compensation process since only part of the calculated motion vectors is used. This consequently reduces the overall memory requirement and computational effort significantly.
Another advantage of the present invention is the fact, that the whole motion estimation and motion compensation process is getting more and more quick which is especially in modern video applications one of the key issue for the establishment of a highly precise picture of the TV-panel.
The present invention describes also a method for motion estimation and motion compensation which operates only in one direction and therefore performs the motion estimation and motion compensation operations using at least one single line buffer memory, the so-called line memory. This offers the possibility to reduce the chip embedded memory to one single line memory for the previous and one single line memory for the current field or frame. This advantageously enables significant silicon area reducing and cost saving implementations.
In a preferred embodiment of the present invention, the MEMC is limited to motion in the horizontal direction only, since most of the motion in natural scenes has this direction.
In video signal processing line memories are often used in other applications which already have access to the previous and current motion portrayal, e.g. like so-called de-interlacer applications or temporal noise reduction applications. In a preferred embodiment these already existing line memories of the video application are now additionally used also for MEMC operations. By using existing line memories of the video signal processing system, no further memory bandwidth has to be added to the memory bus and the memory bandwidth is kept uninfluenced.
Thus, this solution offers the possibility to accomplish the MEMC operations by adding a minimal or in the optimal case no additional memory to the video processing system.
Advantages, embodiments and further developments of the present invention can be found in the further subclaims and in the following description, referring to the drawings.
In a preferred embodiment the method is used for line-based motion estimation.

- In a preferred embodiment the step of calculating comprises the following sub-steps: assigning a counter to each different motion vector; counting the occurrences of identical motion vector by incrementing or decrementing the counter by a given value; comparing the different counter values assigned to the different motion vectors; ranking the different motion vectors in the order of their occurrences in a current frame or field.

In a preferred embodiment the motion vector having the highest rank and/or are the most often used motion vectors is/are selected.
In a preferred embodiment the method further comprises a motion compensation wherein the selected motion vector is used for motion compensation to interpolate a picture.
In a preferred embodiment the motion vectors having the highest rank and/or the most often used motion vector are stored in a memory.
In a preferred embodiment the step of calculating a histogram is done for the whole frame or field.
In a preferred embodiment the step of calculating a histogram is done for parts of the frame or field by splitting the frame or field into horizontal stripes and detect most often used vector for each stripe.
In a preferred embodiment the step of calculating a histogram is done to detect news ticker information, sub-titles or any other written information within a frame or field.
In a preferred embodiment a damping value which depends on the selected motion vector is used to adapt motion vectors with similar counter values.
In a preferred embodiment the histogram information of the rank of an motion vector is used to detect reliable and unreliable motion vectors.
In a preferred embodiment the motion vector contains only motion data for motion of an object in one direction and especially in the horizontal direction.
In a preferred embodiment image data of the previous frame is derived from a first line memory and image data of the current frame is derived from a second line memory.
In a preferred embodiment the first line memory and/or the second line memory is/are further used in a de-interlacer application and/or a temporal noise reduction application.
In a preferred embodiment a histogram generator is provided to establish a motion vector histogram for motion vectors to derive most and less used motion vectors in a current frame or field.
In a preferred embodiment the apparatus further comprises a histogram generator to provide a motion vector histogram for motion vectors to derive most and less used motion vectors in a current frame or field.
In a preferred embodiment the histogram generator further comprises a counting device for counting the occurrences of identical motion vector by incrementing or decrementing the counter by a given value; a ranking device which is designed to compare different counter values assigned to the different motion vectors and which is further designed to rank the different motion vectors on the basis of their occurrence in a current frame or field and to selects the most often used motion vector for the motion compensation.
In a preferred embodiment a motion vector histogram memory is provided to store the most often used motion vectors.
In a preferred embodiment a first line memory for storing image data of the previous frame and a second line memory for storing image data of the current frame are provided.
In a preferred embodiment the first line memory and/or the second line memory are configured to be further used in a de-interlacer device and/or a temporal noise reduction device.
In a preferred embodiment the apparatus is an integrated circuit and/or is implemented within a microcontroller or a microprocessor.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention and advantages thereof, reference is now made to the following description taken in conjunction with the accompanying drawings. The invention is explained in more detail below using exemplary embodiments which are schematically specified in the figures of the drawings, in which:

FIG. 1 shows the result of a standard (i.e. non motion compensated) FRU method;

FIG. 2 shows the change of position of a moving object between two consecutive received image fields;

FIG. 3 show the motion estimation principle for the line-based motion estimation by means of a current frame and the corresponding previous frame;

FIG. 4 shows a block diagram of a first embodiment of a line-based MEMC system according to the present invention;

FIG. 5 shows an example to illustrate the matching process of the motion estimation;

FIG. 6 shows the basic principle for the provision of a motion vector histogram;

FIG. 7 shows a block diagram illustrating an embodiment of the line-based motion estimation according to the present invention;

FIG. 8 shows a block diagram illustrating an embodiment of the line-based motion compensation according to the present invention using adaptive artefact concealments;

FIG. 9 shows a block diagram of a second embodiment of a line-based MEMC system according to the present invention using the line memories assigned to the de-interlacer device also for the motion estimation device.

In all figures of the drawings elements, features and signals which are the same or at least have the same functionality have been provided with the same reference symbols, descriptions and abbreviations unless explicitly stated otherwise.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

In the following description of the present invention first of all a short overview of the motion estimation and motion compensation is presented.
The MEMC method consists mainly of two sections, the motion estimation and the motion compensation method. The motion estimation performs the measurement of the motion and derives the velocity of the displayed regions in pixel per picture (i.e. field or frame). Also the direction of the motion will be indicated by a positive or negative sign. These measured motion information is described in the form of a motion vector. The motion vector is used for the motion compensation to interpolate the picture at the temporal accurate position and to avoid so-called judder effects and/or so-called motion blurring effects.
FIG. 3 shows the motion estimation principle for the line-based motion estimation by means of a current picture (field or frame) 10 (n) and the corresponding previous picture 11 (n−1). According to the temporal positions the motion vector 12, 13 will be split by its length into two parts, where the first vector part 12 points into the previous picture 11 and the second vector part 13 points into the current picture 10. For the interpolation of a missing picture 14 (n−½) between the current and the previous pictures 10, 11 pixels 15 from both temporal pictures 10, 11 are taken into account for the compensation. In line-based MEMC only the pixels 15 within the same line 16 are used at the same time and the MEMC is performed for a single line 16 of a field or frame only. For this kind of MEMC the pixels 15 of the current picture 10 are compared with the corresponding pixels 15 of the previous picture 11 to estimate and compensate the corresponding pixels 15 of the missing picture 14.
FIG. 4 shows a block diagram of a line-based MEMC system according to the present invention. The MEMC system is denoted by reference number 20. The MEMC system 20 comprises an input terminal 21, a bus 22, two line memories 23, 24, a motion estimation device 25, a motion compensation device 26 and an output terminal 27. It is assumed that the bus 22 is an external bus 22 and especially an external memory bus 22. However, it may also be possible, that the bus 22 is an internal bus 22. At the input side the bus 22 is connected to an external memory 28 device such as a SDRAM, a DDR-RAM, etc. Image data to be displayed in a panel 29 such as a plasma- or LCD-panel or a CRT-screen is stored in this external memory 28. Via the input terminal 21 and the memory bus 22 this image data X1, X1′ is transferred to both line memories 23, 24. According to the present invention only two line memories 23, 24 are needed whereas the first line memory 23 is used for buffering image data X1 of the previous picture and the other line memory 24 is used for storing the image data X1′ of the current picture.
A line memory 23, 24 as used in the present patent application indicates an embedded memory of a size of one video line of a frame or a field or at least less of the incoming video signal stream or actually processing video signal stream. A field denotes a video image or picture which comprises either odd or even lines. A frame denotes a video image comprising of the complete video information for one picture, i.e. of a field for the odd lines and the corresponding field for the even lines. A line denotes a full horizontal row within a field of one video picture or at least a part of this row.
Both of the line memories 23, 24 are coupled—on their output sides—to the motion estimation device 25 and to the motion compensation device 26. This enables the image data X1, X1′ stored in the line memories 23, 24 to be transferred to the motion estimation device 25 and to the motion compensation device 26, respectively. In FIG. 4 the corresponding data signals to the motion estimation device 25 are denoted by X2, X2′ and the corresponding data signals motion compensation device 26 are denoted by X3, X3′.
The motion estimation device 25 generates a motion vector signal X4 out of the image data X2, X2′ stored in the line memories 23, 24 by employing a matching process. This vector signal X4 is transferred to the motion compensation device 26. The motion compensation device 26 performs a motion compensation using the image data X3, X3′ stored in the line memories 23, 24 and applying the vector data X4 to this image data X3, X3′. At the output terminal 27, the motion compensation device 27 provides a video signal X5 which comprises information for a motion compensated picture. This video signal X5 is transferred via the output terminal 27 to a display 29, such as a LCD-panel 29 or the like.
With regard to FIG. 5, hereinafter the operation of the motion estimation device 25 is described in more detail:
For the motion estimation a matching process is employed to select a corresponding series of pixels 32 which fits best to a given amount of pixels 30. For this selection a given amount of pixels 30 of a line of a current frame around the centre pixel 31 for which the motion shall be determined is taken from a line memory 24 of the current frame 32. Hereinafter this given amount of pixels 30 is denoted to as series of pixels 30. In the present embodiment a series of pixels 30 comprises 9 single pixels 33. It is self-understood that a series can also comprise a greater or a smaller amount of pixels 33.
For the selection the luminance profile of the pixels 33 is used as the matching parameter. Luminance is a photometric measure of the density of luminous intensity in a given direction. It describes the amount of light that passes through or is emitted from a particular area, and falls within a given solid angle. Thus, luminance is the photometric measure of the brightness in a frame of a motion picture. If the luminance is high, the picture is bright and if it is low the picture is dark. Thus, luminance is the black and white part of the picture.
This luminance profile is used to find out that series of nine pixels 34 out of the previous frame 35 which fits best with the series of nine pixels 30 of the current frame 32. In the embodiment of FIG. 5 the luminance profile of the series of nine pixels 30 of the current frame 32 are compared with the luminance profiles of several corresponding series of nine pixels 34 of the previous frame 35. In order to derive the true motion the series of nine pixels 30 will be shifted over the search range in the horizontal direction 36. It is assumed that that series of nine pixels 34 of the previous frame 35 which shows the best luminance profile matching (with the series of nine pixels 30 of the current frame 32) is the correct series of pixels. These series of pixels 30, 34 are then used for the computation of the motion vector.
A typical value for the search range is, e.g. 64 pixels (+31 . . . −32). However, it may also be possible to use less than 64 pixels, however, then the quality of the result of this comparison is increasingly going down. On the other hand it is also possible to use more than 64 pixels. Then the quality of the selection result is going up, however, this needs more computational effort. Therefore, typically a trade-off which provides an optimization between best quality of the selection result and simultaneously a minimum computation effort is employed.
In a preferred embodiment for each selected motion vector 37 a single matching process is performed in the way described above. This matching process is performed by assigning a quality degree and/or a failure degree for each series of pixels 30. Then, a quality-degree and/or a failure degree are assigned to each one of those series of pixels 30 which are undergoing the matching process. Those series of pixels 30 having the highest quality-degrees and/or the lowest failure degrees are selected as most probable series of pixels. These series of pixels 30 are then used for computing the motion vectors for the horizontal motion. Typically, but not necessarily a SAD method (SAD=sum of absolute difference) and/or ADRC method is used for the comparison of the luminance and/or chrominance values.

Using Pre-Selected Motion Vector Samples for the Motion Estimation:

Assuming the motion of an object in the scene will be constant from frame to frame and the object is larger than a series of pixels (e.g. the above mentioned 9 pixels), then the matching process can then be performed more efficiently if a set 38 of pre-selected motion vectors 37—the so-called motion vector samples 37—are checked for a matching of the luminance profile (see FIG. 5). For example, one selected motion vector 37 can be taken from the neighbouring pixel. A second selected motion vector can be taken from the previous line, if the already estimated motion vectors are stored in a vector memory specially designed for the different motion vector samples.
The zero-vector which indicates no motion of the object is typically one of the most used motion vector samples. This zero-vector is used in order to more efficiently detect regions within a picture showing no motion. In principle the amount of pre-selected motion vectors 37 which will taken into account depend strongly on what kind of motion vector quality is desired.

Variation of Selected Motion Vectors:

In order to set up the process of motion estimation and to follow the deviation from the constant motion, a variation of certain pre-selected motion vectors is required for test operation purposes. That means that for pre-selected motion vector samples a certain amount of motion will be added or subtracted. This can be done by applying a variance with different amount of motion speed to these motion vectors. The tested implementation checks between odd pixels and even pixels alternating an update of +/−1 pixels and +/−4 pixels on the previously determined motion vector. The selection of the variance is adjustable and variable as required or as the need arises and depends e.g. on the resolution of the incoming video signal.
For the line-based motion estimation it is very advantageous that the motion vector will converge quickly for the real motion in the scene. Therefore, the selection of the tested motion vectors is treated differently for the first line of a frame or field. For the first line of a frame or field testing is not possible in the normal way since a line above the first line which is needed for testing is not existing. In the first line of each field the selected motion vectors which normally test the motion vectors of the line above are loaded with vector values, which e.g. vary according to a triangle function from pixel to pixel. The triangle function oscillates between an adjustable minimum value and an adjustable maximum value. For that purpose also other regular oscillating functions e.g. a saw tooth function, a sinusoidal function, and the like may be employed for the determination of the motion vector of the first line.

The Matching Process:

In a preferred embodiment the matching process assigns a failure value to each tested motion vector. In another embodiment this value may be also a quality value. It might also be possible to evaluate as well a failure value and a quality value for the matching process. Preferably, the sum of the absolute difference (SAD) is used as the failure value or to at least derive the failure value. Ideally, to find the optimal motion vector a failure value of zero is needed. However, typically the failure value is different from zero. Therefore, the motion vector corresponding with the lowest failure value is then selected as the most probably motion vector representing the motion of an object in the local scene.

Attenuation of the Vector Selection, Vector Damping:

In a preferred embodiment a damping value is used which depends on the vector attenuation of the different motion vectors. This enables to control the motion vectors with equal failure values and/or to furnish the motion vector selection process with a certain direction.

Vector Memory:

The different motion vectors are advantageously stored in a vector memory. These motion vectors can be then—if required—fetched from the vector memory for further processing and/or for the motion estimation of the next pixels.
The motion estimation process forms a recursive process. Therefore, the size of this vector memory mainly depends on the desired quality level of the matching process. In one embodiment, the tested implementation comprises only one line of a vector memory. In this vector memory every second motion vector will be stored alternately, in order that an access of the motion vectors from the measured line above is possible.

Robustness Improvement by Providing a Vector Histogram:

In a preferred embodiment a motion vector histogram is calculated in order to create a highly reliable and homogeneous field of motion vectors. This vector histogram allows a vector majority ranking to derive most and less used motion vectors in an actual scene.
FIG. 6 shows a preferred embodiment to illustrate the basic principle for the provision of a motion vector histogram accordingly to the present invention. FIG. 6 shows a vector histogram generator 40 to provide a motion vector histogram. In the embodiment in FIG. 6 the vector histogram generator 40 comprises a switching device 41, which is controlled by an +1-incrementing device 42. The switching device 41 is controlled on the one hand by a motion vector 43 information and on the other hand by the incrementing device 42 which shift the switching device 41 to the next input terminal of a counting device 45 whensoever the next identical motion vector 43 occurs. The counting device 45 which comprises different counter cells 44 counts the occurrence of each motion vector and increments the counter by +1 for each occurrence of the motion vector. A ranking device 46—which e.g. comprises a simple comparing means—is coupled to the output terminals of the different counter cells 44 of the counting device 45. This ranking device 46 selects the most often used motion vector and applies this motion vector for the estimation determination. The most often used motion vector may be then stored in a motion vector histogram memory 47.
The provision of a motion vector histogram can be done either for the whole frame or field or only for parts of the frame or field. It is very efficient to split the picture into horizontal stripes and return a most often used vector for each stripe. In very a preferred embodiment news ticker information within a picture can be detected in that way very reliable.
FIG. 7 shows a block diagram illustrating an embodiment of the line-based motion estimation according to the present invention as described above and as implemented in a motion estimation device 25 as shown in FIG. 4.
The motion estimation device 25 comprises a matching device 80, a cost/quality function device 81 and a vector selector device 82, which are arranged in series connection between the input side 83 of the motion estimation device 25 where the image data signals X1, X1′ stored in the both line memories 23, 24 are provided and the output side 84 of the motion estimation device 25 where the motion vector signal X4 in present. In the device elements 80-82 a matching process and a vector selection as described with regard to FIG. 5 is implemented.
The motion estimation device 25 further comprises a vector quality device 85 which is connected on the one hand to the input side 83 and on the other hand to the output side 84. The vector quality device 85 generates a quality signal X6 comprising an information of the vector quality out of the image data signals X1, X1′ and the motion vector signal X4.
The motion estimation device 25 further comprises a vector histogram device 86 and a vector majority device 87 which are arranged in series connection in a feedback path between the output side 84 and the matching device 80. Here, in the device elements 86, 87 a vector histogram is generated to provide a ranking of most and less used vectors in the actual scene as shown and described with regard to FIG. 6. Thus, the elements 86, 87 correspond to the vector histogram generator 40 of FIG. 6.
The motion estimation device 25 may further comprise a further line memory 88 to store the motion vector data X4 and/or the data X6 for the vector quality.
The motion estimation device 25 further comprises a vector sample device 89. This vector sample device 89 is also arranged in the feedback path and is connected at its input side with the line memory 88, the vector majority device 87 and advantageously with a further device 90. This further device 90 performs a variation of the motion vector samples by using a special signal having a certain magnitude, e.g. a sinusoidal signal, a saw tooth signal or the like. This certain signal is then used for a testing and/or matching process and/or an up-dating process of the first line of a frame or field. However, it might also be possible to randomly up-date different lines of the frame or field. On its output side, the vector sample device 89 is connected at its output side to the matching device 80.
The motion estimation device 25 further comprises a vertical motion estimation device 91. For vertical motions the above described one-dimensional motion estimation algorithm is not able to compensate fully motion in the vertical direction. However, the occurrence of vertical motions can be used to reduce the compensation in same regions of the picture by splitting the picture into different regions to derive vertical motion for each region. In this case the luminance values of the lines in the different region of the same picture will be summed up and stored individually for each line of this picture. This results in an accumulated vertical profile for different regions of the same picture. Then, the whole picture can be divided into smaller regions to derive a vertical motion for each of these regions. This vertical motion estimation process is performed in the vertical motion estimation device 91 which is connected to the input side 83 and which provides at its output side a sector based vertical motion index X7.
Thus, the vertical MEMC as sketched above can be performed independently of horizontal MEMC and also in combination with the horizontal MEMC, wherein the combination can be performed in dependence on a certain situation or the motions present, respectively. Further, such a methodology allows an implementation of vertical MEMC, which does not need large amounts of additional memory capacity to analyze data of consecutive frames being the case in the most methodologies of the prior art.
The motion estimation device 25 further comprises a vector damping device 92. In this damping device 92 a damping value as described above may be used to damp vector samples of the vector sample device 89 and to provide damped vector samples to the vector selector 82.
Hereinafter the motion compensation process which is performed in the motion compensation device 26 of FIG. 4 is described with regard to FIG. 8 in more detail. FIG. 8 shows a block diagram illustrating an embodiment of the line-based motion compensation according to the present invention using adaptive artefact concealments as described above.
The motion compensation device 26 comprises a compensation device 100 which performs the temporal motion interpolation according to the motion vectors X4 estimated by the motion estimation device 25. In a preferred embodiment the compensation device 100 comprises a Median Filter which uses as input data the luminance values of the vector compensated previous line, the vector compensated current and the uncompensated previous line. Additionally, also the chrominance values can be compensated.
Depending on the vector quality a replacement vector indicated as reliable vector will be searched in the local area of the vector memory from the line above. If no reliable vector can be found the adaptive blurring typically tries to cover this artefact.
The motion compensation device 26 further comprises a vertical motion control device 101 which provides a control signal X8 to the compensation device 100 in order to incorporate also information of a vertical motion to the compensation device 100.
The motion compensation device 26 further comprises a bad vector modification device 102. Based on information X4, X6 provided by the motion estimation device 25 the bad vector modification device 102 modifies bad vectors. These information X9 about modified bad vectors is then used—together with the control signal X8—to perform the motion compensation within the compensation device 100. The compensation device 100 then generates at its output side a motion compensated image data signal X10.
The motion compensation device 26 further comprises an adaptive blurring device 103. Based on the motion compensated image data signal X10 and a blurring control signal generated by the bad vector modification device 102 this adaptive blurring device 103 performs an adaptive blurring. The adaptive blurring device 103 generates an adaptive blurred image data signal X5′ which might correspond to the image signal X5 of FIG. 4.
FIG. 9 shows a block diagram of a second embodiment of a line-based MEMC system according to the present invention using the line memories assigned to the de-interlacer device also for the motion estimation device.
Unlike the first embodiment in FIG. 4 a de-interlacer device 113 is arranged between the line memories 110, 111, 112 and the motion compensation device 26. The de-interlacer device 113 is typically used to convert a field represented by video data stream into a full frame which is then also represented by another video data stream.
On-chip solutions for video processing which are memory-based have already existing internal line buffers 110-112—the so-called line memories 110-112—which carry video data from the previous and current field or frame. These line buffers 110-112 can be located e.g. within temporal noise reductions or de-interlacing units 113 which operate motion adaptive. With the proposed line-based MEMC these line buffers can be reused additionally for the motion estimation. For that purpose and in order to reduce motion judder artefacts from movie sources, a movie detector which indicates the current interpolated sequence of pull-down mode is used. A line buffer selector transfers the video signal data to the motion estimation device according to the previous and the current video input signal. This technique allows using already existing memory resources also for motion estimation which also prevents additional bandwidth for the temporal up-conversion process. Therefore, the chip area for the motion estimation and the motion compensation can be reduced to a minimum.
The de-interlacer device 113 uses three line memories 110, 111, 112 coupled on their input side to the memory bus 22 and providing at their output side line data. This line data provided by the line memories 110, 111, 112 is processed within the de-interlacer device and then provided to the motion compensation device 26. According to the present invention, these line memories 110, 111, 112 are additionally used also for the motion estimation device 25. For this purpose, the system 20 additionally comprises a selector device 114, where a movie sequence X0 is provided to this selector device 114. This movie sequence X0 may be then stored in an external memory 28 via the memory bus 22 and can be read out from this external memory 28 through the line memories 110, 111, 112. For an IMC operation, this data stored in the line memories 110, 111, 112 of the de-interlacer device 113 can be also used for MEMC. For this purpose the data stored in the line memories 110, 111, 112 is then provided as well to the motion estimation device 25 and the motion compensation 26 device.
While embodiments and applications of this invention have been shown and described above, it should be apparent to those skilled in the art, that many more modifications (than mentioned above) are possible without departing from the inventive concept described herein. The invention, therefore, is not restricted except in the spirit of the appending claims. It is therefore intended that the foregoing detailed description is to be regarded as illustrative rather than limiting and that it is understood that it is the following claims including all equivalents described in these claims that are intended to define the spirit and the scope of this invention. Nor is anything in the foregoing description intended to disavow the scope of the invention as claimed or any equivalents thereof.
It is also noted that the above mentioned embodiments, examples and numerical data should be understood to be only exemplary. That means that additional system arrangements and functional units and operation methods and standards may be implemented within the MEMC-system.
It is self understood that the above mentioned numerical data is merely illustrative and may be adapted to best provide an optimized blurring effect.
At this point, it should be also mentioned, that the present invention is not based necessarily on so-called line-based MEMC systems, although in the above embodiments of the present invention always reference is made on line-based MEMC systems. In fact, the present invention is related generally to all implementations using motion estimation of video image data, i.e. especially so-called block-based motion estimation, line-based motion estimation and the like. It is self understood that for those implementations which do not apply line-based motion estimation typically other memory means than line memories maybe employed.

Claims

1. A method for motion estimation in video image data, especially for field/frame rate up-conversion in consecutive frames/fields of a motion picture, comprising the steps of:

providing a video signal comprising video image data of a video line or part of the video line of the picture;

performing the motion estimation by detecting and analysing the video image data and by deriving motion vectors depending on the detected motion;

calculating a histogram for motion vectors to derive most and less used motion vectors in a current frame or field.

2. The method according to claim 1, wherein the method is used for line-based motion estimation.

3. The method according claim 2, wherein the step of calculating comprises the following sub-steps:

assigning a counter to each different motion vector;

counting the occurrences of identical motion vectors by incrementing or decrementing the counter by a given value;

comparing the different counter values assigned to the different motion vectors;

ranking the different motion vectors in the order of their occurrences in a current frame or field.

4. The method according to at claim 3, wherein the motion vector having the highest rank and/or are the most often used motion vectors is/are selected.

5. The method according to claim 4, wherein the method further comprises a motion compensation wherein the selected motion vector is used for motion compensation to interpolate a picture.

6. The method according to claim 5, wherein motion vectors having the highest rank and/or the most often used motion vector are stored in a memory.

7. The method according to claim 6, wherein the step of calculating a histogram is done for the whole frame or field.

8. The method according to claim 6, wherein the step of calculating a histogram is done for parts of the frame or field by splitting the frame or field into horizontal stripes and detect most often used vector for each stripe.

9. The method according to claim 8, wherein the step of calculating a histogram is done to detect news ticker information, sub-titles or any other written information within a frame or field.

10. The method according to claim 9, wherein a damping value which depends on the selected motion vector is used to adapt motion vectors with similar counter values.

11. The method according to claim 10, wherein the histogram information of the rank of a motion vector is used to detect reliable and unreliable motion vectors.

12. The method according claim 11, wherein the motion vector contains only motion data for motion of an object in one direction and especially in the horizontal direction.

13. The method according to claim 12, wherein image data of the previous frame/field is derived from a first line memory and image data of the current frame/field is derived from a second line memory.

14. The method according to claim 13, wherein the first line memory and/or the second line memory is/are further used in a de-interlacer application and/or a temporal noise reduction application.

15. An apparatus for motion estimation in video image data, especially for motion estimated and compensated field/frame rate up-conversion in consecutive frames or fields of a motion picture, wherein the apparatus is configured to perform a method according to at least one of the preceding claims.

16. The apparatus according to claim 15, wherein a histogram generator is provided to establish a motion vector histogram for motion vectors to derive most and less used motion vectors in a current frame or field.

17. The apparatus according to claim 16, wherein a histogram generator is provided to provide a motion vector histogram for motion vectors to derive most and less used motion vectors in a current frame or field.

18. The apparatus according to claim 17, wherein the histogram generator further comprises

a counting device for counting the occurrences of identical motion vector by incrementing or decrementing the counter by a given value;

a ranking device which is designed to compare different counter values assigned to the different motion vectors and which is further designed to rank the different motion vectors on the basis of their occurrence in a current frame or field and to selects the most often used motion vector for the motion compensation.

19. The apparatus according to 18, wherein a motion vector histogram memory is provided to store the most often used motion vectors.

20. The apparatus according to 19, wherein a first line memory for storing image data of the previous frame and a second line memory for storing image data of the current frame are provided.

21. The apparatus according to claim 20, wherein the first line memory and/or the second line memory are configured to be further used in a de-interlacer device and/or a temporal noise reduction device.

22. The apparatus according to 21, wherein the apparatus is an integrated circuit and/or is implemented within a microcontroller or a microprocessor.

23. A TV-set comprising:

an analogue or digital input terminal to provide a video input signal;

a device to generate a video signal out of the video input signal comprising video image data of a video line or part of the video line of the picture;

an apparatus to perform a line-based motion estimation and to provide a motion compensated image output signal;

a screen to display a motion compensated picture using the motion compensated image output signal.

24-25. (canceled)