Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/56—Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding

Definitions

• the present patent application relates in general to improved motion estimation in order to overcome the aperture problem.
• 3D recursive search 3D recursive search
• Motion estimation algorithms were based on the assumption, that the luminance or chrominance value of a pixel may be approximated by a linear function of the position. This assumption may only be correct for small displacements. The limitation may, however, be resolved by pixel-based motion estimation methods (PEL-recursive methods).
• PEL-recursive methods pixel-based motion estimation methods
• the implementation of motion estimation also includes block estimation.
• a displacement vector D is assigned to the center X of a block of pixels B( X ) in a current field n, by searching a similar block within a search area SA(X ), also centered at X , but in a temporary neighbouring field, for example n-1, n+1.
• the similar block may have a center, which may be shifted with respect to X over the displacement D (X , n).
• D (X , n) To find D (X , n), a set of candidate vectors C are evaluated.
• an error measure ⁇ (c , X , n ) which quantifies block similarities is calculated.
• the displacement vector D (X , n) resulting from the full- search block- matching process is a candidate vector C which yields the minimum values of at least one error function ⁇ (c , X , n ) . This can be expressed by:
• the error value for a given candidate vector C can be a function of the luminance values of the pixels in the current block and those of the shifted block from a previous field, summed over the block B(X).
• the error value can also be any other function of pixel values, and can be expressed as a sum of cost functions:
• ⁇ (c, X,n) ⁇ Cost (F (X, ⁇ ), F (X - C ,n - p )) xs B (X )
• a cost function can, for example, be the Sum of Absolute Difference between two blocks of pixels.
• the error value of a given candidate vector can also be considered as a Cost function or constraint function.
• Constraint functions which relate to the nature of motion within the image can, for example, be the intensity conservation constraint or the spatial coherence constraint. The choice of constraints is best when they separately lead to orthogonal subspaces of solutions. From the image content, it may be possible to impose physical restrictions on the set of possible motion vectors, for example, like motion smoothness, inertia of objects, etc. Mathematically, these restrictions can take the form of an equation having as variables the motion parameters/motion vectors (constraint on the motions parameters/motion vectors), and it can be added to or combined with the cost function to be minimized.
• Fig. 1 depicts a block-matching motion estimation algorithm as described above. Shown are two temporal instances n-1, n of an image sequence 2. Within image sequence 2, various blocks 4 at horizontal position X and vertical position Y are determined.
• various candidate vectors C 8 may be evaluated applying the above-mentioned error measure ⁇ (c , X , n ) .
• One possible error function may be the assumed absolute difference (SAD) criterion, which is
• F (x,n) is the luminance value of pixels within block 4.
• the displacement vector D is assigned to the center J of a block 4 of pixel positions B( X ) in the current image by searching a similar block 10 within a search area SA( X ) 6, also centered at X , but in a previous or following image. This may be a temporally previous or following image or field.
• a correlation measure between the two blocks 4, 10 is therefore optimized to identify the displacement vector D .
• error criteria such as Mean Square Error, and Normalized Cross Correlation Function may be used. Particularly the last may be used in case of calculating in the Fourier domain.
• error criterion may be the number of significantly different pixels.
• the above criteria can be cost functions. The physical restrictions, such as boundary conditions, could provide new candidates for independent cost functions
• a block matching method enables finding candidate vectors close to the true- motion vectors.
• a penalty system has been adopted, which adds to the error function a penalty value, which penalty value can depend on the type of prediction, i.e. spatial prediction of temporal prediction.
• the error/cost functions described above do not fully determine the 2-dimensional motion parameters. These cost functions may suffer from the so called aperture problem.
• additional information about the image content e.g. some physical constraints i.e. the motion is smooth, the object have inertia, etc., might be necessary.
• the SAD does not fully determine the two dimensional components of a motion, but just a component perpendicular to the edge.
• a motion estimation using SAD solely is degenerate, being only determined up to a constant tangential to the single orientation in the sequence, e.g. an edge.
• This problem is also called the aperture problem of motion compensation.
• This problem has been solved by simultaneously imposing additional error functions, such as boundary conditions, or constraints related to the nature of motion in the video scene.
• a necessary condition to remove the degeneracy in determining the motion vector is that each constraint should lead to a reciprocally orthogonal subspace solution.
• the physical constraints imposed on the space of motion vector solutions i.e. smooth motion, object inertia, boundary conditions, etc.
• Each motion vector in a subspace obeys at least one constraint.
• a set of candidate motion vectors with orthogonal subspaces CS max can be used.
• a set of m constraints with error functions ⁇ m can lead to a solution for the motion estimated vector
• one cost function can be degenerate with multiple minima and an absolute minimum for candidate motion vector C kmm .
• the total cost function can have a local minimum for this constraint at the local minimum of the one constraint. If a local minimum of one constraint is much smaller, than the local minima of the other constraints, the non- minimum contributions from the other cost functions can be compensated. Thus, the total error function results in an erroneous minimum, which might not be the minimum value for all the individual constraints.
• the inequality ⁇ (c mm ,X,n) ⁇ ⁇ / ⁇ / (c mm ,X,n)+ ⁇ 4 ⁇ 4 (c mm , X,n) l ⁇ k
• one object of the present application is to provide a solution, which overcomes the degeneration of cost functions.
• Another object of the invention is to provide a motion estimation, which overcomes the aperture problem.
• a further object of the invention is to provide motion estimation, which result in improved estimated motion vectors.
• the application provides according to one aspect a method for determining estimated motion vectors within image signals comprising creating at least two candidate motion vectors for at least one pixel within an image of the signal, calculating for each of said candidate motion vectors at least two error criteria, and choosing the candidate motion vector that minimizes a non- linear function of the error criteria as the estimated motion vector for the at least one pixel.
• Signals according to embodiments can be any images sequence, for example a video sequence. Images within the signals can be composed of pixels. Pixels can be image elements describing the luminance and chrominance of the particular part of the image. A plurality of adjacent pixels within the image can be understood as pixel block.
• Elements within the image can be subject to motion of several frames. Motion of the elements can be described by motion vectors. Motion vectors can describe the direction and speed of movement of particular pixels or blocks of pixels. Motion estimation can be understood as calculating a probability of motion. Motion vectors which are most likely to describe the actual motion within the image can be calculated using motion estimation. With these motion vectors, it can be possible to predict images of following frames. The estimated motion vectors can also be used for de-interlacing interlaced images.
• Candidate motion vectors can be a set of possible vectors describing possible motion of pixels or blocks of pixels.
• the set of candidate motion vectors can be used to determine one estimated motion vector which suits best the actual motion within the image.
• high quality video format conversion algorithms such as for example de- interlacing and temporal up-conversion, and computer vision applications and video compression may require motion estimation.
• the aperture problem during motion estimation arises from the absence of additional knowledge about the nature of the motion in the scene. From signals containing single orientations, i.e. edges, the two dimensional motion components are not determined or are only determined up to a constant in the direction tangential to the edge.
• This uncertainty may lead to multiple minima in a cost function and to a degeneracy of motion estimation.
• the degeneracy may lead to erroneous evaluation of motion vectors, which may lead to artefacts in the video format conversion.
• this problem is solved by imposing multiple error functions with different constraints, which are related to the nature of motion. Such constraints can be intensity conservation or spatial coherence. The choice of the constraints is preferred such that they separately lead to orthogonal solutions.
• the application of a non- linear combination of multiple error functions, which minimizes the most expensive constraint over a set of candidate motion vectors may lead to optimized motion estimation.
• the most expensive cost function can take the maximum value. By minimizing the maximum of all the cost functions, it is possible to minimize the most expensive cost function.
• the at least two candidate motion vectors describe possible displacements of a pixel within a search area. Such displacements can be in the x- and y-direction.
• the vectors can describe the direction of motion by their x- and y- components.
• the speed of motion can be described by the absolute value of the vectors.
• the at least two candidate motion vectors are created using spatial and/or temporal prediction according to embodiments. For example, in scanned images providing scanned image lines, causality prohibits the use of spatial prediction in blocks of the image not yet been transmitted. Instead, temporal prediction can be used.
• the error criteria can be at least one of a summed absolute difference criterion, a mean square error criterion, a normalized cross correlation criterion, or a number of significant pixels criterion. These error criteria can be understood as constraints.
• the non- linear function is the maximum of the error criteria with:
• ⁇ (c,X,n) MAX ⁇ m (c,X,n) ⁇
• the non- linear function can also be the median of the error criteria.
• at least one of the error criteria is calculated from absolute differences of interpolated pixels. At least one of the error criteria can be calculated from absolute differences of an interpolated pixel and an intra-field interpolated pixel. At least one of the error criteria can also be calculated from an absolute difference between one pixel from a current frame or field and a motion compensated interpolated pixel from a previous or following de-interlaced frame or field.
• Another aspect of the invention is a computer program for determining estimated motion vectors within image signals the program comprising instructions operable to cause a processor to create at least two candidate motion vectors for at least one pixel within an image of the signal, calculate for each of said candidate motion vectors at least two error criteria, and choose the candidate motion vector that minimizes a non- linear function of the error criteria as the estimated motion vector for the at least one pixel.
• a further aspect is a computer program product for determining estimated motion vectors within image signals with a program tangibly stored thereon with instructions operable to cause a processor to create at least two candidate motion vectors for at least one pixel within an image of the signal, calculate for each of said candidate motion vectors at least two error criteria, and choose the candidate motion vector that minimizes a non-linear function of the error criteria as the estimated motion vector for the at least one pixel.
• FIG. 1 an illustration of a block matching
• Figs. 2a-b an illustration of a candidate set of vectors of a recursive search block-matcher
• Fig. 3 an illustration of a block matching on a sequence with a single orientation
• Fig. 4 an illustration of orthogonal solution subspaces that can lead to unique solution
• Fig. 5 an illustration of multiple criteria cost functions.
• the block-matcher as depicted in Fig. 1 has been described above.
• a block 4 in the current image n and a test block 10 within the search area 6 in the previous image n-1 are connected using candidate vector C 8.
• a correlation measure, the match error between the two blocks 4, 10 may be optimized to identify the best candidate vector C 8.
• different test block 10 using different candidate vectors C 8 may be tested and the match error may be minimized for a best matching candidate vector.
• Searching the minimum of a match criterion in a block-matcher is a two dimensional optimization problem for which many solutions are available.
• One possible implementation uses a three-step block-matcher, a 2D logarithmic, or cross search method, or the one-at-a-time-search block-matching.
• Different block-matching strategies are disclosed in G. de Haan, "Progress in Motion Estimation for Consumer Video Format Conversion", IEEE transactions on consumer electronics, vol. 46, no. 3, August 2000, pp. 449-459.
• 3D recursive search block-matcher 3D recursive search block-matcher
• This 3D RS accounts for that for objects larger than blocks, a best candidate vector may occur in the spatial neighbourhood of a pixel or block.
• a best candidate vector may occur in the spatial neighbourhood of a pixel or block.
• causality prohibits the use of spatial prediction vectors 4 Ds, right and below the current block Dc 4a.
• temporal prediction vectors D 4c need to be used.
• spatial prediction vectors Ds 4b and temporal prediction vectors Dt 4c are available.
• Fig. 2b shows the use of two spatial prediction vector Ds 4b and one temporal prediction vector Dt 4c to predict a current block 4a.
• CS max is defined as a set of candidate vectors C describing all possible displacements
• n and m are constants limiting SA( X ).
• X, Y may define the block width and height, respectively.
• Causality and the need for pipelining in the implementation prevents that all neighbouring blocks are available, and at initialization, all vectors may be zero.
• Fig. 2a illustrates the relative position of the current block Dc 4a and the block from which the result vectors are taken as candidate vectors Ds 4b, Dt 4c.
• This candidate set CS implicitly assumes spatial and/or temporal consistency.
• the problem of zero vectors at initialization may be accounted for by adding an update vector.
• One possible implementation of omitting some spatio-temporal predictions from the candidate set is depicted in Fig. 2b, where the candidate set CS((X,n) may be defined by
• update vectors U 1 (X ⁇ ) and U 2 (X,n) may be alternately available, and taken from a limited fixed integer, or non-integer, update set, such as
• a model capable of describing more complex object motion than only translation, for instance rotation, or scaling may use segmenting the image in individual objects and estimating motion parameter sets for each of these objects. As the number of blocks usually exceeds the number of objects with more than an order of magnitude, the number of motion parameters that needs to be calculated per image is reduced. However, the calculation complexity increases.
• a pixel block object may be determined, which may be referred to as a group of pixels.
• a motion parameter for example a motion vector for each group of pixels, may be determined.
• Candidate vectors may be tested by calculating the summed absolute difference SAD between the luminance values of the group of pixels in the current image and the corresponding motion compensated luminance values in a second temporally neighbouring image. Two temporal instances may be used in order to estimate the motion parameter sets of a local group of pixels.
• the determination of the estimated motion vector from the candidate motion vectors using the summed absolute difference criterion or any other single criterion as described above, does not fully determine the 2-dimensional motion parameters.
• video content can comprise one single edge 12.
• the motion in the image can be the motion of the single edge 12.
• Motion estimation for one single block 4 can be done using candidate motion vectors 8a-8c.
• the test blocks 10a- 10c for the candidate motion vectors 8a- 8c result in the same value for the cost function.
• using one single cost function does not fully determine the two dimensional components of the motion, but just the component perpendicular to the edge.
• Providing one single cost function is degenerate, being only determined up-to a constant in the direction tangential to the edge. This problem may also be called the aperture problem in motion estimation.
• cost functions can be boundary conditions, or constraints related to the nature of motion in video scenes.
• One important condition to remove the degeneracy in determining the motion vector is that each cost function should lead to a reciprocally orthogonal subspace solution.
• Fig. 4 represents such orthogonal subspaces of the candidate motion vector set CS" 18 ", minimizing to different cost functions E 1 , and ⁇ 2 .
• Cost function E 1 can be characterized by candidate vectors 18a-18e.
• Cost function ⁇ 2 can be represented by candidate vectors 19a-19d.
• Each subspace is characterized by the fact that all its elements have equal, and well defined motion components in the direction perpendicular to the edge. Imposing simultaneously both constraints may lead to a unique solution, if the subspace of solutions of these two constraints are orthogonal to each other.
• a unique solution may be the candidate vectors 18c, 19c.
• a set of m constraints can be imposed simultaneously on the set of vector candidates. These constraints can be:
• ⁇ m (c,X,n) y Co S t m (F ⁇ x,n),F(x - C,n -p) ⁇
• the application renders the overall cost function more robust. This may be done by a non- linear combination of the cost functions, as
• Fig. 5 illustrates the effect of individual cost functions ⁇ 15 ⁇ 2 on the total cost, when calculated differently.
• Slopes 20a, 20b indicate to individual cost function.
• Slope 20a has two local minima and one absolute minimum.
• Cost function 20b has one absolute minimum.
• a total cost function ( ⁇ 15 ⁇ 2 )/2 22 has its minimum value at the absolute minimum of one of the cost functions.
• the illustrated cost functions represent the variation over a candidate vector set of two arbitrary constraints E 1 and ⁇ 2 .
• the linear cost function falls into one of the minima of constraint ⁇ 2 .
• the non- linear combination of the cost function MAX( ⁇ ⁇ , ⁇ 2 ) is illustrated as slope 24.
• the minimum of this total cost function is enforced to take a value close to the common local minimum of both individual constraints. From this example it becomes apparent that the total cost function lies closer to the minima of each of the constraints than a linear combination.
• the missing pixel at the position x can be attributed the value of the luminance calculated at that position by means of a de- interlacing algorithm.
• GST generalized sampling theory
• This first constraint is not sufficient to build up a robust total cost function.
• a second constraint can be imposed. This second constraint can make use of the already de-interlaced previous frame n-1. This allows performing a motion compensated bilinear interpolation to estimate existing pixel values in the current field.
• the application provides a robust solution for motion estimation, which can be applied to different video format conversion algorithms.
• Motion estimation by means of a non- linear combination between different motion estimation criteria leading to orthogonal solutions is proposed.

Priority Applications (1)

Applications Claiming Priority (4)

Publications (1)

Family

ID=37310409

Family Applications (1)

Country Status (5)

Families Citing this family (6)

Family Cites Families (2)

• 2006
  • 2006-06-15 EP EP06765751A patent/EP1897376A2/de not_active Withdrawn
  • 2006-06-15 KR KR1020087001724A patent/KR100942887B1/ko not_active IP Right Cessation
  • 2006-06-15 WO PCT/IB2006/051919 patent/WO2006136983A2/en active Application Filing
  • 2006-06-15 JP JP2008517649A patent/JP5025645B2/ja not_active Expired - Fee Related
  • 2006-06-15 CN CN2006800304778A patent/CN101288310B/zh not_active Expired - Fee Related

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events