WO1999016251A1 - Estimation de mouvements et interpolation compensee en mouvement - Google Patents

Estimation de mouvements et interpolation compensee en mouvement Download PDF

Info

Publication number
WO1999016251A1
WO1999016251A1 PCT/IB1998/001241 IB9801241W WO9916251A1 WO 1999016251 A1 WO1999016251 A1 WO 1999016251A1 IB 9801241 W IB9801241 W IB 9801241W WO 9916251 A1 WO9916251 A1 WO 9916251A1
Authority
WO
WIPO (PCT)
Prior art keywords
motion
image
motion parameter
parameter set
video data
Prior art date
Application number
PCT/IB1998/001241
Other languages
English (en)
Inventor
Gerard De Haan
Robert Jan Schutten
Anna Pelagotti
Original Assignee
Koninklijke Philips Electronics N.V.
Philips Ab
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics N.V., Philips Ab filed Critical Koninklijke Philips Electronics N.V.
Priority to DE69836473T priority Critical patent/DE69836473T2/de
Priority to JP51871999A priority patent/JP2001506472A/ja
Priority to EP98935253A priority patent/EP0940041B1/fr
Priority to KR1019997004575A priority patent/KR100582856B1/ko
Publication of WO1999016251A1 publication Critical patent/WO1999016251A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • H04N19/43—Hardware specially adapted for motion estimation or compensation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/144—Movement detection
    • H04N5/145—Movement estimation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0112—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level one of the standards corresponding to a cinematograph film standard

Definitions

  • the invention relates to a method and a device for motion estimation, and to a video display apparatus comprising a motion-compensated interpolation device.
  • Motion vectors are used in a range of applications, such as coding, noise reduction, and scan rate conversion. Some of these applications, particularly the frame rate conversion, require the true-motion of objects to be estimated [10, 11]. Other applications, e.g. interlaced-to-sequential scan conversion, demand a high accuracy of the motion vectors to achieve a low amplitude of remaining alias [12,13]. Finally, there is a category of applications, e.g. consumer applications of motion estimation, where the cost of the motion estimator is of crucial importance [14,15]. Several algorithms have been proposed to achieve true-motion estimation [3,10, 11, 15-17]. Algorithms have also been proposed to realize motion estimation at a low complexity level, e.g.
  • this task is performed by a so-called block-matcher which divides the image into blocks and calculates a motion vector for every block of pixels by minimizing a match criterion.
  • the risk of such processing is that the motion-compensated image, interpolated from neighboring images and using the motion vectors, may show block distortions if the motion vector field suffers from unwanted inhomogeneities.
  • the IC in [6] applies a block- matcher with improved consistency based on spatial and temporal prediction of candidate vectors [3] .
  • An advantageous side effect of this approach to motion estimation is the very significant reduction of processing power required for the function, which is particularly due to the very limited candidate vector count.
  • a first aspect of the invention provides a method and a device as defined in claims 1 and 9.
  • a second aspect of the invention provides a method and a device as defined in claims 7 and 10.
  • a third aspect of the invention provides a video display apparatus as defined in claim 11. Advantageous embodiments are defined in the dependent claims.
  • At least two motion parameter sets are generated from input video data, a motion parameter set being a set of parameters describing motion in an image, by means of which motion parameter set motion vectors can be calculated.
  • One motion parameter set indicates a zero velocity for all image parts in an image, and each motion parameter set has corresponding local match errors, such as match errors determined per block of pixels.
  • Output motion data are determined from the input video data in dependence on the at least two motion parameter sets, wherein the importance of each motion parameter set (determined by weighting factors W, see equations 17, 18 and between equations 20, 21) in calculating the output motion data depends on the motion parameter sets' local match errors.
  • Local match errors are to be understood in contrast with global match errors, such as match errors calculated for the entire image.
  • at least two motion parameter sets are generated from input video data, one motion parameter set indicating a zero velocity, and each motion parameter set having corresponding match errors, and output video data are interpolated from the input video data in dependence on the at least two motion parameter sets, wherein the importance of each motion parameter set in calculating the output video data depends on the motion parameter sets' match errors.
  • the reduction is so significant that the processing can run on a fully programmable device, more particularly the Philips TriMedia processor.
  • Fig. 1 illustrates two possibilities for motion compensation in accordance with the present invention
  • Fig. 2 shows a first embodiment of a motion-compensated interpolator in accordance with the present invention
  • Fig. 3 shows a second embodiment of a motion-compensated interpolator in accordance with the present invention
  • Fig. 4 shows a block diagram of a motion parameter estimator in accordance with the current invention.
  • Fig. 5 illustrates a preferred parameter estimator in accordance with the present invention.
  • [1,4] described a method for robust motion- compensated temporal interpolation of image data.
  • the basic idea consisted of a three-tap median filter which produces an output pixel selected to be either the motion compensated pixel mcl(eft) from the previous field n-1, the motion- compensated pixel mcr(ight) from the next field n, or the non-motion-compensated average av from both neighboring fields n-1, n:
  • Fig. 1 The pixels used in the motion compensation are schematically drawn in Fig. 1.
  • an even more robust algorithm could be considered for our new very limited motion estimator proposal, including a three-tap median filter which produces an output pixel selecting either the corresponding pixel l(eft) in the previous field n- 1 , the corresponding pixel r(ight) in the next field n, or the motion compensated average mcav from both neighboring fields n-1, n: F ⁇ n-1.) - m. •ncav. r (7)
  • the up-converter is adapted between the first and the second option, depending on the expected quality of the motion vectors.
  • a favorable feature of the proposed interpolators is that switching between the two robust options is not very critical. This implies that a fairly rough decision is acceptable, which can be realized with little processing power on a (spatially) reduced size version of the input sequence. This reduced size input sequence is used to calculate match errors obtained with (at least) two motion vectors per location, either generated from a parameter model or the zero vector.
  • the result is a segmentation which divides the image into layers where the zero vector model or the calculated parameter model is more appropriate.
  • the segmentation mask SM is now used as an extra input of the up-converter UC, which uses the mask SM to switch/ fade between both previously described up-converters (see Fig. 2).
  • the up-converter tends towards the interpolation of equation 2, otherwise towards the interpolation of equation 7.
  • the values 1 and r are applied to a first averager AVI to produce the value av.
  • a first median filter MED1 determines the median of the values av, mcl, and mcr.
  • the values mcl and mcr are applied to a second averager AV2 to produce the value mcav.
  • a second median filter MED2 determines the median of the values mcav, 1, and r.
  • the up-converter UC1 furnishes the interpolated value from the output signals of the median filters MED1, MED2 in dependence upon the segmentation mask SM.
  • the output signal of the up-converter UC1 is applied to a display unit (D) for displaying the output video data (n-Vi) between the input video data (n, n-1).
  • Fig. 3 current image data from the input field n and previous image data from the input field n-1 are applied to the parameter estimators PE2 .. PEn to determine motion parameters p21-p2m .. pnl-pnm.
  • a first parameter estimator PEI just furnishes zero parameters.
  • the input fields n and n-1 are also applied to the segmentation circuit SC via down-samplers Dl, D2.
  • the up-converter UC2 calculates motion vectors in the manner indicated by equation 1 from the parameter set indicated by the segmentation mask SM, to interpolate the output field n-Vi from the input fields n and n-1.
  • the weighting factors W are explained below with reference to equations 17 and 18, and between equations 20 and 21.
  • Each parameter estimator PE2 .. PEn comprises an error calculation to adjust the motion parameters. This calculation is preferably limited to those image parts which correspond to the parameter set delat with by the subject parameter estimator PE. To this end, a weight W is assigned which weight fits with the amount of correspondence (kind of fuzzy logic) . In the end, at the interpolation, for each pixel that parameter set is used that yields the lowest estimation error for each pixel.
  • this error function can be greatly simplified by applying a strong subsampling.
  • a match criterion calculated on just some 300 pixels per field, i.e. a subsampling factor of the order of 1000!
  • the proposal to perform the minimization involves taking a prediction vector (now at least 3-dimensional, in our example 4-dimensional), adding at least one update vector, and selecting the best in accordance with equation 13. Good results could be experimentally obtained when generating a candidate vector set CS p (n) containing three candidate parameter vectors C p ⁇ ri) in accordance with:
  • Penalties can be added to the match error of individual candidate vectors
  • temporal filtering of the parameter vectors is considered to prevent a sudden change of motion vectors from one image to the other.
  • a further alternative includes segmentation, and a fixed number of parameter estimators run in parallel, each focusing on a segment of the image indicated by the segmentation algorithm run on a previous image.
  • Fig. 4 shows a block diagram of a motion parameter estimator according to the current invention.
  • First and second candidate parameter sets Cpl, Cp2 are applied to a multiplexer MUX and to a parameter-to-vector converter PVC to obtain two candidate motion vectors Cvl, Cv2.
  • the first candidate parameter set Cpl is the previous output parameter set P(n) of the multiplexer MUX.
  • the second candidate parameter set Cp2 is obtained by adding (adder AD) an update parameter set Up to the first candidate parameter set Cpl.
  • the update parameter set Up is obtained by applying the result of a mod(n) counter CNT to a look-up table LUT.
  • the candidate motion vectors Cvl, Cv2 are applied to an error calculator EC, to which the present and previous fields n, n-1 are also applied, to obtain two errors El, E2.
  • a minimum circuit MIN determines which error is the smaller, to obtain a selection signal s for the multiplexer MUX in order to obtain the output parameter set P(n) .
  • Motion estimation is applied in coding and scan rate conversion of video data.
  • the picture rate of this video data at the input of the motion estimator is fixed, the picture rate of the video source from which this data originated may differ from that of the processed data. Particularly, this occurs when film material is converted to video, or when video material from one video standard is converted to another standard somewhere in the video chain prior to the motion estimator.
  • a common way of dealing with the required picture rate conversions is to use the most recent picture until a new one becomes available.
  • the resulting video exhibits an irregular motion pattern (judder), which violates the common assumption in motion estimators that motion has a strong temporal consistency.
  • motion estimators that try to profit from this assumption by using temporal prediction vectors, the problem results that the irregular motion behavior eliminates the usefulness of these temporal prediction vectors. A serious degradation of the estimated motion vectors may result.
  • the motion estimator takes temporal prediction vectors from more than one previous image pair (as much as the maximum length of the repetition pattern), and selects the best of these as a basis for the estimation process, or uses all of them as candidates in a matching process.
  • Fig. 5 illustrates a preferred parameter estimator in accordance with the present invention.
  • Current image data from the present field n and previous image data from the previous field n-1 are applied to a motion parameter estimator unit MPE to obtain motion parameters P(n).
  • Picture delays Dl, D2, .. Dn furnish delayed versions TP1 , TP2, .. TPn of the motion parameters P(n) to the motion parameter estimator unit MPE.
  • the following part of this disclosure relates to a layered motion estimation, i.e. the image is segmented into a plurality of layers.
  • Region-based motion estimators have been introduced as an alternative to block-based motion estimators.
  • Block-based motion compensation has been adopted in the international standards for video compression, such as H.261/263 (video-conferencing over ISDN lines), MPEG-1 (multimedia) and MPEG-2 (all-digital TV application). Although these standards do not specify a particular motion estimation method, block-based motion estimation becomes a natural choice. However, the use of blocks as units for motion estimation may result in blocking artifacts, because the boundaries of objects do not generally correspond to block boundaries, and adjacent blocks may be assigned substantially different motion vectors if no spatio-temporal consistency constraint is present.
  • Motion information and pattern information are used in order to achieve a region-based (arbitrarily shaped) motion estimation, the next goal being objects handling and possibly MPEG-4 'Audio- Visual (AV) objects'.
  • the processing starts with an intra-frame segmentation of the image based on pattern information, or on a previously calculated dense motion vector field.
  • the segmentation generally results in a number of small regions. Those regions are then merged, generally using information on their motion, i.e regions with similar motion are merged into one region, and motion parameters are then re-calculated. This procedure proves to be fairly popular when the aim is object-oriented coding. Examples in [26-29]. • Top-down methods
  • the processing starts with an initial image segmentation in large regions. These are subdivided where the calculated motion model lacks accuracy, and motion parameters are re-calculated.
  • the initial segmentation is generally based on a changed/unchanged rule, i.e. the current image is compared with the previous one: when, in the same position, the luminance value in one frame is considerably different from the one in the other one, this pixel is marked as 'changed', or 'unchanged' otherwise.
  • the part of the image classified as 'changed' can be motion-compensated, in accordance with the motion field calculated for that region, and the previously described procedure is iterated in order to identify the different motion regions. Examples in [11,30,31,36].
  • the two techniques can also be combined, e.g. the initial segmentation starts can be random, or based on a previous estimation [11,32], and successive refinements are in both directions.
  • the estimation and the segmentation can also be performed simultaneously, using a statistical approach to the analysis of image sequences, e.g. with a Maximum Likelihood Estimation method [33] .
  • Layered representation methods e.g. the initial segmentation starts can be random, or based on a previous estimation [11,32], and successive refinements are in both directions.
  • the estimation and the segmentation can also be performed simultaneously, using a statistical approach to the analysis of image sequences, e.g. with a Maximum Likelihood Estimation method [33] .
  • the ideal scene segmentation results in separate objects, and involves 3-D information, but this is difficult to obtain and computationally intensive. Therefore, the video data is segmented and described as a set of moving layers, i.e. of image parts, undergoing similar motion, even if disconnected. Then order (depth) of the layers is determined. Examples in [29,34,35]. A model which is less complicated than the full 3-D model and less complicated than a model that deals with all objects in the sequence has been proposed. Since it is the model adopted in a preferred embodiment, it will be described in more details in the following paragraphs.
  • the video data is segmented and described as a set of moving layers, i.e. of regions undergoing similar motion, even if disconnected.
  • the depth order of the layers can then be determined.
  • a layered representation of a video sequence is interesting for several applications such as scan rate conversion, object tracking, video compression, coding, video annotation and indexing.
  • a number of algorithms have already been presented for layered motion estimation [29,34- 37].
  • the sequential approach resolves multiple layers by estimating sequentially a dominant motion, similarly to what is done in the top-down method.
  • the main drawback of such an approach is that, since the final segmentation is not yet known while dealing with one layer, part of the image with a different motion can be included in the estimation of the motion parameters, affecting the results.
  • the simultaneous approach attempts to estimate simultaneously all the layers in the image. This can be done by using a pre-computed dense motion vector field.
  • the initial set of motion models can be derived by using a clustering algorithm on the given motion vector field [29].
  • some smoothness assumptions are generally made. This may lead to a motion vector field in which the boundaries do not correspond to objects/layers boundaries, so that a correct segmentation is not possible.
  • the problem can be formulated as a stochastic problem, and a Maximum- Likelihood Estimation of the multiple models, and their layers of support, can be performed using an Expectation-Maximisation algorithm [36] .
  • the main drawback of the two last- mentioned methods is their complexity.
  • the current aspect of the invention deals with motion estimation / segmentation aiming at a layered representation.
  • the solution consists of a weighting process that limits the pollution of the optimization criterion of a parameter estimator for a given layer by information dealt with by the other parameter estimators running in parallel.
  • Designing a motion estimator to run real-time on existing programmable architectures imposes severe constraints on the problem of motion estimation, since the complexity of the algorithm has to be drastically reduced.
  • a layered motion estimator has been chosen for this purpose, since it is believed that it is potentially easier to implement on a programmable architecture than e.g. a block-based motion estimation, seeing that there are fewer layers than blocks.
  • an image is divided into a number of layers, i.e. parts of the image undergoing a coherent motion, even if disconnected.
  • the apparent motion (optical flow) in a sequence can then be described with parametric models, i.e. it is only due to a combination of camera motions and rigid motion of opaque objects.
  • one set of motion parameters can be estimated for each layer instead of the motion field itself.
  • a new method for a quasi-simultaneous motion estimation and segmentation up to a fixed number of layers is presented.
  • the two goals this hierarchy is meant for are: • To prevent a certain layer from estimatinging on parts of the image that are well covered by layers ranked higher in the hierarchy.
  • the parameter vectors are then estimated in parallel, using a recursive approach, i.e. the earlier estimated parameter vector for each layer is used as a prediction to which update vectors are added.
  • the selected parameter vector is the one resulting in the lowest match error. After this, the parameter vectors of all layers together are used in the segmentation of the image into the desired different layers.
  • each layer 1 is described by a simple motion model. It is assumed that the motion within a layer can be described with a two-dimensional first order linear model.
  • a first layer has 4 or 8 free parameters, while each subsequent layer has less free parameters than the preceding layer to reduce the computational burden.
  • the invention is based on the recognition that the zero vector (no motion) is very common and important in video sequences, and especially important for the intended application in scan rate conversion. Therefore, the proposed algorithm starts with a layer 0, with motion described by the zero parameter vector (which is obviously not estimated).
  • the parameter vectors of additional layers 1, 1 > 0, are estimated separately by their respective parameter estimators PE,.
  • Each PE has the same basic principle as the 3D recursive search block matcher of [3] .
  • a previously estimated parameter vector is updated in accordance with a pseudo- random noise vector, after which the best matching parameter vector is chosen.
  • Wfx is a weighting factor that depends on the position x
  • F s ⁇ x,ri) is the luminance value at position x in the sub-sampled image with index n
  • X is a set of positions x where the motion of layer 1 is to be estimated (the mode of selecting of positions x will be explained below).
  • the images are sub-sampled with a factor of 4 horizontally and 2 vertically on a field base, generating a sub-sampled image F s (n) from each original field F(n). This contributes strongly to the desired reduction of operations count.
  • the sub-sampling is permitted because the objects for which motion is estimated are large enough. In order to achieve pixel or even sub-pixel accuracy on the original pixel grid of F, interpolation is required on the subsampling grid [7] .
  • update parameter UPfri selected from update parameter set :
  • Temporal filtering of the parameter vectors is applied to prevent a sudden change of motion vectors from one image to the other.
  • a hierarchical structure of the layers is proposed. This is achieved by:
  • the selected pixels form groups sparsely distributed over the entire image.
  • a maximum of 50 clusters of 16 pixels is chosen (3 % of all pixels in F s ).
  • a correct selection of X, is necessary to prevent the current estimator from estimating motion that is already covered by previous layers.
  • the location-dependent weighting factor Wfx) is determined by the segmentation mask SM(n-l) found in the previous image. Positions x that belong to the current layer 1 in accordance with the segmentation mask will have a weighting factor greater than one, where positions belonging to a different layer have a weighting factor of one. A correct selection of Wfx) is necessary to prevent the current estimator from estimating motion that can be covered by subsequent layers in the hierarchy.
  • the segmentation step is the most critical step in the algorithm. Its task is to assign one of the layers, i.e. one model of motion, in the image to each group of pixels. This is basically achieved by assigning the best matching model to each group of pixels (a block B , which is typically as large as 8 x 8 pixels on frame base).
  • a match error is calculated in accordance with: e(S,l,n) _ ⁇ _ ⁇ __
  • Segmentation mask SM(B,n) assigns the layer 1 with the lowest e to the block B .
  • the temporal position of the segmentation is defined by , which was set to 1/2 in our experiments.
  • the segmentation mask SM does not have to be calculated for every block B . Instead, the calculated blocks can be sub-sampled in a quincunx pattern, after which the missing positions in the segmentation mask are interpolated (e.g. by choosing the most occurring layer number from a neighborhood) [7] . Segmentation is more difficult as more layers are present, since the segmentation task will resemble more and more that of a full search block matcher. To prevent an output of the motion estimator that has inconsistencies similar to those of a full search block matcher, extra (smoothing) constraints have been added to the algorithm. Current smoothing constraints consist of: • Spatial smoothing: by taking a larger window in the calculation of the e than the size of the block B to which the layer is assigned.
  • Temporal smoothing by reducing the calculated e of a layer with a bonus value if this layer was chosen in the segmentation of the previous image.
  • a simple pre-filtering of the sub-sampling is achieved by averaging pixel values in a block of 4 x 2 pixels. This takes approximately 10 operations per sub-sampled
  • the proposed layered motion estimator was simulated, including usage of the resulting displacement vector for picture rate conversion of 25 Hz film to 50 Hz display .
  • the vector field resulting from the motion estimator proved to be highly consistent and well suited for scan rate conversion.
  • the quality obtained is considered attractive and, for most scenes, comparable with the quality achieved with MELZONIC (SAA4991).
  • the proposed Motion Estimation algorithm has no vector range limitation due to implementation, which is an advantage over MELZONIC (SAA4991).
  • a method of extending global motion estimation algorithms to the estimation of motion parameters in a layered video representation has been presented.
  • a fixed number of parameter estimators is run in parallel, each calculating parameters for one image layer.
  • a segmentation assigns each part of the image to the correct layer.
  • each estimator operates in parallel, some hierarchy exists. Each estimator, apart from the highest in the hierarchy, operates on image parts where higher ranked estimators in the hierarchy were unsuccessful in the previous image. Secondly, each estimator is prevented from pollution by parts of the image that will be better covered by estimators lower in the hierarchy.
  • a first aspect of the invention can be summarized as follows.
  • a new method for global motion-compensated up-conversion is described, and ways are indicated to extend the proposal to application in a layered video representation.
  • parameters describing the global motion are estimated, preferably using a recursive approach.
  • the local motion vectors generated with these parameters are used to generate a motion-compensated image.
  • a segmentation mask is calculated on a reduced size image, the output of which is used to switch between different parameter sets or interpolation methods.
  • An attractive low-cost version is detailed, which is suitable for implementation on currently available fully programmable devices (Natural Motion on a TriMedia).
  • a method, and apparatus realizing this method, for motion compensating video data comprising: at least two means for calculating global motion parameters from the input video data, interpolation means for calculating output video data from one or more input fields, in dependence of the, at least two, sets of global motion parameters, in which one of the at least two means for calculating global motion parameters provides parameters indicating a zero velocity for the entire image, regardless of the image content.
  • the interpolation means is an order statistical filter, e.g. a three-tap median filter, which produces an output pixel from either: the corresponding pixel in the previous field, the corresponding pixel in the next field, and the motion-compensated average from both neighboring fields (first option), or: the motion-compensated pixel from the previous field, the motion- compensated pixel from the next field, and the non-motion-compensated average from both neighboring fields (second option).
  • a segmentation signal activates the first decision, in case the local motion vector calculated from the second parameter set yields the best match on the reduced size input image.
  • the segmentation signal is derived from a reduced size version of the input signal.
  • a method, and apparatus realizing this method, for motion-compensating video data comprising: at least two means for calculating global motion parameters from the input video data, interpolation means for calculating output video data from one or more input fields, in dependence of the, at least two, sets of global motion parameters, and a segmentation signal derived from a reduced size version of the input signal.
  • one of the global motion parameter calculating means provides parameters indicating a zero velocity for the entire image, regardless of the image content.
  • a second aspect of the invention can be summarized as follows.
  • a new method for global motion parameter estimation is described. Essentially, parameters, describing the global motion in the image are estimated using a recursive approach, i.e. an earlier n-dimensional (n is the number of parameters in the motion model) estimate is used as a prediction to which (n-dimensional) update vectors are added.
  • the output parameter vector is the one resulting in the lowest match error.
  • the extremely low complexity of the algorithm, and the high quality make it very attractive for future use in TV and Multi-Media applications, possibly running on fully programmable devices such as TriMedia.
  • a method, and apparatus realizing this method, for estimating motion parameters (the parameter vector) of an image sequence comprising: means for furnishing a prediction parameter vector, i.e. a previously calculated motion parameter estimate, means for selecting at least one update parameter vector from an update set, means for adding said prediction vector to said at least one update vector, means for calculating the quality (cost function) of the resulting, at least two, parameter vectors, using data from at least two fields, means for selecting the best from the aforementioned, at least two, parameter vectors on the basis of their quality, means for outputting the selected parameter vector as the motion parameter estimate.
  • a prediction parameter vector i.e. a previously calculated motion parameter estimate
  • means for selecting at least one update parameter vector from an update set means for adding said prediction vector to said at least one update vector
  • means for calculating the quality (cost function) of the resulting, at least two, parameter vectors using data from at least two fields
  • means for selecting the best from the aforementioned, at least two, parameter vectors on the basis of their quality means for
  • penalties, temporal filtering, and temporal and/or spatial prediction are applied.
  • a third aspect of the present invention can be summarized as follows.
  • a method of estimating motion parameters from video data is disclosed.
  • the invention allows temporal predictive motion estimation on video data that, due to simple picture rate conversion techniques (repetition of the most recent picture), exhibits an irregular motion.
  • the solution consists of using multiple temporal prediction vectors taken from various previous image pairs. This solution is economically justifiable, particularly in object-based motion estimators, where the number of motion vectors to be stored is very small.
  • a software version of the algorithm has been shown to run real-time on the Philips TM1000 (TriMedia) processor.
  • a method, and apparatus realizing this method, of estimating motion parameter vectors from video data, which furnishes, for at least some image-parts, at least two (temporal) prediction vectors estimated from data of different previous image pairs.
  • the above- mentioned at least two prediction vectors are candidates in a vector selection process determining the output vector for an image (part).
  • the best of the above-mentioned at least two prediction vectors is used as a basis for calculating candidate vectors (e.g. updating process) that are input of a vector selection process determining the output vector for an image (part).
  • the decision information (which of the at least two prediction vectors is best, in accordance with a criterion function) over a number of successive images (image parts), is used to detect picture repetition patterns (e.g. 3-2 pull-down and 2-2 pull-down of movie material, but also other patterns due to source-destination picture frequency mismatches).
  • picture repetition patterns e.g. 3-2 pull-down and 2-2 pull-down of movie material, but also other patterns due to source-destination picture frequency mismatches.
  • a fourth aspect of the invention relates to a joint motion estimation and segmentation of video data, and can be summarized as follows.
  • a method of segmenting an image into a fixed number of layers and estimate motion parameters for individual layers is disclosed.
  • the invention provides a solution to the chicken and egg problem of combined motion estimation and segmentation.
  • the solution consists of a weighting process that limits the pollution of the optimization criterion of a parameter estimator for a given layer by information dealt with by the other parameter estimators running in parallel.
  • the extremely low complexity of the algorithm, and the high quality make it very attractive for future use in TV and Multi-Media applications.
  • a software version of the algorithm has been shown to run real-time on the Philips TM1000 (TriMedia) processor.
  • a layered motion estimation algorithm that permits quasi-simultaneous motion estimation / segmentation up to a fixed maximum number of layers.
  • the estimation results in one motion parameter set per layer, and a segmentation map that assigns these sets to different parts of the image (motion layers) .
  • Motion in a layer is modelled with a maximum of four parameters capable of describing pan, tilt and zoom.
  • the concept shows some hierarchy, i.e. a ranking of the motion layers.
  • the motion parameter estimation concerning one layer excludes parts of the image that have been described by a layer ranked higher in the hierarchy and is not polluted by parts of the image that are better described by layers ranked lower in the hierarchy.
  • the concept results in a very low operations count. It has been shown to perform well, even in critical scan rate conversion applications, particularly in picture rate up-conversion. A variant including three layers has been scheduled to run in real-time on a Philips TriMedia processor.
  • a method, and apparatus realizing this method, for segmenting an image into a ranked set of layers and estimating motion parameters for every layer, comprising: a parameter estimation (PE) process for every layer in the current image optimizing a criterion function based upon (groups of) pixels from at least two pictures, a segmentation process (SP) assigning motion parameter sets to image parts, a weighting process (WP) to define the individual effect of information from different image parts on the criterion function of a motion parameter estimator in which the WP reduces or eliminates the effect of information from those image parts that fulfil a first criterion, and increases the effect of information from those image parts that fulfil a second criterion.
  • PE parameter estimation
  • SP segmentation process
  • WP weighting process
  • the first criterion is met if, in a previous iteration of the algorithm on the same or another picture pair, the image parts fell in regions which were adequately described by any of the motion parameter sets estimated by PEs active on layers with a higher rank number. "Adequately" means that an error function, using the parameter sets of the PEs active on layers higher in the hierarchy, stays below a threshold (either fixed, or adapted, e.g. to the average error).
  • the second criterion is met if, in a previous iteration of the algorithm on the same or another picture pair, the image parts fell in regions which were best described by the motion parameter sets estimated by this given PE. "Best" means that an error function, using the parameter sets of the given PE, is lower than that of any of the other PEs.
  • this error function is based upon the motion-compensated difference between the pixels in the current field and the corresponding pixels in the previous field, using the parameter sets to be evaluated (direct method).
  • this error function is based upon the difference between motion vectors calculated with some method, and motion vectors resulting from the motion parameter set to be evaluated (indirect method).
  • picture parts meeting the first criterion are eliminated in the error function of a given PE, and this first criterion is adapted in such a way that the picture area on which the criterion function is calculated remains within a given range (control loop to efficiently use maximum available processing power).
  • the PE and/or the SP, and/or the WP operates on down-scaled and/or subsampled video data.
  • a method, and apparatus realizing this method for segmenting an image into a ranked set of layers and estimating motion parameters for each layer, comprising: an iterative parameter estimation process for every layer in the current image optimizing a criterion function based upon selected (groups of) pixels from at least two pictures, a segmentation process assigning to every part of the image one of the motion parameter sets, a selection process to define upon which (groups of) pixels from the at least two pictures the motion parameter estimator(s) should optimize their criterion function, in which the parameter estimation process iterates on its data more often than the other processes.
  • the selection process selects for a given layer those (groups of) pixels, for which the parameter sets of layers higher in the hierarchy, in a previous picture did not give satisfactory results according to a rule.
  • this rule involves the comparison of an error(sum) of (groups of) pixels with a fixed or adaptive threshold.
  • this threshold is adapted in such a way that the number of pixels on which the criterion function is calculated remains within a given range.
  • the criterion function is a summed error calculated between selected (groups of) pixels from the previous picture and corresponding pixels from the current picture compensated for motion in accordance with the candidate motion parameters.
  • the contribution of the selected pixels to the criterion function is weighted, depending on which layer they were assigned to (in the previous picture).
  • the contribution of the selected pixels to the criterion function is increased if they were assigned to the same layer previously.
  • a method, and apparatus realizing this method, for segmenting an image into a ranked set of layers and estimating motion parameters for every layer comprising: a parameter estimation (PE) process for every layer in the current image optimizing a criterion function based upon (groups of) pixels from at least two pictures a segmentation process (SP) assigning to every part of the image one of the motion parameter sets, a selection process to define upon which (groups of) pixels from the at least two pictures the motion parameter estimator(s) should optimize their criterion function, in which the selection process allows a small fraction of the pixels only to contribute to the criterion function optimized by the PEs, regardless of the size of the layer to which these parameters are assigned by the segmentation process.
  • PE parameter estimation
  • SP segmentation process
  • TriMedia processor more applications are possible. Particularly, the concept can be designed into next generation VGA-controllers. Since this is dedicated silicon, the total cost is negligible. Such a VGA-controller may have an improved performance as compared with the TriMedia solution, because much more processing power is available in dedicated silicon. Furthermore, it is expected that, if more than two parallel parameter estimators are applied, the performance can be brought to a level which is potentially better than that of the current high-end solutions at a possibly lower cost.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Television Systems (AREA)
  • Image Analysis (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Picture Signal Circuits (AREA)

Abstract

L'invention porte sur un procédé d'estimation du mouvement selon lequel: deux ensembles de paramètres de mouvement au moins (PE1-PEn) sont produits à partir de données vidéo d'entrée (n, n-1), un ensemble de paramètres de mouvement se composant de plusieurs paramètres de mouvement décrivant le mouvement dans une image et au moyen duquel on peut calculer les vecteurs de mouvement des ensembles de paramètres de mouvement. Un ensemble de paramètres de mouvement indique une vitesse nulle pour toute les parties d'une image et chacun d'eux présente des erreurs locales d'adaptation. Les données de mouvement de sortie sont déterminés à partir de données d'entrée vidéo (n, n-1) en fonction d'au moins deux ensembles de paramètres. L'importance de chacun des ensembles de paramètres de mouvement lors du calcul des données du mouvement de sortie dépend des erreurs locales d'adaptation des ensembles de paramètres de mouvement.
PCT/IB1998/001241 1997-09-23 1998-08-12 Estimation de mouvements et interpolation compensee en mouvement WO1999016251A1 (fr)

Priority Applications (4)

Application Number Priority Date Filing Date Title
DE69836473T DE69836473T2 (de) 1997-09-23 1998-08-12 Bewegungsschätzung und bewegungskompensierte interpolation
JP51871999A JP2001506472A (ja) 1997-09-23 1998-08-12 動作評価及び動作補償された補間
EP98935253A EP0940041B1 (fr) 1997-09-23 1998-08-12 Estimation de mouvements et interpolation compensee en mouvement
KR1019997004575A KR100582856B1 (ko) 1997-09-23 1998-08-12 움직임 추정 및 움직임 보상된 보간법

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP97202912 1997-09-23
EP97202912.8 1997-11-20

Publications (1)

Publication Number Publication Date
WO1999016251A1 true WO1999016251A1 (fr) 1999-04-01

Family

ID=8228750

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB1998/001241 WO1999016251A1 (fr) 1997-09-23 1998-08-12 Estimation de mouvements et interpolation compensee en mouvement

Country Status (6)

Country Link
US (1) US6385245B1 (fr)
EP (1) EP0940041B1 (fr)
JP (1) JP2001506472A (fr)
KR (1) KR100582856B1 (fr)
DE (1) DE69836473T2 (fr)
WO (1) WO1999016251A1 (fr)

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001045420A1 (fr) * 1999-12-17 2001-06-21 Koninklijke Philips Electronics N.V. Estimation de transmission d'images video
WO2002056597A1 (fr) * 2001-01-11 2002-07-18 Koninklijke Philips Electronics N.V. Reconnaissance d'objets de film ou de video intervenant en parallele dans des champs de signaux de television uniques
WO2002067576A1 (fr) * 2001-02-21 2002-08-29 Koninklijke Philips Electronics N.V. Dispositif et procede facilitant l"estimation de mouvement
EP1361541A1 (fr) * 2002-04-09 2003-11-12 STMicroelectronics S.r.l. Procédé et processus d'éstimation de mouvement globale dans une série d'images, par exemple pour souris optiques
EP1096791A3 (fr) * 1999-10-28 2004-03-24 Sony Corporation Traitement d'image et traitement de données
DE10318620A1 (de) * 2003-04-24 2004-11-25 Micronas Gmbh Verfahren und Vorrichtung zur Ermittlung eines Abstandes zwischen einer ersten und zweiten Signalfolge
EP1592247A1 (fr) * 2004-04-30 2005-11-02 Matsushita Electric Industrial Co., Ltd. Compensation de mouvement en fonction du type de block
EP1592246A1 (fr) * 2004-04-30 2005-11-02 Matsushita Electric Industrial Co., Ltd. Détermination de mode film dans des zones d'image vidéo immobiles
EP1592250A1 (fr) * 2004-04-30 2005-11-02 Matsushita Electric Industrial Co., Ltd. Detection de mode cinéma dans des séquences vidéo
EP1592255A1 (fr) * 2004-04-30 2005-11-02 Matsushita Electric Industrial Co., Ltd. Estimation de vecteur de mouvement avec une sélection ameliorée de vecteur de mouvement
CN100463495C (zh) * 1999-06-01 2009-02-18 索尼公司 噪声量估算装置和方法
KR101204377B1 (ko) * 2004-12-09 2012-11-26 톰슨 라이센싱 움직임 보상된 화상을 생성하기 위한 방법 및 장치
EP1393572B1 (fr) * 2001-04-20 2018-06-13 Nxp B.V. Procede et appareil d'estimation de vecteurs de mouvement

Families Citing this family (109)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100308016B1 (ko) 1998-08-31 2001-10-19 구자홍 압축 부호화된 영상에 나타나는 블럭현상 및 링현상 제거방법및 영상 복호화기
WO2000019713A1 (fr) * 1998-09-30 2000-04-06 Micronas Munich Gmbh Circuit permettant de transformer un debit binaire
US6535643B1 (en) * 1998-11-03 2003-03-18 Lg Electronics Inc. Method for recovering compressed motion picture for eliminating blocking artifacts and ring effects and apparatus therefor
EP1075147A1 (fr) * 1999-08-02 2001-02-07 Koninklijke Philips Electronics N.V. Estimation de mouvement
JP3753578B2 (ja) * 1999-12-07 2006-03-08 Ｎｅｃエレクトロニクス株式会社 動きベクトル探索装置および方法
FI116819B (fi) * 2000-01-21 2006-02-28 Nokia Corp Menetelmä kuvien lähettämiseksi ja kuvakooderi
US6714258B2 (en) * 2000-03-15 2004-03-30 Koninklijke Philips Electronics N.V. Video-apparatus with noise reduction
JP4409035B2 (ja) * 2000-03-22 2010-02-03 本田技研工業株式会社 画像処理装置、特異箇所検出方法、及び特異箇所検出プログラムを記録した記録媒体
FR2809267B1 (fr) * 2000-05-19 2002-09-27 Thomson Multimedia Sa Procede de detection de saturation d'un champ de vecteurs mouvement
US6532380B1 (en) * 2000-06-30 2003-03-11 Cedars Sinai Medical Center Image guidance for coronary stent deployment
US6760376B1 (en) * 2000-11-06 2004-07-06 Koninklijke Philips Electronics N.V. Motion compensated upconversion for video scan rate conversion
US7850304B2 (en) 2001-01-23 2010-12-14 Kenneth Martin Jacobs Continuous adjustable 3Deeps filter spectacles for optimized 3Deeps stereoscopic viewing and its control method and means
US7405801B2 (en) * 2001-01-23 2008-07-29 Kenneth Jacobs System and method for Pulfrich Filter Spectacles
US9781408B1 (en) 2001-01-23 2017-10-03 Visual Effect Innovations, Llc Faster state transitioning for continuous adjustable 3Deeps filter spectacles using multi-layered variable tint materials
US8750382B2 (en) 2001-01-23 2014-06-10 Kenneth Martin Jacobs System and method for calculating 3Deeps action specs motion estimation from the motion vectors in an MPEG file
US7604348B2 (en) * 2001-01-23 2009-10-20 Kenneth Martin Jacobs Continuous adjustable 3deeps filter spectacles for optimized 3deeps stereoscopic viewing and its control method and means
US7522257B2 (en) * 2001-01-23 2009-04-21 Kenneth Jacobs System and method for a 3-D phenomenoscope
US7508485B2 (en) * 2001-01-23 2009-03-24 Kenneth Martin Jacobs System and method for controlling 3D viewing spectacles
US10742965B2 (en) * 2001-01-23 2020-08-11 Visual Effect Innovations, Llc Faster state transitioning for continuous adjustable 3Deeps filter spectacles using multi-layered variable tint materials
WO2003079746A2 (fr) * 2001-02-05 2003-10-02 Bristol-Myers Squibb Company Recepteur humain couple aux proteines g, hgprbmy14, associe au recepteur gpcr orphelin gpr73
US7042945B2 (en) * 2001-04-24 2006-05-09 Bellers Erwin B 3-D recursive vector estimation for video enhancement
US6738099B2 (en) * 2001-02-16 2004-05-18 Tektronix, Inc. Robust camera motion estimation for video sequences
US6704055B1 (en) 2001-03-30 2004-03-09 Pixelworks, Inc. 3:2 pull-down film mode detection using fuzzy logic
US20020196854A1 (en) * 2001-06-15 2002-12-26 Jongil Kim Fast video encoder using adaptive hierarchical video processing in a down-sampled domain
KR100525785B1 (ko) * 2001-06-15 2005-11-03 엘지전자 주식회사 이미지 화소 필터링 방법
US7110525B1 (en) 2001-06-25 2006-09-19 Toby Heller Agent training sensitive call routing system
KR100396558B1 (ko) * 2001-10-25 2003-09-02 삼성전자주식회사 적응 움직임 보상형 프레임 및/또는 레이트 변환 장치 및그 방법
US7085431B2 (en) * 2001-11-13 2006-08-01 Mitutoyo Corporation Systems and methods for reducing position errors in image correlation systems during intra-reference-image displacements
KR100955414B1 (ko) * 2002-01-17 2010-05-04 엔엑스피 비 브이 현재 움직임 벡터 추정용 유닛 및 그 추정 방법
US7092924B1 (en) * 2002-02-28 2006-08-15 Raytheon Company Method and system for assigning observations
JP4724351B2 (ja) * 2002-07-15 2011-07-13 三菱電機株式会社 画像符号化装置、画像符号化方法、画像復号装置、画像復号方法、および通信装置
DE10236207A1 (de) * 2002-08-07 2004-02-26 Micronas Gmbh Verfahren zur bewegungsvektorgestützten Interpolation eines Bildpunktes eines Zwischenbildes einer Bildfolge
US7197075B2 (en) * 2002-08-22 2007-03-27 Hiroshi Akimoto Method and system for video sequence real-time motion compensated temporal upsampling
EP1406447A1 (fr) * 2002-10-04 2004-04-07 STMicroelectronics S.r.l. Méthode et Système pour le traitement de signaux via une quantification vectorielle perceptive, et programme d'ordinateur
US8023563B2 (en) * 2002-10-04 2011-09-20 Stmicroelectronics S.R.L. Method and system for processing signals via perceptive vectorial quantization, computer program product therefor
KR100486284B1 (ko) * 2002-11-22 2005-04-29 삼성전자주식회사 연속되는 두 개의 디인터레이스 프레임들을 출력할 수있는 디인터레이스 장치 및 디인터레이스 방법
US9818136B1 (en) 2003-02-05 2017-11-14 Steven M. Hoffberg System and method for determining contingent relevance
US7197074B2 (en) * 2003-02-20 2007-03-27 The Regents Of The University Of California Phase plane correlation motion vector determination method
DE10327576A1 (de) * 2003-06-18 2005-01-13 Micronas Gmbh Verfahren und Vorrichtung zur bewegungsvektorgestützten Bildpunktinterpolation
DE10327577A1 (de) * 2003-06-18 2005-01-13 Micronas Gmbh Verfahren zur Ermittlung eines Verschiebungsvektors in der Bildverarbeitung
US7596284B2 (en) * 2003-07-16 2009-09-29 Hewlett-Packard Development Company, L.P. High resolution image reconstruction
US20050013496A1 (en) * 2003-07-16 2005-01-20 Bruls Wilhelmus Hendrikus Alfonsus Video decoder locally uses motion-compensated interpolation to reconstruct macro-block skipped by encoder
US7620254B2 (en) * 2003-08-07 2009-11-17 Trident Microsystems (Far East) Ltd. Apparatus and method for motion-vector-aided interpolation of a pixel of an intermediate image of an image sequence
US20050105621A1 (en) * 2003-11-04 2005-05-19 Ju Chi-Cheng Apparatus capable of performing both block-matching motion compensation and global motion compensation and method thereof
KR20050045746A (ko) * 2003-11-12 2005-05-17 삼성전자주식회사 계층 구조의 가변 블록 크기를 이용한 움직임 추정 방법및 장치
US7830959B2 (en) * 2003-12-26 2010-11-09 Electronics And Telecommunications Research Institute Apparatus and method for performing intra prediction for image decoder
JP4198608B2 (ja) * 2004-01-15 2008-12-17 株式会社東芝 補間画像生成方法および装置
KR100995398B1 (ko) * 2004-01-20 2010-11-19 삼성전자주식회사 수직 및 수평 방향의 패턴을 고려한 전역 움직임 보상순차주사화 방법
US7433497B2 (en) * 2004-01-23 2008-10-07 Hewlett-Packard Development Company, L.P. Stabilizing a sequence of image frames
KR20050081730A (ko) * 2004-02-16 2005-08-19 엘지전자 주식회사 움직임 보상 기반의 영상 신호 프레임율 변환 방법
JP4375080B2 (ja) * 2004-03-29 2009-12-02 ソニー株式会社 画像処理装置および方法、記録媒体、並びにプログラム
US8036494B2 (en) * 2004-04-15 2011-10-11 Hewlett-Packard Development Company, L.P. Enhancing image resolution
EP1592248B1 (fr) * 2004-04-30 2006-09-20 Matsushita Electric Industrial Co., Ltd. Estimation de vecteur de mouvement utilisant prediction temporelle
DE602004030993D1 (de) 2004-04-30 2011-02-24 Panasonic Corp Bewegungsschätzung unter Verwendung von adaptiven räumlichen Verfeinerungsvektoren
US7881500B2 (en) * 2004-06-30 2011-02-01 Trident Microsystems (Far East) Ltd. Motion estimation with video mode detection
US7684628B2 (en) * 2004-08-12 2010-03-23 Industrial Technology Research Institute System and method for non-iterative global motion estimation
US7730406B2 (en) * 2004-10-20 2010-06-01 Hewlett-Packard Development Company, L.P. Image processing system and method
JP2006165935A (ja) * 2004-12-07 2006-06-22 Nec Corp 制御情報の変換装置及び制御情報の変換方法
US9225994B2 (en) * 2005-03-14 2015-12-29 British Telecommunications Public Limited Company Global motion estimation using reduced frame lines
EP1703736A1 (fr) * 2005-03-14 2006-09-20 BRITISH TELECOMMUNICATIONS public limited company Estimation de mouvement globale
US7522749B2 (en) * 2005-04-08 2009-04-21 Microsoft Corporation Simultaneous optical flow estimation and image segmentation
US7760956B2 (en) 2005-05-12 2010-07-20 Hewlett-Packard Development Company, L.P. System and method for producing a page using frames of a video stream
KR100676704B1 (ko) * 2005-07-07 2007-02-01 삼성전자주식회사 움직임 추정장치 및 그 추정방법
US7668366B2 (en) * 2005-08-09 2010-02-23 Seiko Epson Corporation Mosaic image data processing
US20070074251A1 (en) * 2005-09-27 2007-03-29 Oguz Seyfullah H Method and apparatus for using random field models to improve picture and video compression and frame rate up conversion
US20070076796A1 (en) * 2005-09-27 2007-04-05 Fang Shi Frame interpolation using more accurate motion information
US9258519B2 (en) * 2005-09-27 2016-02-09 Qualcomm Incorporated Encoder assisted frame rate up conversion using various motion models
EP1841231A1 (fr) * 2006-03-29 2007-10-03 Sony Deutschland Gmbh Procédé d'estimation de mouvement
JP2008035436A (ja) * 2006-07-31 2008-02-14 Fujitsu Ltd プルダウンシーケンス検出装置およびプルダウンシーケンス検出方法
US7809212B2 (en) * 2006-12-20 2010-10-05 Hantro Products Oy Digital mosaic image construction
TWI327866B (en) * 2006-12-27 2010-07-21 Realtek Semiconductor Corp Apparatus and related method for decoding video blocks in video pictures
JP4821642B2 (ja) * 2007-02-15 2011-11-24 株式会社ニコン 画像処理方法、画像処理装置、ディジタルカメラ及び画像処理プログラム
US20080212687A1 (en) * 2007-03-02 2008-09-04 Sony Corporation And Sony Electronics Inc. High accurate subspace extension of phase correlation for global motion estimation
HUE037450T2 (hu) * 2007-09-28 2018-09-28 Dolby Laboratories Licensing Corp Videó információ kezelése
US20090161978A1 (en) * 2007-12-20 2009-06-25 Marinko Karanovic Halo Artifact Removal Method
US8559518B2 (en) * 2008-06-30 2013-10-15 Samsung Electronics Co., Ltd. System and method for motion estimation of digital video using multiple recursion rules
JP4687772B2 (ja) * 2008-11-06 2011-05-25 ソニー株式会社 画像処理装置、画像処理方法、プログラム
WO2010093430A1 (fr) * 2009-02-11 2010-08-19 Packetvideo Corp. Système et procédé d'interpolation de trames pour un flux binaire vidéo compressé
FI20095273A0 (fi) * 2009-03-17 2009-03-17 On2 Technologies Finland Oy Digitaalinen videokoodaus
KR101710883B1 (ko) 2009-11-04 2017-02-28 삼성전자주식회사 필터 정보를 이용한 영상 압축 장치 및 방법과 영상 복원 장치 및 방법
US8233085B1 (en) 2009-12-08 2012-07-31 Kolorific, Inc. Method and system for interpolating a pixel value of a pixel located at an on-screen display
US8488007B2 (en) * 2010-01-19 2013-07-16 Sony Corporation Method to estimate segmented motion
JP5583992B2 (ja) * 2010-03-09 2014-09-03 パナソニック株式会社 信号処理装置
US8285079B2 (en) * 2010-03-19 2012-10-09 Sony Corporation Method for highly accurate estimation of motion using phase correlation
EP2405661B1 (fr) 2010-07-06 2017-03-22 Google, Inc. Transmission vidéo résistante à la perte utilisant deux décodeurs
GB201113527D0 (en) * 2011-08-04 2011-09-21 Imagination Tech Ltd External vectors in a motion estimation system
US9014265B1 (en) 2011-12-29 2015-04-21 Google Inc. Video coding using edge detection and block partitioning for intra prediction
US9288484B1 (en) 2012-08-30 2016-03-15 Google Inc. Sparse coding dictionary priming
US9210424B1 (en) 2013-02-28 2015-12-08 Google Inc. Adaptive prediction block size in video coding
US9300906B2 (en) * 2013-03-29 2016-03-29 Google Inc. Pull frame interpolation
US9313493B1 (en) 2013-06-27 2016-04-12 Google Inc. Advanced motion estimation
US9286653B2 (en) 2014-08-06 2016-03-15 Google Inc. System and method for increasing the bit depth of images
US9153017B1 (en) 2014-08-15 2015-10-06 Google Inc. System and method for optimized chroma subsampling
US10726593B2 (en) * 2015-09-22 2020-07-28 Fyusion, Inc. Artificially rendering images using viewpoint interpolation and extrapolation
US11095869B2 (en) 2015-09-22 2021-08-17 Fyusion, Inc. System and method for generating combined embedded multi-view interactive digital media representations
US10242474B2 (en) 2015-07-15 2019-03-26 Fyusion, Inc. Artificially rendering images using viewpoint interpolation and extrapolation
US11006095B2 (en) 2015-07-15 2021-05-11 Fyusion, Inc. Drone based capture of a multi-view interactive digital media
US10222932B2 (en) 2015-07-15 2019-03-05 Fyusion, Inc. Virtual reality environment based manipulation of multilayered multi-view interactive digital media representations
US10147211B2 (en) 2015-07-15 2018-12-04 Fyusion, Inc. Artificially rendering images using viewpoint interpolation and extrapolation
US9807416B2 (en) 2015-09-21 2017-10-31 Google Inc. Low-latency two-pass video coding
US11783864B2 (en) 2015-09-22 2023-10-10 Fyusion, Inc. Integration of audio into a multi-view interactive digital media representation
US11202017B2 (en) 2016-10-06 2021-12-14 Fyusion, Inc. Live style transfer on a mobile device
US10437879B2 (en) 2017-01-18 2019-10-08 Fyusion, Inc. Visual search using multi-view interactive digital media representations
US10595039B2 (en) 2017-03-31 2020-03-17 Nvidia Corporation System and method for content and motion controlled action video generation
US10313651B2 (en) 2017-05-22 2019-06-04 Fyusion, Inc. Snapshots at predefined intervals or angles
US11069147B2 (en) 2017-06-26 2021-07-20 Fyusion, Inc. Modification of multi-view interactive digital media representation
AT520839B1 (de) * 2018-01-25 2019-08-15 Ait Austrian Inst Tech Gmbh Verfahren zur Erstellung einer Bildstapel-Datenstruktur
US10592747B2 (en) 2018-04-26 2020-03-17 Fyusion, Inc. Method and apparatus for 3-D auto tagging
CN116848843A (zh) * 2021-02-08 2023-10-03 华为技术有限公司 可切换的密集运动向量场插值

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1997046022A2 (fr) * 1996-05-24 1997-12-04 Philips Electronics N.V. Estimation de mouvement

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0250533B1 (fr) * 1985-12-24 1993-01-27 British Broadcasting Corporation Procede de codage d'un signal video a des fins de transmission dans une largeur de bande limitee
EP0294960B1 (fr) * 1987-06-09 1994-09-28 Sony Corporation Traitement du vecteur de mouvement dans des images de télévision
US5093720A (en) * 1990-08-20 1992-03-03 General Instrument Corporation Motion compensation for interlaced digital television signals
US5371539A (en) * 1991-10-18 1994-12-06 Sanyo Electric Co., Ltd. Video camera with electronic picture stabilizer
KR0180170B1 (ko) * 1995-06-30 1999-05-01 배순훈 움직임 추정 방법 및 추정 장치
GB9519923D0 (en) * 1995-09-29 1995-11-29 Philips Electronics Nv Motion estimation for predictive image coding
US5825930A (en) * 1997-03-05 1998-10-20 Samsung Electronics Co., Ltd. Motion estimating method
DE69727911D1 (de) * 1997-04-24 2004-04-08 St Microelectronics Srl Verfahren zur Erhöhung der bewegungsgeschätzten und bewegungskompensierten Bildfrequenz für Videoanwendungen, und Vorrichtung zur Verwendung eines solchen Verfahrens
JPH10327415A (ja) * 1997-05-22 1998-12-08 Mitsubishi Electric Corp 動きベクトル検出装置
KR100235356B1 (ko) * 1997-08-13 1999-12-15 전주범 개선된 움직임 추정 장치 및 그 추정 방법

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1997046022A2 (fr) * 1996-05-24 1997-12-04 Philips Electronics N.V. Estimation de mouvement

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
IEEE COMPUTER SOCIETY CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, PROCEEDINGS, June 1996, YAIR WEISS et al., "A Unified Mixture Framework for Motion Segmentation. Incorporating Spatial Coherence and Estimating the Number of Models", ISBN 0-8186-7258-7, pages 321-326. *
PROCEEDINGS OF THE SPIE: IMAGE AND VIDEO PROCESSING II, Vol. 2182, San Jose, February 1994, JOHN Y.A. WANG et al., "Spatio-Temporal Segmentation of Video Data", pages 120-131. *

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100463495C (zh) * 1999-06-01 2009-02-18 索尼公司 噪声量估算装置和方法
EP1096791A3 (fr) * 1999-10-28 2004-03-24 Sony Corporation Traitement d'image et traitement de données
WO2001045422A1 (fr) * 1999-12-17 2001-06-21 Koninklijke Philips Electronics N.V. Estimation de mouvement dans des images video
WO2001045421A1 (fr) * 1999-12-17 2001-06-21 Koninklijke Philips Electronics N.V. Estimation de transmission d'images video
WO2001045420A1 (fr) * 1999-12-17 2001-06-21 Koninklijke Philips Electronics N.V. Estimation de transmission d'images video
WO2002056597A1 (fr) * 2001-01-11 2002-07-18 Koninklijke Philips Electronics N.V. Reconnaissance d'objets de film ou de video intervenant en parallele dans des champs de signaux de television uniques
US6937655B2 (en) 2001-01-11 2005-08-30 Koninklijke Philips Electronics N.V. Recognizing film and video objects occuring in parallel in single television signal fields
WO2002067576A1 (fr) * 2001-02-21 2002-08-29 Koninklijke Philips Electronics N.V. Dispositif et procede facilitant l"estimation de mouvement
EP1393572B1 (fr) * 2001-04-20 2018-06-13 Nxp B.V. Procede et appareil d'estimation de vecteurs de mouvement
EP1361541A1 (fr) * 2002-04-09 2003-11-12 STMicroelectronics S.r.l. Procédé et processus d'éstimation de mouvement globale dans une série d'images, par exemple pour souris optiques
US7099512B2 (en) 2002-04-09 2006-08-29 Stmicroelectronics S.R.L. Process and device for global motion estimation in a sequence of images and a computer program product therefor
DE10318620A1 (de) * 2003-04-24 2004-11-25 Micronas Gmbh Verfahren und Vorrichtung zur Ermittlung eines Abstandes zwischen einer ersten und zweiten Signalfolge
US7145604B2 (en) 2003-04-24 2006-12-05 Micronas Gmbh Method and device for determining the spacing between a first and a second signal sequence
EP1592246A1 (fr) * 2004-04-30 2005-11-02 Matsushita Electric Industrial Co., Ltd. Détermination de mode film dans des zones d'image vidéo immobiles
EP1592255A1 (fr) * 2004-04-30 2005-11-02 Matsushita Electric Industrial Co., Ltd. Estimation de vecteur de mouvement avec une sélection ameliorée de vecteur de mouvement
CN100417188C (zh) * 2004-04-30 2008-09-03 松下电器产业株式会社 块模式自适应运动补偿
US7440032B2 (en) 2004-04-30 2008-10-21 Matsushita Electric Industrial Co., Ltd. Block mode adaptive motion compensation
CN100433791C (zh) * 2004-04-30 2008-11-12 松下电器产业株式会社 在静止区域的影片模式纠正
EP1592250A1 (fr) * 2004-04-30 2005-11-02 Matsushita Electric Industrial Co., Ltd. Detection de mode cinéma dans des séquences vidéo
US7693218B2 (en) 2004-04-30 2010-04-06 Panasonic Corporation Motion vector estimation with improved motion vector selection
EP1592247A1 (fr) * 2004-04-30 2005-11-02 Matsushita Electric Industrial Co., Ltd. Compensation de mouvement en fonction du type de block
KR101204377B1 (ko) * 2004-12-09 2012-11-26 톰슨 라이센싱 움직임 보상된 화상을 생성하기 위한 방법 및 장치

Also Published As

Publication number Publication date
US6385245B1 (en) 2002-05-07
DE69836473D1 (de) 2007-01-04
EP0940041B1 (fr) 2006-11-22
KR20000069107A (ko) 2000-11-25
KR100582856B1 (ko) 2006-05-24
JP2001506472A (ja) 2001-05-15
DE69836473T2 (de) 2007-09-20
EP0940041A1 (fr) 1999-09-08

Similar Documents

Publication Publication Date Title
US6385245B1 (en) Motion estimation and motion-compensated interpolition
Choi et al. Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation
Chalidabhongse et al. Fast motion vector estimation using multiresolution-spatio-temporal correlations
Nam et al. A fast hierarchical motion vector estimation algorithm using mean pyramid
Choi et al. New frame rate up-conversion using bi-directional motion estimation
US5929940A (en) Method and device for estimating motion between images, system for encoding segmented images
Jeon et al. Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure
Hilman et al. Using motion-compensated frame-rate conversion for the correction of 3: 2 pulldown artifacts in video sequences
Schutten et al. Real-time 2-3 pull-down elimination applying motion estimation/compensation in a programmable device
US6278736B1 (en) Motion estimation
Xu et al. Adaptive motion tracking block matching algorithms for video coding
Po et al. A novel four-step search algorithm for fast block motion estimation
US6269174B1 (en) Apparatus and method for fast motion estimation
De Haan IC for motion-compensated de-interlacing, noise reduction, and picture-rate conversion
JP2002511687A (ja) 動き推定システムおよび方法
Choi et al. Frame rate up-conversion using perspective transform
De Haan Progress in motion estimation for consumer video format conversion
Baek et al. An efficient block-matching criterion for motion estimation and its VLSI implementation
Wang et al. Evaluation of mesh-based motion estimation in H. 263-like coders
Han et al. Frame rate up-conversion for high-definition video applications
Ju et al. A fast rate-optimized motion estimation algorithm for low-bit-rate video coding
Tang et al. Comparison between block-based and pixel-based temporal interpolation for video coding
Schutten et al. Layered motion estimation
Cetin et al. An adaptive true motion estimation algorithm for frame rate conversion of high definition video
Hang et al. Motion estimation for image sequence compression

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): JP KR

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE

WWE Wipo information: entry into national phase

Ref document number: 1998935253

Country of ref document: EP

ENP Entry into the national phase

Ref country code: JP

Ref document number: 1999 518719

Kind code of ref document: A

Format of ref document f/p: F

WWE Wipo information: entry into national phase

Ref document number: 1019997004575

Country of ref document: KR

121 Ep: the epo has been informed by wipo that ep was designated in this application
WWP Wipo information: published in national office

Ref document number: 1998935253

Country of ref document: EP

WWP Wipo information: published in national office

Ref document number: 1019997004575

Country of ref document: KR

WWG Wipo information: grant in national office

Ref document number: 1019997004575

Country of ref document: KR

WWG Wipo information: grant in national office

Ref document number: 1998935253

Country of ref document: EP