KR20080002915A - Apparatus and method for processing video data - Google Patents

Abstract

An apparatus and methods for processing video data are described. The invention provides a representation of video data that can be used to assess agreement between the data and a fitting model for a particular parameterization of the data. This allows the comparison of different parameterization techniques and the selection of the optimum one for continued video processing of the particular data. The representation can be utilized in intermediate form as part of a larger process or as a feedback mechanism for processing video data. When utilized in its intermediate form, the invention can be used in processes for storage, enhancement, refinement, feature extraction, compression, coding, and transmission of video data. The invention serves to extract salient information in a robust and efficient manner while addressing the problems typically associated with video data sources.

Description

Apparatus and method for processing video data {APPARATUS AND METHOD FOR PROCESSING VIDEO DATA}

This application is filed March 31, 2005, and entitled "System and Method For Video Compression Employing Principal Component Analysis." US Provisional Application No. 60 / 667,532, filed April 13, 2005, entitled " Apparatus and Methods for Processing Video Data " Claim priority on 60 / 670,951. This application is filed in the United States application No. United States Application No. 11 / 336,366, filed on November 16, 2005. Some continuing applications of 11 / 280,625, filed September 20, 2005, filed in US Application No. Part of the ongoing application on 11 / 230,686, filed on July 28, 2005, in US Application No. Some continuing applications of 11 / 191,562. Each of the foregoing applications is incorporated herein by reference in its entirety.

TECHNICAL FIELD The present invention generally relates to the field of digital signal processing, and more particularly, to a computer device and a computer implemented method for efficient representation and processing of signal or image data, and most particularly to efficient representation and processing of video data. Computer apparatus and a computer implemented method.

A general system description of the prior art to which the present invention belongs may be expressed as in FIG. Here, the block diagram shows a conventional prior art video processing system. Such a system typically includes the following steps: input step 102, processing step 104, output step 106, and one or more data storage mechanisms 108.

The input step 102 may include means for retrieving data from a camera sensor, camera sensor array, area search sensor, or storage mechanism. The input step provides video data representing a time correlation sequence of anthropogenic and / or naturally occurring phenomena. Salient components of the data may be shielded or contaminated by noise or other unwanted signals.

Video data in the form of data streams, arrays or packets may be provided to the processing step 104 either directly or via an intermediate storage element 108 in accordance with a predetermined delivery protocol. Processing step 104 may be a dedicated analog or digital device, or program such as a central processing unit (CPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), to perform a desired set of video data processing operations. It may have the form of a possible device. Processing step 104 typically includes one or more CODECs (coders / decoders).

The output step 106 generates a signal, display, or other response that may affect the user or external device. Typically, the output device is used to generate an indicator signal, display, hard copy, representation of the processed data in the repository, or to initiate the transfer of data to a remote site. This may be used to provide an intermediate signal or control parameter for use in subsequent processing operations.

The repository is provided as an optional element in such a system. When used, storage element 108 may be nonvolatile, such as a read-only storage medium, or may be volatile, such as dynamic random access memory (RAM). It is not unusual for a single video processing system to contain several types of storage elements having various relationships with input, processing and output stages. Examples of such storage elements include input buffers, output buffers, and processing caches.

The primary purpose of the video processing system of FIG. 1 is to process input data to produce meaningful output in a particular application. To achieve this goal, various processing including noise reduction or removal, feature extraction, object segmentation and / or normalization, data categorization, event detection, editing, data selection, data re-coding, and transcoding Operations may be used.

Many data sources that produce incompletely limited data are important to humans, particularly acoustic and visual images. In most cases, the essential features of these source signals adversely affect the purpose of efficient data processing. The inherent variability of the source data is an obstacle to processing the data in a reliable and efficient manner without introducing errors arising from pure empirical and heuristic methods used to derive engineering hypotheses. If the input data is naturally or deliberately limited to a narrowly defined feature set (eg, a limited set of symbol values or narrow bandwidth), this changeability for applications is reduced. With all of these limitations, processing techniques with low commercial value arise very frequently.

The design of a single processing system is influenced by the expected characteristics of the source signal used as input and the intended use of the system. In most cases, the required performance efficiency will be an important design factor. In turn, performance efficiency is affected by the amount of data to be processed relative to the available data store and the computational complexity of the application relative to the available computational power.

Conventional video processing methods suffer from a number of inefficiencies in the form of slow data communication speeds, large storage needs, and disturbing conscious artifacts. These can be serious problems because of the various ways that people want to use and manipulate video data, and because of the inherent sensitivity people have to certain types of visual information.

An "optimal" video processing system is efficient, reliable and robust in carrying out a desired set of processing operations. Such operations include storage, transmission, display, compression, editing, encryption, enhancement, categorization, feature detection, and recognition of data. Secondary operations include integrating this processed data with other information sources. Equally important, for video processing systems, the output must be compatible with human vision by preventing the introduction of conscious artifacts.

A video processing system can be expressed as "robust" if its speed, efficiency, and quality do not depend on the details of any particular characteristic of the input data. Robustness also relates to the ability to perform operations when part of the input is an error. Many video processing systems are not robust enough to allow for a general class of applications-they only provide application to the same narrowly defined data that was used for development in the system.

Salient information may be lost when discretizing a continuous value data source due to the sampling rate of the input element that does not match the signal characteristic of the sensed phenomenon. In addition, there is a loss when the strength of the signal is saturated beyond the limit of the sensor. Similarly, information is lost when the precision of the input data is reduced, which occurs in all quantization processes when the values of the entire range of input data are represented by a set of discrete values, thereby reducing the accuracy of the data representation. .

Total variability refers to all unpredictability in the class of data or information source. The data representation of visual information has a very large total changeability rating because visual information is not typically limited. The visual data may represent all spatial array sequences or space-time sequences that may be formed by light incident on the sensor array.

In modeling visual phenomena, video processors typically impose some set of restrictions and / or structure on the way data is represented or translated. As a result, this method may include: quality of output; Trust given to the output; And types of subsequent processing tasks that can be reliably executed on the data; Introduce systematic errors that affect.

The quantization method reduces the accuracy of the data in the video frame while attempting to maintain statistical changes in the data. Typically, when video data is analyzed, the distribution of data values is collected in a probability distribution. Another method is to project the data into phase space to characterize the data as a mixture of spatial frequencies, thereby giving off a reduction in accuracy in a less reluctant way. When used heavily, this quantization method can produce perceptually unacceptable colors and cause sharp pixilation in the original flat region of the video frame.

Differential coding is also commonly used to take advantage of local spatial similarity of data. Data in a portion of a frame tends to cluster near similar data in that frame and also in similar locations in subsequent frames. Representing data spatially in terms of adjacent data can then be combined with quantization, and the end result is that representing the difference for a given precision is more accurate than using the absolute value of the data. This assumption fits well when the spectral resolution of the original video data is limited, such as black and white video or low color video. As the spectral resolution of the video increases, the assumption of similarity is significantly off. This deviation is due to the inability to selectively preserve the precision of the video data.

The rest of the coding is similar to differential encoding in that the error in the representation is further differentially encoded to restore the precision of the original data to the desired level of accuracy.

Variations of this method seek to transform video data into alternative representations that expose data correlations in spatial phase and scale. Once the video data has been transformed in this manner, then quantization and differential coding methods can be applied to the transformed data, increasing the preservation of the salient image characteristics. Two of these widely used transformed video compression techniques are discrete cosine transform (DCT) and discrete wavelet transform (DWT). Errors in the DCT transform appear in wide variations in video data values, and therefore DCT is typically used for blocks of video data to localize such false correlations. Artifacts from this localization often appear along the boundaries of the block. In the case of DWT, more complex artifacts occur when there is a mismatch between the basic function and a given texture, which causes a blurring effect. To counter the negative effects of DCT and DWT, the precision of representation is increased with low distortion at the expense of precise bandwidth.

The present invention is a computer-implemented video processing method that provides both computational and analytical advantages over existing modern methods. The main method of the present invention is the integration of the linear decomposition method, the spatial partitioning method, and the spatial normalization method. Spatially constrained video data greatly increases the robustness and applicability of the linear decomposition method. In addition, it may further function to increase the benefits derived from the spatial partition spatial normalization itself of the data corresponding to the spatial normalization.

In particular, the present invention provides a means by which signal data can be efficiently processed into one or more useful representations. The present invention is efficient for processing many commonly occurring data sets and is particularly efficient for processing video and image data. The method of the present invention provides one or more concise representations of the data to analyze the data and to facilitate processing and encoding. Each new, more compact data representation reduces computational processing, transmission bandwidth, and storage requirements for many applications, including but not limited to encoding, compression, transmission, analysis, storage, and display of video data. To make it possible. The present invention includes a method for the identification and extraction of salient elements of video data, and enables prioritization in the processing and presentation of data. Since noise and other portions of the unwanted signal are identified at low priority, further processing may be focused on analyzing and representing higher priority portions of the video signal. As a result, the video signal is much more concise than previously possible. And the loss in precision is concentrated on a portion of the video signal that is not conceptually important.

1 is a block diagram illustrating a prior art video processing system.

2 is a block diagram that provides an overview of the present invention showing the major modules for processing video.

3 is a block diagram illustrating a motion estimation method of the present invention.

4 is a block diagram showing a wide area registration method of the present invention.

5 is a block diagram illustrating a normalization method of the present invention.

6 is a block diagram illustrating a hybrid spatial normalized compression method.

7 is a block diagram illustrating the mesh generation method of the present invention used in local normalization.

8 is a block diagram illustrating the mesh-based normalization method of the present invention used in local normalization.

9 is a block diagram illustrating the combined global and local normalization method of the present invention.

10 is a block diagram illustrating the GPCA-based polynomial fitting and differential method of the present invention.

11 is a block diagram illustrating an iterative GPCA purification method of the present invention.

12 is a block diagram showing a background decomposition method.

13 is a block diagram showing the object segmentation method of the present invention.

14 is a block diagram illustrating an object interpolation method of the present invention.

In video signal data, the frames of video are assembled into a sequence of images that generally depict the projected and imaged three-dimensional scene on a two-dimensional imaging surface. Each frame or image consists of pixels (pels) representing image sensor responses to the sampled signal. Often, the sampled signal corresponds to electromagnetic energy (eg, electromagnetic, acoustic, etc.) that has been reflected, refracted, or radiated and sampled by the two-dimensional sensor array. By successive sequential sampling, a spatiotemporal data stream is generated having two spatial dimensions per frame and a temporal dimension corresponding to the order of the frames in the video sequence.

As shown in Figure 2, the present invention analyzes the signal data and identifies the protruding components. When a signal consists of video data, the spatiotemporal stream represents a salient component, which is often a specific object such as a face. The identification process quantifies the presence and importance of overhanging components and selects one or more of the most important of these quantified overhanging components. This does not limit the identification and processing of other less protruding components after or concurrently with the currently described processing. The above-described overhang component is then further analyzed to identify variable and immutable subcomponents. Identification of an invariant subcomponent is the process of modeling certain characteristics of the component, which represents the parameterization of the model by which the component is synchronized to a desired level of accuracy.

In one embodiment, the foreground object is detected and tracked. The pixels of the object are identified and segmented from each frame of video. Block based motion estimation is applied to partitioned objects in multiple frames. This motion estimation is then integrated into the higher layer motion model. The motion model is used to warp instances of objects for common spatial configurations. For some data, in this configuration, more properties of the object are aligned. This normalization allows a linear decomposition of the pixel's value of an object over multiple frames to be represented compactly. Extruding information pertaining to the appearance of the object is included in this compact representation.

The preferred embodiment of the present invention details the linear decomposition of the front video object. Objects are spatially normalized to create a compact linear appearance model. Another preferred embodiment further partitions the front object from the back of the video frame prior to spatial normalization.

 A preferred embodiment of the present invention applies the present invention to video of a person talking to a camera while performing a small amount of motion.

The preferred embodiment of the present invention applies the present invention to a given object in a video that can be well represented through spatial transformation.

Preferred embodiments of the present invention use block-based motion estimation to determine finite differences between two or more frames of video. Higher grade motion models are factored from the above defined differences to provide more efficient linear decomposition.

Detection and tracking

It is known in the art to detect an object within a frame and track that object through a predetermined number of subsequent frames. Among the algorithms and programs that can be used to perform object tracking are Viola / Jones: P. Viola and M.Jones, "Robust Real-time Object Detection" in Proc. 2nd Int'l Workshop on Statistical and Computational Theories of Vision-Modeling, Learning, Computing and Sampling, Vancouver, Canada, July 2001. Similarly, there are numerous algorithms and programs that can be used to track the detected object through successive frames. Examples include: C. Edwards, C. Taylor, and T. Cootes. "Learning to identify and track faces in an image sequence." Proc. Int'l Conf. Auto. Face and Gesture Recognition, pages 260-265, 1998.

The result of the object detection process is a data set that specifies the general position of the center of the object within the frame and an indication of the scale (size) of the object. The result of the tracking process is a data set that indicates the temporary label of the object and at some level of probability confirms that the object detected in successive frames is the same object.

Object detection and tracking algorithms can be applied to one object in a frame or to two or more objects in frames.

It is also known to track one or more features of an detected object within a group of consecutive frames. If the object is a human face, for example, the feature can be an eye or a coil. In one technique, a feature is represented by the intersection of "lines" which can be described as roughly "corners." Preferably, the "edges" which are intense and spaced apart from each other are selected as a feature. Features can be identified through spatial density field gradient analysis By using hierarchical multiresolution estimation of optical flow, it is possible to determine the transient displacement of features in successive frames. Y.Yacoob's "Tracking and recognizing rigid and non-rigid facial motion ysing local parametric models of image motions" In Proceedings of the International Conference on Computer Vision, pages 374-381, Boston, Mass., June 1995 This is an example of an algorithm using this technique.

Once the component salient components of the signal are determined, these components may be limited and all other signal components may be reduced or eliminated. The process of detecting the protruding component is shown in FIG. 2 and the video frame 202 is processed by one or more object detection 206 processes such that one or more objects are identified and then tracked. The restricted component represents an intermediate form of video data. This intermediate data can then be encoded using techniques that are not typically available for current video processing methods. Since intermediate data exists in several forms, standard video encoding techniques may be used to encode these various intermediate forms. For each example, the present invention determines and uses the most efficient encoding technique.

In one preferred embodiment, the protrusion analysis process detects and classifies the protrusion signal mode. One embodiment of this process uses a combination of specifically specified spatial filters to generate a response signal, the strength of which is related to the detected protrusion of an object within the video frame. The classifier is applied at different spatial scales and at different video frame positions. The intensity of the response from the classifier indicates the possibility of the presence of the salient signal mode. If a prominent overhanging object is in the center, the process classifies it as a strong response. Detection of the salient signal mode differentiates the present invention by enabling subsequent processing and analysis of salient information in the video sequence.

Given the detection position of the salient signal mode in one or more video frames, the present invention analyzes the invariant characteristics of the salient signal mode. In addition, the present invention analyzes the remaining signals that are in a "less protruding" signal mode for invariant properties. Identification of invariant features reduces the redundant information and provides a basis for dividing (ie, separating) signal modes.

Attribute point tracking

In one embodiment of the present invention, the spatial position of one or more frames is determined through spatial intensity field gradient analysis. This feature corresponds to several intersections of "lines", which lines may be loosely described as "corners". This embodiment selects a set of corners that are both strong corners and spatially separated from each other, which is referred to herein as a feature point. In addition, the hierarchical multi-resolution estimation of the optical flow makes it possible to determine the transform displacement of a characteristic point over time.

In FIG. 2, an object tracking 220 process is shown that derives a detection instance from the object detection process 208, and further corresponds to the correspondence of the properties of one or more detected objects with respect to the plurality of video frames 202 and 204. A process 222 of identifying a last name is shown.

Non-limiting embodiments of feature tracking can be used so that features can be used to qualify more consistent gradient analysis methods such as block-based motion estimation.

Another embodiment anticipates the prediction of motion estimation based on characteristic tracking.

Object based detection and tracking

In a non-limiting embodiment of the present invention, a robust object classifier is used to track the faces in the frame of video. This classifier is based on the serial response for the directed edge trained for the faces. In this classifier, the edges are defined as the set of basic Haar properties and the rotation of these properties by 45 degrees. Serial classifiers are a variation of the AdaBoost algorithm. In addition, the response calculation can be optimized through the use of the summed region table.

Local registration

Registration includes assignment of a correspondence between elements of an object identified in two or more video frames. This correspondence is the basis for modeling the spatial relationship between video data at temporally distinct points in the video data.

Various non-limiting registration means have been described with reference to the present invention in order to illustrate specific embodiments for implementing with respect to known algorithms and their derivatives, and their associated reductions.

One means of modeling the apparent optical flow in space-time sequences can be achieved through the generation of finite differential fields from two or more frames of video data. If the correspondence follows certain invariant constraints in the spatial and intensity sense, the optical flow field can be estimated roughly.

As shown in FIG. 3, frame 302 or 304 is spatially subsampled through decimation process 306 or some other sub-sampling process (eg, a low pass filter). These spatially reduced images 310 and 312 can also be further subsampled.

Diamond search

Given a non-overlapping split of blocks of video frames, the previous frame of video is searched for a match for each block. Full search block based (FSBB) block motion estimation searches for the location of the previous frame of video with the least error as compared to the block in the current frame. Implementing FSBB is computationally very expensive and often does not yield good matching compared to other motion estimation schemes based on the assumption of localized motion. Diamond Search Block Based (DSBB) Gradient Falling Motion Estimation is a common alternative to FSBB that uses diamond shaped search patterns of various sizes to iteratively traverse the error gradient towards the best match for the block.

In one embodiment of the invention, DSBB is used in the analysis of the image gradient field between one or more frames of video to produce a finite difference whose value is later decomposed into a higher grade motion model.

Those skilled in the art will appreciate that block-based motion estimation can be seen as an equivalent of the analysis of regular mesh vertices.

Mesh based motion estimation

Mesh-based prediction uses a geometric mesh of vertices connected by edges to depict discrete regions of a video frame, and then those in subsequent frames through a deformation model controlled by the position of the mesh vertices. Predict the deformation and movement of regions. When the vertices are moved, the pixels in the area defined by the vertices are also moved to predict the current frame. The relative movement of the original pixel value and the resulting approximation is performed through some interpolation that associates the pixel location with the location of the vertices in the periphery of that pixel. Additional modeling of scaling and rotation compared to pure transformation can produce more precise predictions of the pixels of a frame when such motion is present in the video signal.

In general, a mesh model can be defined as being regular or adaptive. The regular mesh model is designed without considering the underlying signal characteristics, while the adaptive method attempts to spatially place vertices and edges in relation to the characteristics of the underlying video signal.

Assuming that the imaged object in the video has a spatial discontinuity that corresponds well with the edges in the mesh, regular mesh representation provides a means by which motion or evenly inherent deformations in motion can be predicted or modeled.

Adaptive meshes are formed with substantially more consideration for the characteristics of the underlying video signal than regular meshes. In addition, the adaptive nature of this mesh allows for various refinements of the mesh over time.

The present invention adjusts vertex search alignment using homogeneity criteria to perform mesh and evenly pixel registration. The vertices that are spatially associated with the heterogeneous intensity gradients are motions that are estimated before having more homogeneous gradients.

In a preferred embodiment, the vertex motion estimation of the mesh is additionally prioritized through spatial flood-filling of motion estimation for vertices having the same or nearly the same homogeneity.

In a preferred embodiment, the initial mesh space structure and the final mesh structure are mapped to each other at the face level by filling the mapping image with face identifiers using a standard graphic filling routine. The affine transformation associated with each triangle can be found quickly in the translation table and the pixel positions associated with the face in one mesh can be quickly transformed into positions in the other mesh.

In a preferred embodiment, preliminary motion estimation is performed on the vertices to estimate the remaining error associated with each motion estimation match. This preliminary estimation is additionally used to prioritize the motion estimation order of the vertices. The advantage of this remaining error analysis is that motion estimation associated with less distortion will result in a more likely mesh topology.

In the preferred embodiment, the mesh vertex motion estimation scales down to some limited range, and multiple motion estimations are made through multiple iterations so that the mesh can access a more globally optimal and topologically accurate solution. .

In a preferred embodiment, block-based motion estimation using rectangular tile proximity centered at each vertex is used to determine the vertex displacement taking into account interpolated polygonal proximity. In addition to avoiding spatial interpolation and distortion of pixels for reducing error gradients, this technique also allows parallel computation of motion estimation.

Phase based motion estimation

In the prior art, block-based motion estimation has typically been performed as a spatial search resulting in one or more spatial matches. As shown in Figure 3, a phase based normalized cross correlation (PNCC) transforms blocks from the current frame and the previous frame into "phase space" and searches for the cross correlation of these two blocks. Cross correlation is expressed as a field of values whose position corresponds to the "phase shift" of the edge between two blocks. These positions are separated through the thresholds and are therefore inversely transformed into spatial coordinates. The spatial coordinates are the individual edge displacements and correspond to the motion vectors.

Advantages of the PNCC include contrast masking, which allows tolerance of gain / exposure adjustments in the video stream. In addition, the PNCC allows for elapsed from a single step, which may take many iterations from spatially based motion estimation. In addition, motion estimation is subpixel accuracy.

One embodiment of the present invention uses PNCC in the analysis of an image gradient field between one or more frames of video to produce a finite difference whose value is later decomposed into a higher grade motion model.

Global registration

In one embodiment, the present invention creates a correspondence model by using a relationship between corresponding elements of an object detected in two or more video frames. This relationship decomposes one or more linear models from the field of finite difference estimation. The term field refers to each finite difference with spatial location. This finite difference may be the transient displacement of the corresponding object feature in the heterogeneous frames of video described in the detection and tracking section. The field in which this sampling occurs is referred to herein as a general population of finite differences. The described method is described in M.A.Fischler, R.C. Bolles. "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography" Comm. We use a robust estimation similar to that of the RANSAC algorithm described in of the ACM, Vol 24, pp 381-395, 1981.

As shown in FIG. 4, in the case of global motion modeling, the finite difference is a transformed motion estimation 402 that is collected into a general population pool 404, which pools 402 to a random sampling 410 of such motion estimation. Is repeatedly processed and a linear model is factored from these samples (420). The results are then used to adjust the population 404 to better classify the linear model through the exclusion of outliers to the model when looking through the random process.

The present invention may utilize one or more robust estimators, one of which may be a RANSAC robust estimation process. Robust estimators are well described in the prior art.

In one embodiment of the linear model estimation algorithm, the motion model estimator is based on a linear least square solution. Due to this dependency, the estimator is separated by outlier data. Based on RANSAC, the disclosed method is a robust method that reverses the effect of outliers through iterative estimation of a subset of data, and searches for a motion model to account for an important subset of data. The model generated by each search is tested against the percentage of data represented by this model. If there are enough repetitions, a model will be found that fits the largest subset of data. A description of how to perform this robust linear least square regression is described in R. Dutter and P.J.Huber's "Numerical methods for the nonlinear robust regresion problem." Journal of Statistical and Computational Simulation, 13: 79-113, 1981.

As understood and described in FIG. 4, the present invention discloses an innovation that exceeds the RANSAC algorithm, which is a form of modification of the algorithm including initial sampling of finite differences (samples) and least squares estimation of the linear model. The comprehensive error is evaluated for all samples in the general population using the solved linear model. The rank is assigned to the linear model based on the number of samples whose remainder matches the predetermined threshold, which rank is considered a "candidate consensus".

Initial sampling, solutions, and rankings are performed repeatedly until the result standard is met. Once the standard is met, the linear model with the highest rank is considered the final consensus of the population.

The option refinement step includes iteratively analyzing the subset of samples at the best fit class for the candidate model, and increasing the size of the subset until one or more sample additions exceed the remaining error thresholds for the entire subset. It includes.

As shown in FIG. 4, the global model estimation process 450 is repeated until the consensus rank acceptance test is satisfied 452. When no rank is obtained, the population of finite differences 404 is classified for the model found in an effort to show a linear model. The best (highest rank) motion model is added to the solution set in process 460. The model is then reestimated at process 470. Upon completion, population 404 is rearranged.

The described non-limiting embodiment of the present invention is further generalized as a general method of sampling the aforementioned vector space as a field of finite difference vectors to determine the subspace manifold in another parameter vector space corresponding to a particular linear model.

An additional consequence of the wide area registration process is that the difference between the wide area registration process and the local block process results in local registration rest. This remainder is an error in the global model in approximating the local model.

Normalization

Normalization means resampling a spatial intensity field towards a standard, or common spatial configuration. When this relative spatial configuration is a reversible spatial change between the configurations, the resampling and the accompanying interpolation of pixels are invertible to the topological limit. The normalization method of the present invention is disclosed in FIG.

When two or more spatial intensity fields are normalized, increased computational efficiency may be achieved by maintaining an intermediate normalization calculation.

The spatial transformation model used to sample the image for registration, or equivalently for normalization, includes global and local models. The global model increases the grade from transformation to projection. The local model is a finite difference that applies interpolation for neighboring pixels that are basically determined by blocks or, more complexly, linear meshes.

Interpolation of the original intensity field to the normalized intensity field increases the linearity of the PCA emergence model based on a subset of the intensity field.

As shown in FIG. 2, object pixels 232 & 234 can be sampled 240 again to produce a normalized version of object pixels 242 & 244.

Mesh-based normalization

A further embodiment of the invention makes the feature points a triangular-based mesh, the vertices of the mesh are tracked, and the relative position of each triangle vertex is 3 perpendicular to the plane coinciding with these three vertices. Used to estimate the dimensional surface. When the vertical surface coincides with the protruding axis of the camera, it can provide a minimal deformed rendering of the object corresponding to the imaged triangle. Generating a normalized image that tends to favor perpendicular orthogonal surfaces can produce pixels that preserve intermediate data shapes that increase the linearity of subsequent appearance-based PCA models.

Another embodiment uses conventional block-based motion estimation to implicitly model the global motion model. In one non-limiting embodiment, the method factors the global affinity motion model from the motion vectors described by conventional block-based motion estimation / prediction.

The method of the present invention utilizes one or more global motion estimation techniques including linear solutions to a set of pseudo-protruding solutions. Other protruding models and solution methods are described in the prior art.

9 illustrates a method of combining global and local normalization.

Progressive Geometry Normalization

A classification of spatial discontinuity is used to align the tessellation mesh to implicitly model the discontinuity when the discontinuity coincides with the mesh edges.

Homogeneous region boundaries are approximated by polygonal trajectories. The trajectories are successively approximated with successively lower precision to determine the salience priority of each polygonal vertex. Vertex priority propagates across regions to preserve vertex priority for shared vertices.

In one embodiment of the present invention, the polygon decomposition method enables the prioritization of the boundaries associated with homogeneous classification of fields. The pixels are classified according to homogeneity criteria such as spectral similarity, and then the classification label is spatially connected to the regions. In a further preferred non-limiting embodiment, 4- or 8-connectedness criteria are applied to determine spatial connectivity.

In a preferred embodiment, the boundaries of these spatial regions are then discretized into polygons. The spatial overlay of all polygons for all homogeneous regions is then tessellated and combined into a preliminary mesh. The vertices of this mesh are decomposed using a number of criteria, creating a simpler mesh representation that retains the many perceptible protrusions of the original mesh.

In a preferred embodiment, the image registration method disclosed elsewhere herein is biased towards these high priority vertices using strong image gradients. The resulting deformation model tends to maintain the spatial discontinuity associated with the geometry of the imaged object.

In a preferred embodiment, active contours are used to refine the region boundaries. The active trajectory for each polygon is allowed to propagate one iteration. The "deformation" or motion of each active trajectory vertex is combined into an averaged motion to allow forced propagation of the implied mesh, all of which have membership.

In a preferred embodiment, the vertices are assigned to a count of the number of adjacent vertices that they have for adjacent vertices that are also part of the trajectory of the different area. These other vertices are defined as opposites. In the case of a vertex with a count of 1, it does not have a massive vertex and thus needs to be preserved. Both adjacent opposite vertices have a count of 1 (meaning that these two vertices are in different polygons and are adjacent to each other), and then one vertex is analyzed for the other. When a vertex of 1 opposes a neighboring polygonal vertex with a value of 2, a vertex with a count of 1 is decomposed into a vertex with a count of 2, and the vertex count becomes 1. Therefore, if there is another neighboring opposite vertex, this vertex can be decomposed again. For this case, it is important to store the original vertex count, so when the vertex is decomposed, we can deflect the direction of decomposition based on the original vertex count. Thus, after vertex a is decomposed to vertex b, vertex b will not decompose to vertex c, instead vertex c should be decomposed to vertex b, because b has already been used in one decomposition.

In a preferred embodiment, the T junctions are specifically processed. These junctions are points in a polygon that do not have points in adjacent polygons. In this case, each polygon vertex is first plotted on an image point map, which map identifies the vertex's spatial location and the vertex's polygon identifier. Each polygon boundary is then measured by traverse method and tested to see if there are adjacent vertices from another polygon. If there are neighboring vertices from other regions, they are tested to see if they each have neighboring vertices from the current polygon. Otherwise, the current point is added as the vertex of the current polygon. This extra test ensures that isolated vertices in other polygons are used to create the T junction. Otherwise, it will just add new vertices-this region already had matching vertices. Thus, opposite vertices are added as long as neighboring vertices are not opposed by this current region. In a further embodiment, the efficiency of detecting the T junction is increased by using a mask image. The polygon vertices are visited sequentially, and the mask is updated to identify the pixels of the vertices as belonging to the polygon vertices. Then, if polygon boundary pixels are measured by the traverse method and they coincide with polygon vertices, they are recorded as vertices in the current polygon.

In a preferred embodiment, when the spectral region has been remapped by one or more overlapping homogeneous image oblique regions and another homogeneous spectral region also overlaps, all regions already remapped are identical to those regions currently being remapped. Receive a label. Thus in essence, if the spectral regions overlap by two homogeneous regions, all the spectral regions overlapped by these two homogeneous regions will get the same label, so that one spectral region replaces the two homogeneous regions Seems to be truly covered by one homologous region.

In one embodiment of the invention, it is advantageous to process the region map rather than the region list to find adjacent merge regions. In a further embodiment, the spectral split classifier may be modified to train the classifier using nonhomogeneous regions. This allows processing to concentrate on the edge of the spectral region. In addition, adding different partitions based on using edges (eg, canny edge detector), and then feeding them with active trajectories to identify the initial set of polygons is more of a homogeneous region. Enable large discrimination

Local normalization

The present invention provides a means by which pixels in a space-time stream can be registered in a 'local' manner.

One such localized method describes the localized coherency of the imaged mapping when solving the apparent image brightness invariance ambiguity in relation to the imaged mapping or specifically the local deformation of the imaged object. The spatial application of the geometric mesh is used to provide a means for analyzing the pixels as much as possible.

This mesh is used to provide a piece-wise linear model of surface deformation in the image plane as a means of local normalization. Imaged mapping often corresponds to this model when the temporal analysis of the video stream is high compared to motion in the video. Exceptions to model assumptions are addressed through various techniques, including topological constraints, peripheral vertex limitations, and analysis of homogeneity of pixel and image gradient regions.

In one embodiment, the feature point is used to create a mesh composed of triangular elements whose vertices correspond to the feature point. Corresponding feature points are another frame that suggests an interpolated "warping" of the triangle and corresponding pixels to produce a local deformation model.

7 illustrates the creation of such an object mesh. 8 illustrates the use of such an object mesh to normalize a frame locally.

In one preferred embodiment, a triangular map is generated that identifies the triangle from which each pixel of the map was obtained. In addition, the affine transformation corresponding to each triangle is precomputed as an optimization step. In addition, when generating a local deformation model, an anchor image is measured by a traverse method in advance using spatial coordinates to determine coordinates of a source pixel to be sampled. This sampled pixel will replace the current pixel position.

In another embodiment, local modifications are performed after global modifications. In the previously disclosed specification, global normalization has been described as a process in which a global registration method is used to spatially normalize pixels in two or more frames of video. The resulting globally normalized video frame can be further normalized locally. The combination of these two methods limits local normalization to refinements that arrive globally in the year. This significantly reduces the ambiguity that the local method needs to solve.

In another non-limiting embodiment, the feature points, or vertices in the case of a "normal mesh", are qualified through analysis of the image tilt around these points. This image slope can be calculated either directly or through some indirect calculations such as Harris response. In addition, these points may be filtered by motion estimation errors or spatial constraints associated with the falling of the image tilt. Fitted points can be used as the basis for a mesh by one of many tessellation techniques, resulting in a mesh whose elements are triangular. For each triangle an affine model is generated based on the points and their residual motion vector.

The method of the present invention utilizes one or more image density gradient analysis methods that include a Harris response. Other image density gradient analysis methods are described in the prior art.

In a preferred embodiment, a list of triangular affine parameters is maintained. This list is repeated and the current / previous point list is constructed (using the vertex survey map). The current / previous point list is passed to a routine used to estimate the transform that computes the affine parameter for the triangle. The affine parameter or model is then stored in the triangle affine parameter list.

In a further embodiment, the method measures the triangle identifier image map by traverse method, wherein each pixel in the map includes an identifier for a triangle in a mesh in which the pixel has membership. And for each pixel belonging to the triangle, the corresponding global and local deformation coordinates for that pixel are calculated. These coordinates are again used to sample the corresponding pixel and apply its value to the corresponding "normalized" position.

In a further embodiment, spatial constraints are applied to the points based on image intensity corresponding strength and density generated from the retrieval of the image tilt. The points are aligned after the motion estimation is done based on some norms of the rest of the image intensity. The points are then filtered based on the spatial density constraints.

In further embodiments, spectral spatial partitioning is used, where small homogeneous spectral regions are merged with neighboring regions based on spatial proximity, similarity of their density, and / or color. Homogeneous merging is then used to combine the spectral regions together based on their overlap with the regions of the homogeneous texture (image gradient). Further embodiments then use a center-surround point where small areas are surrounded by larger areas as a fitted point of interest to support the vertices of the mesh. In a further non-limiting embodiment, the central enclosing point is defined as the area where a bounding box exists within one pixel of size 3x3 or 5x5 or 7x7 pixels, for this bounding box. The spatial image slope is a corner shape. The center of this region can be classified as a corner and further adapts this position to an advantageous vertex position.

In further embodiments, horizontal and vertical pixel finite difference images are used to classify the intensity of each mesh edge. If an edge has many finite differences that coincide with the spatial position, the edges and therefore the vertices of this edge are considered to be very important for the local deformation of the imaged mapping. If there is a large differential coefficient difference between the means of the sum of the finite differences of the edges, then the area edges probably correspond to the texture changing edges, not the quantization step.

In further embodiments, spatial density model termination conditions are used to optimize the processing of mesh vertices. When a sufficient number of points have been examined that cover most of the initial spatial area of the detection rectangle, processing can be terminated. Closing generates a score. Vertices and feature points input to the processing are sorted by this score. If the points are too close in space to existing points, or if the points do not correspond to edges in the image slant, this point is ignored. If not, then the image slope around the point is lowered, and if the rest of the slope exceeds the limit, this also is ignored.

In a preferred embodiment, local deformation modeling is performed iteratively, converging on the solution as the vertex displacement per iteration decreases.

In another embodiment, local deformation modeling is performed and the model parameters are ignored if the global deformation has already provided the same normalization benefit.

Regular Mesh Normalization

The present invention extends the above-described local normalization method using regular meshes. This mesh is constructed irrespective of the underlying pixel, but is positioned and sized corresponding to the detected object.

Given the detected object region, a scale representing the spatial frame position and the size of the surface produces a regular mesh with respect to the beginning of the surface region. In a preferred embodiment, a diagonal set of tiles is used to create a regular mesh with triangular mesh elements after using a non-overlapping set of tiles to outline a square mesh. In a further preferred embodiment, the tiles are proportional to those used in conventional video compression algorithms (eg MPEG-4 AVC).

In a preferred embodiment, the vertices associated with the mesh described above are prioritized through analysis of the area of pixels surrounding these vertices in specific frames of video used for training. Analysis of the slope for this region provides confidence in the processing (eg block based motion estimation) associated with each vertex that depends on the local image slope.

Correspondence of vertex positions in multiple frames is found through a simple descent of the image tilt. In a preferred embodiment, this is obtained through block based motion estimation. In this embodiment, the peak of high reliability enables the correspondence of high reliability. Low reliability vertex correspondence is implicit by solving ambiguous image gradients through inference from the high reliability vertex correspondence.

In one preferred embodiment, a regular mesh is created for the initiation tracking rectangle. The tiles are produced 16 × 16 and cut diagonally to form a triangular mesh. The vertices of these triangles are motion estimated. Motion estimation depends on the type of texture each point has. The texture is divided into three classes, corner, edge, and homogeneous, which also defines the order of processing of the vertices. Corner vertices use neighboring vertex estimation, that is, motion estimation of neighboring points (if available) is used for the predictive motion vector, and motion estimation is applied to each. The motion vector that provides the lowest mad error is used as this vertex motion vector. The search strategy used for the corners is everything (wide, small, first). For the edges, the closest peripheral motion vectors are again used as the predictive motion vector, and the one with the least amount of error is used. The search strategy for the edge is small and the first one. For uniform planes, neighboring vertices are searched and motion estimation with the lowest error is used.

In one preferred embodiment, the image slope for each triangle vertex is calculated and aligned based on the class and magnitude. Thus the corners are before the edges and the edges are before the uniform plane. For corners, the strong corner is before the weak corner, and for the edge, the strong edge is before the weak edge.

In one preferred embodiment, the local deformation for each triangle is based on the motion estimation associated with that triangle. Each triangle has an affine estimated for it. If the triangle is not topologically inverted or altered, the pixels that are part of the triangle are used to sample the current image based on the estimated affines obtained.

Segmentation

The spatial discontinuities identified through the segmentation process described further are efficiently encoded through the geographical parameterization of their respective boundaries, which is called the spatial discontinuity model. This spatial discontinuity model may be encoded in an advanced way that enables an even more concise limit description corresponding to a subset of encoding. Advanced encoding provides a robust means of prioritizing the spatial geometry while retaining most of the spatial discontinuity of the protruding properties.

Preferred embodiments of the present invention combine multiresolution segmentation analysis with gradient analysis of spatial density fields and use temporary stability constraints to obtain robust segmentation.

As shown in FIG. 2, once the correspondence of an object's features has been tracked 220 and modeled 224 over time, faithful support for this motion / deformation model is required to segment 230 the pixels corresponding to the object. Can be used. This process may be repeated for the detected number of objects 206 & 208 in the video 202 & 204. The result of this processing is the segmented object pixel 232.

One form of invariant characterization used by the present invention focuses on the identification of spatial discontinuities. This discontinuity reveals edges, shadows, occlusions, lines, corners, or some other visible property that causes sudden and identifiable separation between pixels of one or more imaged frames of the video. Additionally, subtle spatial discontinuities between similarly colored and / or textured objects are revealed when the pixels of an object in a video frame undergo coherent motion with respect to the object itself, but in different motions with respect to each other. The present invention utilizes a combination of spectrum, texture, and motion segmentation to robustly identify spatial discontinuity in relation to the salient signal mode.

Temporal splitting

Temporal integration of transform motion vectors, or finite difference measurements into equivalent higher-grade models, is a form of motion partitioning described in the prior art.

In one embodiment of the invention, a dense field of motion vectors is generated that represents the finite difference of object motion in the video. These derivatives are spatially grouped with each other through constant division of tiles, or by some initial procedure such as spatial division. Each group of "derivatives" is integrated into a higher class motion model using linear least squares estimation. The final motion model is then clustered as a vector in motion model space using k-means clustering techniques. Derivatives are classified based on which cluster is best suited for them. The cluster labels are then spatially clustered as the evolution of the spatial partition. The process continues until the partitioning is stable.

In a further embodiment of the invention, the motion vectors of a given aperture are interpolated with a set of pixel positions corresponding to the aperture. If a block defined by such interpolation extends to a pixel corresponding to an object boundary, the final classification is a predetermined anomalous diagonal division of the block.

In the prior art, the least squares estimator used to integrate the derivatives is very sensitive to outliers. Sensitivity can generate a motion model that biases the motion modeling clustering method for points with widely divergent iterations.

In the present invention, the motion segmentation method identifies spatial discontinuity through analysis of explicit pixel motion for two or more frames of video. Clear motion is analyzed for matches to frames of video and incorporated into a parametric motion model. The spatial discontinuity associated with this constant motion is identified. Motion segmentation can be referred to as temporal segmentation because temporal changes are caused by motion. However, the temporal change may be caused by some other phenomenon such as local deformation, lighting change, and the like.

Through the aforementioned method, the protruding signal mode corresponding to the normalization method can be identified and separated from the surrounding signal mode (background or non-object) through one of several background subtraction methods. Often, this method statistically models the background as pixels representing the minimum amount of change in each time instance. The change can be characterized as a pixel value difference.

Segmented boundary based global deformation modeling can be obtained by creating a boundary around an object and then collapsing the boundary towards the detected center of the object until the boundary vertices obtain a position consistent with the heterogeneous image tilt. Motion estimates are collected for these new vertex positions, and robust affine estimates are used to find the global deformation model.

Finite differences based on segmented mesh vertex image gradient descent are incorporated into the global deformation model.

Split Object

The block diagram in FIG. 13 shows one preferred embodiment of object segmentation. The process begins with a sum 1302 of normalized images, which are then pair-wise differentiated between the sum. This difference is then accumulated 1306 in an element-wise accumulation buffer. The cumulative buffer is thresholded 1310 to identify more significant error areas. The thresholded element mask is then morphologically analyzed 1312 to determine the spatial support of the accumulated error region 1310. The resulting extraction 1314 of morphological analysis 1312 is then compared 1320 to the detected object position to focus on subsequent processing on the accumulated error area that matches the object. The boundary of the isolated spatial region 1320 is then approximated (1322) using a polygon from which a vertex outside is generated 1324. The outer trajectory is then adjusted 1330 to better initialize the vertex position for active trajectory analysis 1332. Once the active trajectory analysis 1332 has converged to a low energy solution in the accumulated error space, the trajectory is used as the final trajectory 1334 and the pixels bound within the trajectory are considered to be object pixels, and pixels outside of the trajectory are non-objects. Are considered to be pixels.

In a preferred embodiment, the motion segmentation can be lumped given the detected position and scale of the protruding image mode. Distance transformation is used to determine the distance of all pixels from the detected position. If the pixel value associated with the maximum distance is maintained, a reasonable model of the background can be solved. In other words, the ambient signal is temporarily resampled using the signal difference metric.

Additional embodiments include using a distance transform for the current detection position to assign a distance to each pixel. If the distance to the pixel is greater than the distance in the predetermined maximum pixel distance table, the pixel value is recorded. After an appropriate training period, if the maximum distance to this pixel is large, it is assumed that the pixel has the greatest likelihood of being a background pixel.

Given a model of the ambient signal, the complete salient signal mode at each moment is differentiated. Each such difference can be resampled into a spatially normalized signal difference (absolute difference). These differences are then aligned and accumulated with respect to each other. Since these differences are spatially normalized with respect to the protruding signal mode, the peak of the difference will most likely correspond to the pixel position associated with the protruding signal mode.

In one embodiment of the present invention, a training period is defined, where the object detection positions are determined and the center of these positions allows such frame differences to produce background pixels with the greatest likelihood of becoming non-object pixels. It is used to determine the optimal number of frames using the detection position spaced from the.

In one embodiment of the invention, active trajectory modeling is used to segment the foreground object from the non-object background by determining the trajectory vertex position in the accumulated error “image”. In a preferred embodiment, the active trajectory edges are subdivided to be the same size as the scale of the detected object to create a greater degree of freedom. In a preferred embodiment, the final trajectory position is snapped to the nearest regular mesh vertices to produce regularly spaced trajectories.

In one non-limiting embodiment of object segmentation, an oriented kernel is used to generate an error image filter response for a temporarily bidirectional image. The response to the filter oriented perpendicular to the direction of motion of the gross tends to increase the error surface when motion to the background results from occlusion and leakage of the background.

The normalized image frame intensity vector for the sum of the normalized images is differentiated from one or more reference frames producing a residual vector. This residual vector is accumulated in the element direction to form the accumulated residual vector. This accumulated residual vector is then probed spatially to define a spatial object boundary for spatial division of object and non-object pixels.

In one preferred embodiment, an initial statistical analysis of the accumulated residual vector is performed to obtain a statistical threshold that can be used to set the threshold of the accumulated residual vector. Through erosion and subsequent dilation morphological operations, a preliminary object region mask is created. The trajectory polygon points of the area are then analyzed to reveal the convex hull of these points. The convex hull is then used as the first trajectory for the active trajectory analysis method. The active trajectory propagates until it converges to the spatial boundaries of the accumulated residue of the object. In a further preferred embodiment, the edges of the preliminary trajectory are further refined by adding midpoint vertices until a minimum edge length is obtained for all edge lengths. This additional embodiment is designed to increase the degree of freedom of the active trajectory model to more accurately align the outline of the object.

In a preferred embodiment, an improved trajectory is used to create a pixel mask representing the pixels of the object by overwriting the polygons implied by the trajectory and by overwriting the polygons in the normalized image.

Non-object  Resolution

The block diagram shown in FIG. 12 discloses one preferred embodiment of non-object segmentation, or equivalently, background decomposition. Using the initialization 1206 of the background buffer and the initial maximum distance value 1204, the process operates to determine the most stable non-object pixel by associating "stability" with the longest distance from the detected object location 1202. do. Given the newly detected object position 1202, the process checks each pixel position (1210). For each pixel location 1210, the distance from the detected object location 1202 is calculated using the distance transform. If the distance to this pixel is greater than the pre-stored position in the maximum distance buffer 1204 (1216), the previous value is replaced by the current value 1218 and the pixel value is written to the pixel buffer (1220).

Given a decomposed background image, the error between the image and the current frame can be spatially normalized and accumulated in time. This decomposed background image is described in the "Background Decomposition" section. Background decomposition through this method is thought to be a time-based occlusion filter process.

The resulting accumulated error is then thresholded to provide the first trajectory. The trajectory is then propagated spatially to the balance error residue against trajectory deformation.

In an alternative embodiment, an absolute difference between the current frame and the resolved background frames is calculated. The absolute difference of the element scheme is then divided into distinct spatial regions. The box mean pixel value surrounding this area is calculated so that when the decomposed background is updated, the difference between the current and decomposed background average pixel values can be used to perform the constraint shift, so that the current area is more than the decomposed background. Can be mixed effectively. In another embodiment, the vertices in the normalized frame mask are motion estimated and stored for each frame. These are then processed using SVD to generate local distortion prediction for each frame.

Slope Split

The texture segmentation method, or equivalently intensity gradient segmentation, analyzes the local gradient of pixels in one or more frames of the video. Gradient response is a statistical measure that characterizes the spatial discontinuities local to pixel locations within a video frame. One of several spatial clustering techniques is then used to combine the gradient response into the spatial domain. The boundaries of these regions are useful for identifying spatial discontinuities in one or more video frames.

In one embodiment of the present invention, the summed region table concept from computer graphics texture generation is used for the purpose of facilitating the calculation of the slope of the intensity field. Four lookups combined with four additional operations produce a field of progressively summed values that facilitates the sum of a given rectangle of the original field.

A further embodiment utilizes a Harris response generated for the image, with pixels neighboring each pixel being classified into uniform planes, edges or corners. The response value is generated from this information and indicates the degree of edge state or corner state for each element in the frame.

Multiscale Slope Analysis

One embodiment of the present invention further enforces image tilt support by generating image tilt values over several spatial scales. This method can help to adapt the image gradient so that spatial discontinuities at different scales are used to support each other-as long as the "edge" is identified at several different spatial scales, the edges are "projected" Should be More adapted image inclinations tend to correspond to more prominent features.

In a preferred embodiment, a text response field is generated first, and the value of this field is then adapted to several bins based on k-means binning / partitioning. The original image gradient values are then processed progressively using each bin as an interval of values at which one iteration can apply branch segmentation. The advantage of this approach is that it is defined in terms of relative uniformity for strong spatial bias.

Spectral segmentation

The spectral segmentation method analyzes the statistical probability distribution of black and white, gray scale, or color pixels in video pixels. The spectral classifier is constructed by performing a clustering operation on the probability distribution of these pixels. The classifier is then used to classify one or more pixels as belonging to a probability class. The final probability class and its pixels are given a class label. This class label is then spatially related to the region of the pixel having a distinct boundary. This boundary identifies the spatial discontinuity of one or more video frames.

The present invention can use spatial segmentation based on spectral classification to segment the pixels in a frame of video. Moreover, the correspondence between the regions can be determined based on the overlap of the spectral regions with regions in the preceding segment.

If the video frame is roughly composed of contiguous color regions spatially connected to larger regions corresponding to objects in the video frame, identification and tracking of colored (or spectral) regions facilitates subsequent segmentation of objects in the video sequence. It can be done.

Background split

The disclosed invention includes a method for video frame background modeling based on a temporal maximum of a spatial distance measurement between a detected object and each individual pixel in each frame of a video. Given the detected position of the object, the distance transform is applied to produce a scalar distance value for each pixel in the frame. For each pixel, a map of the maximum distance for all video frames is maintained. When the maximum value is assigned for the first time or later updated with a new and different value, the corresponding pixel for this video frame remains in the "decomposed background" frame.

Appearance modeling

A common purpose of video processing is often to model and maintain the appearance of a sequence of video frames. It is an object of the present invention to enable a limited appearance modeling technique that is applied in a robust and widely applicable manner through the use of processing. The above registration, partitioning, and normalization are described for this purpose.

The present invention discloses a means of appearance change modeling. In the case of a linear model, the main basis for modeling the appearance change is the analysis of feature vectors representing a compact basis using linear correlation. The feature vector representing the spatial intensity field pixel may be assembled into an appearance change model.

In an alternative embodiment, the appearance change model is calculated from the divided subset of pixels. Moreover, the feature vector can be spatially divided into non-overlapping feature vectors. Such spatial decomposition may be achieved with spatial tiling. Computational efficiency may further be achieved through processing such temporal ensemble without sacrificing the dimensional reduction of the wider PCA method.

When generating an appearance change model, spatial intensity field normalization can be used to reduce PCA modeling of space change.

transform modelling

Local deformation can be modeled as vertex displacement and an interpolation function can be used to determine the resampling of a pixel according to the vertices associated with these pixels. This vertex displacement can provide a large amount of deformation in motion when found as a set of parameters across many vertices. Correlation in these parameters can greatly reduce the dimensionality of this parameter space.

PCA

A preferred means of generating an appearance change model is to assemble a frame of video as a pattern vector into a training matrix, or ensemble, and an application of critical component analysis (PCA) to the training matrix. If this extension is omitted, the final PCA transform matrix is used to analyze and synthesize subsequent frames of video. Based on the level of omission, a varying level of quality of the original appearance of the pixel can be obtained.

Specific means of construction and decomposition of the pattern vector are well known to those skilled in the art.

Given the spatial division of the protruding signal mode and the spatial normalization of this mode from the ambient signal, the appearance of the pixel itself, or an equivalent final normalized signal, results in a direct exchange between the bit rate and the approximation error for the representation of the pixel appearance. It can be factored into linearly correlated components using low rank parameterization that allows. One way to obtain a low rank approximation is through truncation of bytes / bits of encoded data. A low rank approximation is considered that the compression of the original data is determined by the specific application of this technique. For example, in video compression, if the truncation of the data does not excessively degrade the perceptible quality, an application specific goal may be achieved according to the compression.

As shown in FIG. 2, to produce a concise version 252 & 254 in the size of the data, the normalized object pixels 242 & 244 are projected into vector space and a linear correspondence can be obtained using a decomposition process 250 such as PCA. Can be modeled.

continuity PCA

PCA uses a PCA transform to encode the pattern into PCA coefficients. If the pattern is better represented by a PCA transform, less coefficients are needed to encode the pattern. Recognizing that a pattern vector may deteriorate over time between acquisition of a training pattern and acquisition of a pattern to be encoded, updating the transformation helps to counteract the degradation. As an alternative to creating a new transformation, continuous updating of the current pattern is in some cases more computationally efficient.

Many modern video compression algorithms predict frames of video from one or more other frames. The prediction model is commonly based on partitioning each expected frame into related transform positions parameterized by non-overlapping tiles and offset motion vectors matched with corresponding patches of other frames. This spatial displacement, optionally combined with the frame index, provides a "motion predicted" version of the tile. If the error of prediction is below a certain threshold, the pixels of the tile are suitable for the remaining encodings; There is a corresponding gain in compression efficiency. Otherwise, the pixels of the tile are encoded directly. This type of tile-based-alternatively referred to as block-based-motion prediction method models video by interpreting tiles comprising pixels. If imaged events in the video support this type of modeling, the corresponding encoding efficiency increases. This modeling constraint assumes that a given level of temporal decomposition, or number of frames per second, is provided for an imaged object undergoing motion in order to follow the analytical assumptions inherent in block-based prediction. Another requirement for this analytical model is that the spatial displacement for any temporal decomposition must be limited; That is, the time difference between the frame to which the prediction is derived and the frame to be predicted should be a relatively short amount of absolute time. This temporal decomposition and motion constraints facilitate the identification and modeling of any excess video signal components present in the video stream.

In the method of the present invention, continuous PCA is combined with an embedded zero-tree wavelet to further increase the use of the hybrid compression method. The continuous PCA technique provides a means by which conventional PCAs can be enhanced for signals with temporary coherency or temporarily local smoothness. Built-in zero-tree wavelets provide a means by which local smooth spatial signals can be decomposed into spatial-scale representations to increase the robustness of specific processing and also increase the computational efficiency of the algorithm. For the present invention, these two techniques combine to increase the expressiveness of the deformation model and also provide a representation of these models that are concise and aligned such that much of the representational power of the basis is provided by the cleavage of the basis.

In another embodiment, continuous PCA is applied to a fixed input block size and a fixed tolerance to increase the weight bias on the first most powerful PCA elements. For longer data sequences, these first PCA elements are often only PCA elements. This affects the visual quality of the reconstruction and may limit the use of the approach in some of the ways described. The present invention uses different standards for the selection of PCA elements that are desirable for the use of the least square standard used in the prior art. This type of model selection avoids transient approximation by the original PCA element.

In another embodiment, a PCA process with a fixed input block size and a prescribed number of PCA element blocks per data block is used to provide a useful uniform reconstruction that is exchanged for using a relatively large number of elements. In a further embodiment, the block PCA is used in combination with the continuous PCA, and the block PCA restarts the continuous PCA after a set of steps including a block PCA step. This provides a useful and uniform approximation with a reduction in the number of PCA elements.

In another embodiment, the present invention utilizes a situation where the PCA elements are visually similar before and after encoding-decoding. The quality of the image sequence reconstruction before and after encoding-decoding is also similar, which often depends on the degree of quantization used. The method of the present invention decodes PCA elements and then renormalizes them to have a unit standard. For proper quantification, the decoded PCA elements are approximately vertical. At higher levels of quantization, the decoded PCA elements are partially recovered by the application of the SVD to obtain orthogonal basis and modified reconstruction coefficients.

In another embodiment, variable and adaptable block sizes are applied with the hybrid continuous PCA method to obtain improved results with respect to synthesis quality. The present invention is based on the block size in the maximum number of PCA elements and given errors for these blocks. The method then extends the current block size until the maximum number of PCA elements is reached. In a further embodiment, the sequence of PCA elements is considered a data stream, which causes an additional reduction in dimension. The method performs a post-processing step wherein variable data blocks are collected from each block for the first PCA element and SVD is applied to further reduce the dimension. The same process then applies to the collection of the second, third, etc. elements.

Symmetric decomposition

In one embodiment of the present invention, decomposition is performed based on a symmetric ensemble. This ensemble represents the square image as the sum of six orthogonal elements. Each element corresponds to a different symmetry of the square. By symmetry, each orthogonal element is determined by a "basal region" that is mapped to a complete element by the act of symmetry. The sum of the base regions has the same cardinality as the input image, assuming that the input image itself does not have any particular symmetry.

Rest based decomposition

In MPEG video compression, the current frame consists of a motion that compensates for the previous frame using a motion vector, followed by the application of the remaining updates to the compensation block, and finally any block that does not have sufficient matching is new. It is encoded as a block.

The pixels corresponding to the remaining blocks are mapped to the pixels of the previous frame through the motion vectors. The result is the temporal path of the pixel through the video that can be synthesized through a continuous application of the remaining values. These pixels are identified as pixels that can be best represented using PCA.

Occlusion-based decomposition ( Acclusion -based Decomposition)

A further extension of the present invention determines whether the motion vector provided in the block causes the previous frame to be occluded (covered) by moving the pixels. For each occlusion event, occlusion pixels are divided into new layers. There are also exposed pixels without history. The exposed pixels fit them in the current frame, and are placed in a given layer where historical conformance is to be made for that layer.

Temporal continuity of pixels is supported through division and combining of pixels into different layers. Once a stable layer model is formed, the pixels of each layer can be grouped based on membership to the coherent motion model.

Subband Time Quantization

An alternative embodiment of the present invention uses Discrete Cosine Transform (DCT) or Discrete Wavelet Transform (DWT) to decompose each frame into subband images. Critical Component Analysis (PCA) is then applied to each such "sub-band" video. The concept is that subband decomposition of a video frame reduces spatial variation in one of the subbands compared to the original video frame.

In the case of video of moving objects (persons), spatial changes tend to dominate the changes modeled by the PCA. Subband decomposition reduces the spatial variation in any one decomposition video.

In the case of DCT, the decomposition coefficients for any one subband are spatially arranged into subband video. For example, the DCT coefficients are taken from each block and ordered into subband video that looks like a stamped version of the original video. This is repeated for all other subbands, and the final subband video is each processed using the PCA.

In the case of DWT, the subbands are already aligned in the manner described for DCT.

In a non-limiting embodiment, the omission of the PCA coefficients is varied.

Wavelet

When data is decomposed using Discrete Wavelet Transform (DWT), multiple band pass data sets result in lower spatial resolution. Until only a single scalar value is obtained, the transformation process can be applied to the repeatedly derived data. Scalar elements of the decomposed structure are typically related in a hierarchical parent / child manner. The final data includes multiresolution hierarchy and finite differences.

When DWT is applied to the spatial intensity field, many naturally occurring image events are represented by small perceptual losses by the first or second low pass induced data structures due to the low spatial frequency. Omission of the hierarchical structure provides a compact representation unless high frequency spatial data is provided or considered noise.

While PCA can be used to achieve accurate adaptation with a small number of coefficients, the transformation itself can be very large. To reduce the size of this "initial" transform, the built-in zero tree (EZT) structure of wavelet decomposition can be used to build progressively more sophisticated versions of the transform matrix.

Subspace classification

As can be well understood by those skilled in the art, the discrete sampling event data and the derived data can be represented by a set of data vectors corresponding to the algebraic vector space. Such data vectors include, in a non-limiting manner, pixels in the normalized appearance of the segmented object, motion parameters, and the structural location of two- or three-dimensional features or vertices. Each of these vectors is in vector space, and an analysis of the geometry of the space can be used to produce a concise representation of the sampled or parametric vector. Useful geometry states are typed by parameter vectors that form a compact subspace. When one or more subspaces are mixed to create subspaces that are more complex on the surface, the subspaces of the components can be difficult to identify. There are several partitioning methods that enable the separation of these subspaces by examining the data in a high dimensional vector space created through some interaction (eg, dot product) of the original vectors.

One way of partitioning the vector space involves projecting the vector into a Veronese vector space representing a polynomial. This method is well known in the art as generalized PCA or GPCA. With this projection, the standards for polynomials are found and grouped, and the original vectors associated with these standards can be grouped together. An example use of this technique is to factor the two-dimensional spatial point correspondence tracked over time into a three-dimensional structural model and the motion of this three-dimensional model.

The GPCA technique is incomplete when applied as defined and results only when the data vector is generated with less noise. The prior art takes supervised user intervention to guide the GPCA algorithm. This constraint greatly limits the potential of the present technology.

The present invention extends the conceptual basis of the GPCA method to robustly handle the identification and segmentation of multiple subspaces in the presence of noise and common dimensions. This innovation provides an unsupervised improvement of the technology over the state of the art.

In the prior art, GPCA operates on the normal vectors of the polynomial of the Veronese map, irrespective of the tangent space of these normal vectors. The method of the present invention extends the GPCA to find the tangent space orthogonal to the space of the normal vector typically found in the Veronese map. This "tangent space", or subspace of the Veronese map, is then used to factor the Veronese map.

Tangent space is identified through the application of the Regendre transformation between the position and tangent plane axes and the planar wave extension, which is the duality in the representation of geometric objects, specifically the normal to the polynomial of the Veronese map. Expose the tangent. Discrete Rejangre transformation is applied through convex analysis to define the constrained form of the derivative corresponding to the normal vector. This approach is used to segment the data vector by calculation of the normal vector in the presence of noise. This convex analysis is integrated with GPCA to provide a more robust algorithm.

The present invention uses an iterative factorization approach when applying GPCA. In particular, the derivative based implementations found in the prior art are extended to purify the sum of the data vectors sorted through the very same GPCA described herein. When applied repeatedly, this technique can be used to robustly search for candidate normal vectors in Veronese mapping, and then adapt these vectors using an extended GPCA technique. For the factorization step, original data associated with the refined set of vectors is removed from the original data set. The rest of the dataset can be similarly analyzed with an innovative GPCA technique. This innovation is important for using GPCA algorithms in an unsupervised way. 11 shows iterative refinement of data vectors.

The extension of the invention to the GPCA technique is even more advantageous when there are multiple roots in the Veronese polynomial vector space. In addition, the prior art techniques suffer from cases where the normals in the Veronese map deteriorate when parallel to the vector space axis, but the method does not deteriorate.

10 illustrates a method of basic polynomial fitting and differentiation.

In a preferred embodiment, the GPCA is implemented with polynomial derivatives for any common co-dimension subspace. SVD is used to obtain the normal spatial dimension at each data point and cluster data point along the normal spatial dimension. Within each cluster, data points are assigned to the same subspace when they all belong to the maximum set having the same rank as the common normal spatial dimension. It is recognized that this method is optimal for noise free data.

Another non-limiting embodiment of GPCA using polynomial derivatives has any common dimensional subspace. This is an adaptation of the "polynomial differential" method. Since noise tends to increase the rank of a set of closely aligned normal vectors, the division step of the polynomial clusters the data points along the SVD dimension and then has the smallest residual error in the cluster with the smallest common dimension. It is initialized by selecting a point. The normal space at this point is then applied using polynomial partitioning to approximately reduce the Veronese map.

In a further embodiment, the slope weighted residual error is minimized for all data points, and SVD is applied at the optimal point to estimate common dimensions and base vectors. The basis vector is then applied using polynomial partitioning to approximately reduce the Veronese map.

In a preferred embodiment, the RCOP error is used to set the numerical tolerance due to scaling the numerical tolerance linearly with the noise level. In a preferred embodiment, the GPCA is implemented in this way to apply an SVD to the normal vectors estimated at each point and identify the points where the normal vector SVD has the same rank. Continuous SVD is then applied to each collection of normal vectors at points with the same rank. The points at which consecutive SVDs change rank are identified as different subspaces.

Hybrid space normalization compression

The present invention extends the efficiency of a block-based motion predicted coding scheme through the addition of dividing a video stream into two or more "normalized" streams. These streams are then individually encoded to make the interpretation motion assumptions of the conventional codec valid. When decoding a normalized stream, the streams are denormalized to their proper location and synthesized with each other to produce the original video sequence.

In one embodiment, one or more objects are detected in the video stream and the pixels associated with each individual object are subsequently split, leaving non-object pixels. Next, a global spatial motion model is created for objects and non-objects. The global model is used to spatially normalize object and non-object pixels. This normalization effectively removed non-translational motion from the video stream and provided a set of videos with minimal occlusion interactions. These are all useful features of the method of the invention.

New video of objects and non-objects with spatially normalized pixels is provided as input to conventional block-based compression algorithms. When decoding video, global motion model parameters are used to denormalize the decoded frame, and the object pixels are mixed with each other and mixed on non-object pixels to produce an approximation of the original video stream.

As shown in FIG. 6, previously detected object instances 206 & 208 for one or more objects 630 & 650 are each processed into separate instances of the conventional video compression method 632. Additionally, non-objects 602 resulting from segmentation 230 of objects are also compressed using conventional video compression 632. The result of each such individual compressed coding 632 is a separate conventional encoded stream 634 for each corresponding to each video stream individually. At some point, possibly after transmission, this intermediate encoded stream 234 may be decompressed 636 into a normalized non-object synthesis 610 and a number of objects 638 & 658. These synthesized pixels can be denormalized 640 to their denormalized versions 622, 642 & 662, so that the compositing process 670 can combine the object and non-object pixels to form a composite 672 of the entire frame. Position the pixels precisely spatially with respect to each other.

In a preferred embodiment, the switching between encoding modes is performed based on a statistical distortion metric, for example based on the PSNR, which would allow a conventional versus subspace method to encode frames of video.

In another embodiment of the invention, the encoded parameters of the appearance, global transformation, and local transformation are interpolated to produce a prediction of the intermediate frames that should not be encoded. The interpolation method may be any of standard interpolation methods such as linear, cubic, and spline.

As shown in FIG. 14, the object interpolation method may be obtained through interpolation analysis 1408 of a series of normalized objects 1402, 1404, & 1406 when represented by appearance and deformation parameters. The analysis determines the temporary range 1410 to which the interpolation function can be applied. Range details 1410 may then be combined with normalized object details 1414 & 1420 to approximate and finally synthesize intermediate normalized objects 1416 & 1418.

hybrid  Codec integration

In combining the conventional block-based compression algorithm with the normalization-splitting scheme as described in the present invention, there are several resulting inventive methods. Primarily there are specialized data structures and required communication protocols.

Main data structures include global spatial transformation parameters and object segmentation specification masks. The main communication protocols are layers that contain the transmission of global spatial transformation parameters and object segmentation specification masks.

Claims (26)

A computer implemented method of generating an encoded form of video signal data from a plurality of video frames, the method comprising: Detecting one or more objects in two or more video frames; Tracking the one or more objects through the two or more frames of the video frames; Identifying corresponding elements of one or more objects in the two or more video frames; Analyzing the corresponding elements to create relationships between the corresponding elements; Generating a correspondence model using the relationships between the corresponding elements; Resampling pixel data associated with the one or more objects in the two or more video frames using the correspondence model and thereby generating resampled pixel data, wherein the resampled pixel data comprises: Indicates a first intermediate form of data; And Restoring a spatial position of the resampled pixel data using the correspondence model, thereby generating a reconstructed pixel, No detection dictates indirect detection for the entire frame, The detecting and tracking step includes using a Viola / Jones face detection algorithm, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. A computer implemented method of generating an encoded form of video signal data from a plurality of video frames, the method comprising: Detecting one or more objects in two or more video frames; Tracking the one or more objects through the two or more frames of the video frames; Dividing pixel data associated with the one or more objects from other pixel data in the two or more video frames to produce a second intermediate form of data, wherein the partitioning uses spatial partitioning of the pixel data; Identifying corresponding elements of one or more objects in the two or more video frames; Analyzing the corresponding elements to create relationships between the corresponding elements; Generating a correspondence model using the relationships between the corresponding elements; Integrating the relationships between the corresponding elements into a model of global motion; Resampling pixel data associated with the one or more objects in the two or more video frames using the correspondence model and thereby generating resampled pixel data, wherein the resampled pixel data comprises: Indicates a first intermediate form of data; Restoring a spatial position of the resampled pixel data using the correspondence model, thereby generating a reconstructed pixel; And Recombining the reconstructed pixels with an associated portion of the second intermediate form of the data to produce an original video frame, No detection dictates indirect detection for the entire frame, The detecting and tracking step includes using a face detection algorithm, Generating the correspondence model includes using a robust estimator for a solution of a multidimensional projecting motion model, Analyzing the corresponding elements comprises using appearance based motion estimation between the two or more video frames; Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 1, Dividing the pixel data associated with the one or more objects from other pixel data in the two or more video frames to produce a second intermediate form of the data, wherein the partitioning uses temporal integration; And Reassembling the reconstructed pixel with the associated portion of the second intermediate form of data to produce an original video frame, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 1, comprising a method of factoring the correspondence model into a global model, the method comprising: Incorporating relationships between the corresponding elements into a model of global motion, Generating the correspondence model includes using robust sampling consensus for a solution of a two-dimensional affine motion model, Analyzing the corresponding element comprises using a sampling population based on finite differences generated from block-based motion estimation between two or more of the video frames; Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 1, comprising encoding a first intermediate form of the data, the encoding comprising: Decomposing the resampled pixel data into an encoded representation, the encoded representation representing a third intermediate form of the data; Truncating zero bytes or more from the encoded representation; And Reconstructing the resampled pixel data from the encoded representation, The decomposition and reconstitution step using the principal component analysis (Principle Component Analysis), Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 3, comprising a method of factoring the correspondence models into global models, the method comprising: Integrating the relationships between the corresponding elements into a model of global motion; Decomposing the resampled pixel data into an encoded representation, the encoded representation representing a fourth intermediate form of the data; Truncating at least zero bytes of the encoded representation; And Reconstructing the resampled pixel data from the encoded representation, The decomposition and reconstitution step uses a principal component analysis (Principle Component Analysis), Generating the correspondence models includes using robust sampling consensus for a solution of a two-dimensional affine motion model, Analyzing the corresponding elements comprises using a sampling population based on finite differences generated from block based motion estimation between two or more of the video frames; Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 6, wherein each of the two or more video frames comprises object pixels and non-object pixels, the method comprising: Identifying corresponding elements in non-object pixels in the two or more video frames; Analyzing corresponding elements in the non-object pixels to create a relationship between corresponding elements in the non-object pixels; Generating second correspondence models using the relationship between corresponding elements in the non-object pixels; The analysis of the corresponding elements comprises a time based occlusion filter, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 7, wherein Factoring the corresponding models into global models; Integrating the relationships between the corresponding elements into a model of global motion; Decomposing the resampled pixel data into an encoded representation, the encoded representation representing a fifth intermediate form of the data; Truncating zero bytes or more from the encoded representation; And Reconstructing the resampled pixel data from the encoded representation, Each of the decomposition and reconstruction steps utilizes a conventional video compression / decomposition process, Generating the correspondence model includes using robust sampling consensus for a solution of a two-dimensional affine motion model, Analyzing the corresponding element comprises using a sampling population based on finite differences generated from block-based motion estimation between two or more of the video frames; Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. A computer-implemented method of separating data vectors present in discrete linear subspaces, (a) performing subspace segmentation on the data vectors; And (b) forcing subspace segmentation criteria through the application of tangent vector analysis to the implicit vector space, Performing the subspace partitioning comprises using GPCA; The implicit vector space includes a Veronese Map; The tangent vector analysis includes a Regendre transformation, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 9, Holding a subset of the set of data vectors; Performing (a) and (b) on a subset of the set of data vectors, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 5, (a) performing subspace partitioning on the first intermediate form of the data; (b) forcing subspace partitioning criteria through the application of tangent vector analysis to the implicit vector space; Retaining a subset of the first intermediate form of the data; Performing (a) and (b) on a subset of said first intermediate form of said data, Performing the subspace partitioning comprises using GPCA; The implicit vector space includes a Veronese Map; The tangent vector analysis includes a Regendre transformation, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. 8. The method of claim 7, comprising a method for factoring the correspondence models into global models, the method comprising: (a) integrating the relationships between the corresponding elements into a model of global motion; (b) performing subspace segmentation on the data vectors, wherein performing the subspace segmentation comprises using GPCA; (c) forcing subspace partitioning criteria through the application of tangent vector analysis to the implicit vector space; (d) holding a subset of the set of data vectors; (e) performing (b) and (c) on the subset of the set of data vectors, wherein the implicit vector space comprises a Veronese map; The tangent vector analysis comprises a Regards transform; After (a) to (e) has been carried out, the method is: (f) decomposing the resampled pixel data into an encoded representation, the encoded representation representing a fourth intermediate form of the data; (g) truncating at least zero bytes of the encoded representation; And (h) reconstructing the resampled pixel data from the encoded representation, The decomposition and reconstitution step uses a principal component analysis (Principle Component Analysis), Generating the correspondence model includes using robust sampling consensus for a solution of a two-dimensional affine motion model, Analyzing the corresponding elements comprises using a sampling population based on a finite difference generated from block based motion estimation between the two or more video frames; Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 1, comprising factoring the correspondence models into local deformation models, the method comprising: Defining a two-dimensional mesh on top of pixels corresponding to one or more objects, the mesh based on a regular grid of vertices and edges; And Generating a model of local motion from the relationships between the corresponding elements, wherein the relationships include vertex displacements based on finite differences generated from block based motion estimation between two or more of the video frames. , Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 13, wherein the vertex corresponds to discrete image features, and the method includes identifying salient image features corresponding to the object using analysis of an image gradient Harris response. Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 4, wherein Forwarding a first intermediate form of the data to be factored into a local deformation model; Defining a two-dimensional mesh on top of pixels corresponding to one or more objects, the mesh based on a regular grid of vertices and edges; And Generating a model of local motion from the relationships between the corresponding elements, wherein the relationships include vertex displacements based on finite differences generated from block based motion estimation between two or more of the video frames. , Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 6, Sending a fourth intermediate form of data to be factored into a local deformation model; Defining a two-dimensional mesh on top of pixels corresponding to one or more objects, the mesh based on a regular grid of vertices and edges; And Generating a model of local motion from the relationships between the corresponding elements, wherein the relationships include vertex displacements based on finite differences generated from block based motion estimation between two or more of the video frames. , The local motion model is based on residual motion not approximated by the global motion model, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 12, Sending a fourth intermediate form of data to be factored into a local deformation model; Defining a two-dimensional mesh on top of pixels corresponding to one or more objects, the mesh based on a regular grid of vertices and edges; And Generating a model of local motion from the relationships between the corresponding elements, wherein the relationships include vertex displacements based on finite differences generated from block based motion estimation between two or more of the video frames. , The local motion model is based on residual motion not approximated by the global motion model, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 2, comprising encoding the first intermediate form of the data, the encoding comprising: Decomposing the resampled pixel data into an encoded representation, the encoded representation representing a third intermediate form of the data; Truncating at least zero bytes of the encoded representation; And Reconstructing the resampled pixel data from the encoded representation, Each of the decomposition and reconstitution steps uses principal component analysis, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 2, comprising a method of factoring the correspondence model into a global model, the method comprising: Integrating the relationships between the corresponding elements into a model of global motion; Truncating at least zero bytes of the encoded representation; Reconstructing the resampled pixel data from the encoded representation, Each of said decomposition and reconstitution steps uses principal component analysis, Generating the responsiveness model includes using a robust estimator for the solution of the multidimensional projecting motion model, Analyzing the corresponding elements comprises using a sampling population based on a finite difference generated from block-based motion estimation between the two or more video frames. Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 19, wherein each of the two or more video frames comprises an object pixel and a non-object pixel, the method comprising: Identifying corresponding elements in non-object pixels in the two or more video frames; Analyzing corresponding elements in the non-object pixels to create a relationship between corresponding elements in the non-object pixels; Generating second correspondence models using the relationship between corresponding elements in the non-object pixels; The analysis of the corresponding elements comprises a time based occlusion filter, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 20, Factoring the corresponding models into global models; Decomposing the resampled pixel data into an encoded representation, the encoded representation representing a fifth intermediate form of the data; Truncating at least zero bytes of the encoded representation; And Reconstructing the resampled pixel data from the encoded representation, Each of the decomposition and reconstruction steps utilizes a conventional video compression / decomposition process, Generating the correspondence model includes using a robust estimator for a solution of a multidimensional projecting motion model, Analyzing the corresponding element comprises using a sampling population based on finite differences generated from block-based motion estimation between two or more of the video frames; Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 20, comprising a method of factoring the correspondence models into global models, the method comprising: (a) integrating the relationships between the corresponding elements into a model of global motion; (b) performing subspace segmentation on the data vectors, wherein performing the subspace segmentation comprises using GPCA; (c) forcing subspace partitioning criteria through the application of tangent vector analysis to the implicit vector space; (d) holding a subset of the set of data vectors; (e) performing (b) and (c) on the subset of the set of data vectors, wherein the implicit vector space comprises a Veronese map; The tangent vector analysis comprises a Regards transform; After (a) to (e) has been carried out, the method is: (f) decomposing the resampled pixel data into an encoded representation, the encoded representation representing a fourth intermediate form of the data; (g) truncating at least zero bytes of the encoded representation; And (h) reconstructing the resampled pixel data from the encoded representation, The decomposition and reconstitution step uses a principal component analysis (Principle Component Analysis), Generating the correspondence model includes using a robust estimator for a solution of a multidimensional projecting motion model, Analyzing the corresponding elements comprises using a sampling population based on a finite difference generated from block based motion estimation between the two or more video frames; Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 2, comprising a method for factoring the correspondence models into local deformation models, the method comprising: Defining a two-dimensional mesh on top of pixels corresponding to one or more objects, the mesh based on a regular grid of vertices and edges; And Generating a model of local motion from the relationships between the corresponding elements, wherein the relationships include vertex displacements based on finite differences generated from block based motion estimation between two or more of the video frames. , Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 22, wherein the vertex corresponds to discrete image features, and the method includes identifying salient image features corresponding to the object using analysis of image density gradients. Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 19, Forwarding a fourth intermediate form of the data to be factored into a local deformation model; Defining a two-dimensional mesh on top of pixels corresponding to one or more objects, the mesh based on a regular grid of vertices and edges; And Generating a model of local motion from the relationships between the corresponding elements, wherein the relationships include vertex displacements based on finite differences generated from block based motion estimation between two or more of the video frames. , The local motion model is based on residual motion not approximated by the global motion model, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames. The method of claim 23, wherein Sending a fourth intermediate form of data to be factored into a local deformation model; Defining a two-dimensional mesh on top of pixels corresponding to one or more objects, the mesh based on a regular grid of vertices and edges; And Generating a model of local motion from the relationships between the corresponding elements, wherein the relationships include vertex displacements based on finite differences generated from block based motion estimation between two or more of the video frames. , The local motion model is based on residual motion not approximated by the global motion model, Computer-generated method for generating an encoded form of video signal data from a plurality of video frames.

Images