KR20070086350A - Apparatus and method for processing video data - Google Patents

Abstract

An apparatus and methods for processing video data are described. The invention provides a representation of video data that can be used to assess agreement between the data and a fitting model for a particular parameterization of the data. This allows the comparison of different parameterization techniques and the selection of the optimum one for continued video processing of the particular data. The representation can be utilized in intermediate form as part of a larger process or as a feedback mechanism for processing video data. When utilized in its intermediate form, the invention can be used in processes for storage, enhancement, refinement, feature extraction, compression, coding, and transmission of video data. The invention serves to extract salient information in a robust and efficient manner while addressing the problems typically associated with video data sources.

Description

Apparatus and method for processing video data {APPARATUS AND METHOD FOR PROCESSING VIDEO DATA}

This application is directed to US Patent 60 / 628,861 "System and Method For Video Compression Employing Principal Component Analysis", filed Nov. 17, 2004, and US Patent 60 / 628,819, "Apparatus and Methods for, filed November 17, 2004." Processing and Coding Video Data. " This application is partly filed in US Patent 11 / 191,562 filed July 28, 2005, and partly filed in US Patent 11 / 230,686, filed September 20, 2005. Each of the preceding patents is incorporated herein by reference in its entirety.

FIELD OF THE INVENTION The present invention generally relates to the field of digital signal processing, more particularly to computer or computer-implemented methods for the processing and efficient representation of signal or image data, more particularly video data.

The general system technology in the prior art to which the present invention pertains can be expressed as shown in FIG. 1. Here, the block diagram shows a conventional conventional video processing system. Such systems typically include the following steps: input step 102, processing step 104, output step 106, and one or more data storage mechanism (s) 108.

The input step 102 may include elements such as camera sensors, camera sensor arrays, range finding sensors, or means for retrieving data from the storage mechanism. The input step provides video data representing naturally occurring phenomena and / or artificial time correlated sequences of man-made. Salient components of the data may be masked or masked by noise or other unwanted signals.

Video data in the form of data streams, arrays, or packets may be passed through the intermediate storage element 108 or directly to the processing step 104 in accordance with a predefined transport protocol. Processing step 104 may be dedicated analog or digital devices, or programmable devices such as central processing units (CPUs), digital signal processors (DSPs), or a desired series of video. It may take the form of field programmable gate arrays (FPGAs) for performing data processing operations. Processing step 104 typically includes one or more codecs (coders / decoders).

The output step 106 yields a signal, display, or other response that may affect the user or external device. Typically, the output device is used to generate an indicator signal, display, hard copy, representation of the processed data in storage, or to initiate data transfer to a remote site. The output device can also be used to generate an intermediate signal or control parameter for use in subsequent processing operations.

In this system the reservoir is represented as an optical element. When used, storage element 108 may be nonvolatile, such as read-only storage media, or may be volatile, such as dynamic random access memory (RAM). It is not uncommon for an individual video processing system to include several types of storage elements, which have various relationships with respect to the input, processing, and output stages. Examples of such storage elements include input buffers, output buffers, and processing caches.

The primary purpose of the video processing system of FIG. 1 is to process input data to produce meaningful output for a particular application. To accomplish this goal, various processing operations may be utilized, such as noise reduction or removal, feature extraction, object segmentation and / or normalization, data categorization, event detection, editing, data selection, data re-coding, and transform coding. have.

Many data sources that produce poorly constrained data are important to people, especially sound and visual images. In most cases, the essential features of the source signals conflict with the purpose of efficient data processing. Intrinsic variability of source data is an obstacle to processing the data in a reliable and efficient manner without errors arising from the original empirical and heuristic methods used to derive engineering estimates. The variability is reduced for applications when the input data is naturally or deliberately constrained to a narrowly defined feature set (such as a limited set of symbol values or narrow bandwidth). All of the above constraints cause processing techniques with low commercial value too often.

The design of the signal processing system is influenced by the intended use of the system and the expected characteristics of the source signal used as the input. In most cases, the required performance efficiency can also be a significant design factor. Performance efficiency is affected by the amount of data to be processed compared to the available data storage capacity as well as the computational complexity compared to the available computing power.

Conventional video processing methods suffer from a number of inefficiencies that manifest themselves in the form of slow data communication speeds, large storage capacity requirements, and perceptual artifacts. These are serious problems because of the variety of ways people want to use and manipulate video data and the inherent sensitivity people have for some form of visual information.

An "optimal" video processing system is efficient, reliable, and robust in performing a desired series of processing operations. Such operations may include storage, transmission, display, compression, editing, encryption, enhancement, categorization, feature detection, and data recognition. Second operations include incorporation of the processed data with other information sources. Equally important, in the case of a video processing system, the outputs must be compatible with human vision by preventing the introduction of perceptual artifacts.

A video processing system can be described as "strong" if its speed, efficiency, and quality are not heavily dependent on the specifics of any particular features of the input data. Robustness also relates to the ability to perform actions when some of the input goes wrong. Many video processing systems fail to be robust enough to take into account general grades of applications-providing only the application for the same narrowly constrained data that was used in the development of the system.

Critical information may be lost in discretization of the continuously-evaluated data source due to the sampling rate of the input element that does not match the signal characteristics of the sensed phenomenon. In addition, loss occurs when the strength of the signal exceeds the limits of the sensor, resulting in saturation. Similarly, when the full range of values of the input data is represented as a series of discrete values, information is lost when the accuracy of the input data decreases in any quantization process, thereby causing the data to be lost. The accuracy of the representation is reduced.

Ensemble variability refers to any unpredictability of a class of data or information sources. Data samples of visual information have a very large ensemble variability, because visual information is typically unconstrained. The visual information may represent a sequence of constructions or any spatial array sequence that may be formed by incident light on the sensor array.

In modeling visual phenomena, video processors generally impose some set of structures and / or constraints on the way data is represented or interpreted. As a result, these methods can cause systematic errors that may conflict with the quality of the output, the reliability that allows the output to be evaluated, and the type of subsequent processing tasks that can be reliably performed on the data.

Quantization methods attempt to maintain a statistical variation of the data while at the same time reducing the accuracy of the data of the video frames. Typically, video data is analyzed such that distributions of data values are collected into probability distributions. There are also methods to make the data phase space in order to characterize the data as a mixture of spatial frequencies, whereby the reduction in accuracy can be spread in a less desirable manner. Significantly utilized, the quantization methods often cause perceptually implausible colors and can lead to sudden pixilation in the original smooth regions of the video frame.

In addition, different codings may typically be used to take advantage of local spatial similarity of data. Data in one portion of a frame tends to cluster around similar data in the frame, and also in a similar position in subsequent frames. Representing data by spatially adjacent data can be combined with quantization, and the net result is that representing the differences in the case of some accuracy is more accurate than using absolute values of the data. This assumption fits well when the spectral resolution (resolution) of the original video data, such as black and white video or low-color video, is limited. As the spectral resolution of the video increases, the assumption of similarity breaks down considerably. The breakdown is due to the inability to selectively guarantee the accuracy of the video data.

Residual coding is similar to differential encoding in which errors in the representation are encoded differently to restore the accuracy of the original data to the desired level of precision.

Variations of the methods attempt to transform video data into alternate representations that reveal data correlations in spatial phase and scale. Once the video data is transformed in this manner, quantization and differential coding methods can be applied to the transformed data and the protection of important image characteristics is increased. The two most prevalent transform video compression techniques are discrete cosine transform (DCT) and discrete wavelet transform (DWT). Errors in the DCT transform appear in wide variability of video data values, and therefore DCT is typically used in blocks of video data to localize the false correlations. Artifacts from the localization often appear along the border of blocks. In the case of DWT, more complex artifacts occur when there is a mismatch between the basic function and certain textures, which causes a blurry effect. To counteract the negative effects of DCT and DWT, the accuracy of representation is increased with lower distortion at expensive bandwidth values.

The present invention is a computer-implemented video processing method that provides both computational and analytical advantages over existing conventional methods. The principle of the method according to the invention is the integration of a linear decompositional method, a spatial partitioning method, and a spatial normalization method. Spatially constrained video data greatly increases the robustness and applicability of linear decomposition methods. Additionally, spatial partitioning of data corresponding to spatial normalization can further increase the benefits derived from spatial normalization only.

In particular, the present invention provides a means by which signal data can be efficiently processed into one or more beneficial representations. The present invention is efficient in the processing of many commonly occurring data sets and is particularly efficient in the processing of video and image data. The method according to the invention provides one or more concise representations of the data for analyzing the data and for utilizing processing and encoding.

Each new, more concise data representation allows for reduction of computational processing, transmission bandwidth, and storage requirements for many applications, including but not limited to encoding, compression, transmission, analysis, storage, and display of video data. do. The present invention includes methods for identification and extraction of important components of video data and allows prioritization in processing and presentation of the data. The noise and other unwanted parts of the signal are identified as lower priority so that further processing can be focused on analyzing and representing higher priority parts of the video signal. As a result, the video signal is represented more accurately than previously possible. And the loss of precision is concentrated in parts of the video signal that are less perceptually important.

1 is a block diagram of a video processing system according to the prior art;

2 is a block diagram showing an overview of the present invention showing the main modules for processing video;

3 is a block diagram showing a motion estimation method of the present invention;

4 is a block diagram showing a global registration method of the present invention;

5 is a block diagram showing a normalization method of the present invention;

FIG. 6 is a block diagram illustrating a hybrid spatial normalization compression method.

7 is a block diagram showing a mesh generation method of the present invention used in local normalization;

8 is a block diagram illustrating a mesh-based normalization method of the present invention used in local normalization;

9 is a block diagram illustrating a combined global and regional normalization method of the present invention;

10 is a block diagram illustrating a GPCA-based polynomial binding and differentiation method of the present invention;

11 is a block diagram illustrating a recursive GPCA detailed method of the present invention.

In video signal data, frames of video are assembled into a sequence of images that typically draw the projected and imaged three-dimensional scene onto a two-dimensional imaging surface. Each frame, or image, consists of pixels (pixels) that represent imaging sensor responses to sampled signals. Often, the sampled signal corresponds to the portion that is reflected, refracted, or radiated (e.g., electromagnetic, acoustic, etc.) by the two-dimensional sensor array, and sampled. Successive sequential sampling results in a spatiotemporal data stream having a temporal dimension corresponding to the order of frames in the video sequence and two spatial dimensions per frame.

The present invention as shown in FIG. 2 analyzes the signal data and identifies important components. When the signal consists of video data, analysis of the space-time stream reveals important components that are often specific objects, such as faces. The identification process qualifies the presence and importance of the key components and selects one or more of the most important components from the above defined key components. This does not limit the identification and processing of other components that are less important in parallel or later with the processing described soon. The key components mentioned above are further analyzed to identify variant and invariant subcomponents. Identification of invariant subcomponents is the process of modeling some aspects of the components, which results in parameterization of the model that allows the components to be synthesized to the desired level of precision.

In one embodiment of the invention, a foreground object is detected and tracked. The pixels of the object are identified and segmented from each frame of video. Block-based motion estimation is applied to the divided object in multiple frames. The motion estimates are then integrated into the higher order motion model. A motion model is used to warp instances of the object into a common spatial configuration. In the above configuration, for certain data, further properties of the object are aligned. The normalization allows the linear decomposition of the values of the pixels of objects across multiple frames to be represented densely. Important information suitable for the appearance of the object is included in the compact representation.

A preferred embodiment of the present invention describes in detail the linear decomposition of the foreground video object. The object is spatially normalized, resulting in a compact linear appearance model. Another preferred embodiment further partitions the foreground object from the background of the video frame prior to spatial normalization.

The preferred embodiment of the present invention applies the present invention to video of a person speaking into a camera while experiencing little movement.

The preferred embodiment of the present invention applies the present invention to any object of video that can be well represented through spatial transforms.

A preferred embodiment of the present invention particularly uses block-based motion estimation to determine finite differences between two or more frames of video. The higher order motion model is factored from the finite differences to provide more effective linear decomposition.

Detection & Tracking

Once the constitutive important components of the signal are determined, the components will be retained and all other signal components will be reduced or eliminated. The process of detection of critical components is shown in FIG. 2, where video frame 202 is processed by one or more object detection 206 processes, as a result of which one or more objects are identified and subsequently tracked. The components maintained represent an intermediate form of the video data. The intermediate data may be encoded by techniques not normally available in existing video processing methods. Since intermediate data exists in several forms, standard video encoding techniques can also be used to encode the various intermediate forms. For example, the present invention determines and uses the most efficient encoding technique.

In one preferred embodiment, a saliency analysis process detects and classifies critical signal modes. One embodiment of the process uses a combination of spatial filters specifically designed to generate a response signal whose intensity is proportional to the detected importance of the object in the video frame. The classifier is applied at different positions of the video frame and at different spatial scales. The strength of the response from the classifier indicates the likelihood of the existence of the critical signal mode. When concentrated over strongly important objects, the process classifies them into corresponding strong responses. Detection of critical signal modes characterizes the present invention by enabling subsequent processing and analysis of critical information of the video sequence.

Given the detection location of the critical signal mode in one or more frames of video, the present invention analyzes the invariant characteristics of the critical signal mode. In addition, the present invention analyzes the residual, “less-significant” signal modes of the signal, for invariant properties. Identification of the invariant characteristics provides the basis for the reduction of redundant information and the division (ie separation) of signal modes.

Property Point Tracking

In one embodiment of the present invention, the spatial positions of one or more frames are determined through spatial intensity field gradient analysis. These properties correspond to some intersection of "lines" which can be roughly described as "corners". The embodiment also selects a series of corners that are both strong corners and spatially disparate ones referred to herein as feature points. In addition, using hierarchical multi-resolution estimation of the light flow enables the determination of the translational displacement for characteristic points over time.

In FIG. 2, the object tracking 220 process pulls together the detection instances from the object tracking processes 208, and also of properties of one or more detected objects across many video frames 202 & 204. It is shown to identify the correspondences 222.

Non-limiting embodiments of feature tracking can be used such that the features are used to qualify more common gradient analysis methods, such as block-based motion estimation.

Another embodiment processes the prediction of motion estimates based on the feature tracking.

Object-Based Detection and Tracking

In one non-limiting embodiment of the invention, a robust object classifier is used to track the faces of video frames. This classifier is based on chained responses to oriented edges directed at faces. In the classifier, the edges are defined as a series of basic Haar properties and rotation of the properties by 45 degrees. The cascade classifier is a variant of the AdaBoost algorithm. In addition, response operations can be optimized through the use of summed region tables.

Regional registration

Registration is accompanied by the assignment of correspondences between elements of objects identified in two or more video frames. The correspondences are the basis for modeling spatial relationships between video data at points that are distinct in time in the video data.

In order to describe specific embodiments and related reductions for carrying out with respect to known algorithms and derivatives according to the invention of the above algorithms, various non-limiting means for registration are described. It is described for the invention.

One means for modeling apparent light flow in a space-time sequence can be achieved through the generation of a finite difference field from two or more frames of video data. The light flow field can be weakly estimated if the correspondences form constant constancy constraints in both spatial and intensity sense.

As shown in FIG. 3, the frame 302 or 304 may be spatially partially through a decimation process 306, or some other portion-sampling process (eg, a low pass filter). Sampled. The spatially reduced images 310 & 312 may be further partially-sampled.

Diamond navigation

Given a non-overlapping division into blocks of a video frame, the preceding frame of the video is searched for a match for each block. Full search block-based (FSBB) motion estimation finds the position of the preceding frame of video with the lowest error when compared to any block of the current frame. Performing FSBB can be computationally expensive and often does not yield a better match compared to other motion estimation skins based on localized motion estimation. Diamond search block-based (DSBB) gradient descent motion estimation uses FSBB to use diamond shape search patterns of various sizes to repeatedly traverse the error gradient towards the best match for any block. Is a common alternative.

In one embodiment of the present invention, DSBB is used in the analysis of the image gradient field between one or more frames of video to produce finite differences whose values are later factored into higher order motion models.

It will be apparent to those skilled in the art that block-based motion estimation can be seen as equivalent to vertex analysis of normalized meshes.

Phase-based motion estimation

In the prior art, block-based motion estimation is typically implemented as spatial search resulting in one or more spatial matches. Phase-based normalized cross correlation (PNCC) as shown in FIG. 3 converts the blocks from the preceding frame and the current frame into “phase space” and finds the cross correlation of the two blocks. . Cross correlation is expressed as a field of values whose positions correspond to 'phase shifts' of the edges between the two blocks. The positions can be isolated through threading and inverted to spatial coordinates. The spatial coordinates are separate edge replacements and correspond to motion vectors.

Advantages of the PNCC include contrast masking, which allows for allowance of gain / exposure adjustment in the video stream. In addition, the PNCC allows for results from a single step that may take many iterations from a spatial based motion estimator. Also, the motion estimates are accurate part-pixels.

One embodiment of the present invention utilizes PNCC for analysis of an image gradient field between one or more frames of video to produce finite differences whose values are later factored into higher order motion models.

Global registration

In one embodiment, the present invention factors one or more linear models from the field of finite difference estimates. The field in which the sampling takes place is referred to herein as a general population of finite differences. The described method uses robust estimation similar to that of the RANSAC algorithm.

As shown in FIG. 4, for global motion modeling, finite differences are collected in analytical motion estimates 402 collected into a general population pool 404 that is repeatedly processed by random sampling of the motion estimates 410. And a linear model factors the samples (420). The results are then used to adjust the population 404 to make the linear model more explicit through the exclusion of outliers for the model as found through any process.

In one embodiment of the linear model estimation algorithm, the motion model estimator is based on a linear least squares solution. The dependency causes the estimator to be discarded by unusual value data. Based on RANSAC, the disclosed method is a robust method that considers the effects of unusual values through iterative estimation on a subset of the data, while examining the motion model that will describe the significant subset of the data. The model generated by each probe is checked against the percentage of data it represents. If there are a sufficient number of iterations, a model will be found that combines the largest subset of data.

As recognized and shown in FIG. 4, the present invention discloses innovations beyond the RANSAC algorithm, which is an iterative form of the algorithm, accompanied by initial sampling of finite differences (samples) and least squares estimation of the linear model. Using the solved linear model, the synthesis error is evaluated for all samples of the general population. A rank is assigned for the linear model based on the number of samples whose residuals reach a preset threshold. The ranking is considered "candidate consensus."

Initial sampling, solving, and ranking are performed repeatedly until the termination criteria are met. Once the criteria are met, the linear model with the highest rank is considered the final consensus of the population.

The optional refinement step iteratively analyzes a subset of samples in the order that best matches the candidate model, and repeats the subset size until adding more samples exceeds the residual error threshold for the entire subset. To increase.

As shown in FIG. 4, the global model estimation process 450 is repeated 452 until the consensus rank acceptability test is met. If the ranking is not achieved, the population of finite differences 404 is aligned in proportion to the found model in an effort to reveal a linear model. The best (highest rank) motion model is added to the solution set in step 460. The model is then re-evaluated in step 470. Upon completion, population 404 is re-aligned.

The described non-limiting embodiments of the present invention, in order to determine subspace manifolds in other parameter vector spaces, which may correspond to a particular linear model, as described above as fields of finite difference vectors, It can be further generated as a general method of sampling vector space.

Another result of the global registration process is that the difference between the registration process and the regional registration process yields a regional registration balance. This residual is an error in the global model in approximating the regional model.

Normalization

Normalization refers to resampling of spatial contrast fields towards a standard, or usually, spatial configuration. If the relative spatial configurations are invertible spatial transformations between these configurations, accompanying and resampling the interpolation of the pixels is also reversible up to a topological limit. The normalization method of the present invention is described in FIG.

If two or more spatial contrast fields are normalized, increased computational efficiency can be achieved by preserving intermediate normalization operations.

Spatial transformation models used to resample images for registration or equally normalization include global and regional models. Global models are in increasing order from translational to projective. Regional models are basically finite differences that imply an interpolant on a pixel's neighborhood as determined by a block or more complexly by a piece-wise linear mesh.

Interpolation of the original lightness fields to the normalized lightness field increases the linearity of the PCA appearance models based on subsets of the lightness field.

As shown in FIG. 2, object pixels 232 & 234 may be resampled 240 to yield a normalized version of object pixels 242 & 244.

Mesh -Based normalization

Another embodiment of the invention mosaics the feature points into a triangle-based mesh, vertices of the mesh are tracked, and the relative positions of the vertices of each triangle are three-dimensional for a plane coincident with the three vertices. It is used to estimate the three-dimensional surface normal. When the surface normal coincides with the camera's projection axis, the imaged pixels provide a minimal-distortion rendering of the object corresponding to the triangle. Generating a normalized image that tends to favor orthogonal surface normals can produce pixels that preserve intermediate data types that will increase the linearity of future appearance-based PCA models.

Another embodiment utilizes conventional block-based motion estimation to implicitly model a global motion model. In one non-limiting embodiment, the method factors the global affine motion model from the motion vectors described by conventional block-based motion estimation / prediction.

9 describes a method of combining global and regional normalization.

Progressive Geometry Normalization

Classification of spatial discontinuities is used to align mosaicized meshes to implicitly model discontinuities as they correspond to mesh edges.

Homogeneous region boundaries are approximated by polygonal contours. The contours are sequentially approximated with decreasing accuracy in order to determine the critical priority of each polygon vertex. Vertex priorities are passed across areas to preserve the vertex priorities for shared vertices.

In one embodiment of the invention, the polygon decomposition method allows for prioritization of the boundaries associated with homogeneous classification of fields. Pixels are classified according to the same homogeneity criteria, such as spectral similarity, and the classification labels are spatially connected to regions. In another preferred non-limiting embodiment, a four or eight-connectedness criterion is applied to determine the spatial coupling relationship.

In a preferred embodiment, the boundaries of the spatial regions are discretized into polygons. The spatial overlay of all polygons for all homogeneous regions is mosaicized and merged into a preliminary mesh. The vertices of the mesh are decomposed by several criteria to reveal simpler mesh representations that retain much of the perceptual importance of the original mesh.

In a preferred embodiment, as disclosed elsewhere in this specification, the image registration method is biased towards the high priority vertices with strong image gradients. The resulting deformation models tend to preserve the spatial discontinuities associated with the geometry of the imaged object.

In a preferred embodiment, active contours are used to refine region boundaries. It is allowed to carry one repetition in the active contour for each polygonal area. The movement or “deformation” of each active contour vertex in different regions is combined in average operations to allow for constrained transfer of the implied mesh to ensure that they have all of their memberships.

In a preferred embodiment, the vertices are assigned a count of the number of adjacent vertices that it has in the mesh for adjacent vertices which is also part of the contour of the different area. The other vertices are defined as in opposition. For vertices with a count of 1, the vertices do not have opposite vertices and therefore need to be preserved. If two adjacent opposite vertices all have a count of one (ie, the two vertices are in different polygons and are adjacent to each other), one vertex is resolved to the other. If a vertex of 1 is opposite a neighboring polygon vertex with a value of 2, a vertex with a count of 1 resolves into a vertex with a count of 2, and the vertex count becomes 1. So, if there are other neighboring vertices, the vertices can be decomposed again. In this case, it is important to preserve the original vertex count, as a result of which it is possible to bias the direction of resolving based on the original vertex count when the vertex is decomposed. This means that vertex (c) is a vertex because vertex (b) has already been used at one resolution, rather than vertex (a) is decomposed into vertex (b), then vertex (b) is decomposed into vertex (c). to be decomposed to (b).

In a preferred embodiment, T-junction points are treated specially. The T-match points are points of the polygon that do not have points in adjacent polygons. In this case, each polygonal vertex is first plotted on an image point map, which map identifies the spatial location of the vertex and its polygon identifier. The perimeter of each polygon is then traversed and examined to see if there are any adjacent vertices from other polygons. If there are neighboring vertices from another area, they are each checked to see if they already have neighboring vertices from the current polygon. Otherwise, the current point is added as a vertex of the current polygon. The further check ensures that isolated vertices of other polygons are used to generate T-match points. Otherwise, it may just add new vertices whose region already has matching vertices. Thus, the opposite vertex is added only if the neighboring vertex is not on the opposite side by the current region. In another embodiment, the detection efficiency of the T-matches is increased by using a mask image. The vertices of the polygons are visited sequentially, and the mask is updated so that the pixels of the vertices are identified as belonging to the polygonal vertices. Then the pixels around the polygon are traversed and if they match the polygon vertices, they are recorded as vertices within the current polygon.

In a preferred embodiment, if the spectral region is remapping by one or more overlapping homogeneous image gradient regions and the other homogeneous spectral region also overlaps, the previously remapped all of the previously remapped regions are currently remapped. The same labels are given as areas. So in essence, if the spectral region is overlapped by two homogeneous regions, then all of the spectral regions overlapped by the two homogeneous regions will have the same label, so that one spectral region is actually two Will be covered by one homologous region instead of three homologous regions.

In one embodiment of the invention, it is useful to process region maps rather than region lists to find adjacency merge criteria. In another embodiment, the spectral splitting classifier may be modified to train the classifier using non-homologous regions. This allows the processing to be concentrated at the edges of the spectral regions. In addition, like a canny edge detector, adding different partitions based on using the edges, and then providing it to the active contour to identify the initial set of polygons is a greater distinction of homogeneous regions. To make it possible.

Regional normalization

The present invention provides a means by which a spatiotemporal stream can be registered in a 'local' manner.

One such localized method resolves the localized coherency of an imaged phenomenon to resolve apparent image brightness constancy ambiguities with respect to the imaged phenomenon or local deformation of the imaged object. Spatial application of geometric meshes is used to provide a means of analyzing pixels to take time into account.

This mesh is used to provide a single linear model of surface deformation in the image plane as a means of local normalization. The imaged phenomena can often correspond to this model when the temporal resolution of the video stream is high compared to the motion of the video. Exceptions to model estimates are handled through various techniques, including topological constraints, neighbor vertex constraints, and homogeneity analysis of pixels with image gradient regions.

In one embodiment, feature points are used to create a mesh whose vertices are composed of triangular elements corresponding to the feature points. Corresponding feature points are other frames, and correspondingly pixels, suggesting interpolated "warping" of triangles, to create a local deformation model.

7 shows the creation of such an object mesh. 8 illustrates the use of this object mesh to normalize frames locally.

In one preferred embodiment, a triangular map is created that identifies the triangle from which each pixel of the map comes. In addition, the affine transformation corresponding to each triangle is pre-computed as an optimization step. And also, upon generating a regional deformation model, the anchor image (leading) is traversed using spatial coordinates to determine the coordinates of the source pixel for the sample. The sampled pixel will replace the current pixel position.

In another embodiment, the regional transformation is performed after the global transformation. In the foregoing specification, global normalization has been described as a process that allows a global registration method to be used to spatially normalize a pixel in two or more frames of video. Globally normalized resulting video frames may also be localized. The combination of the two methods constrains local normalization for tablets that are globally reached in the solution. This can greatly reduce the ambiguity that local methods require for decomposition.

In another non-limiting embodiment, vertices in the case of feature points or “normal meshes” are defined through analysis of image gradients in the neighborhood of the points. The image gradient may be computed directly or through some indirect operation, such as a Harris response. Additionally, the points can be filtered by motion estimation errors associated with spatial constraints and descent of the image gradient. The defined points can be used as the basis for a mesh by one of many mosaicization techniques, resulting in a mesh whose elements are triangles. For each triangle, an affine model is generated based on the points and their residual motion vector.

In a preferred embodiment, a list of triangle affine parameters is maintained. The list is repeated and a current / leading point list is constructed (using the vertex search map). The current / preceding point list is passed to a routine used to estimate the transform, which calculates the affine parameters for the triangle. The affine parameters, or model, are stored in the triangle affine parameter list.

In another embodiment, the method traverses a triangle identifier image map, where each pixel of the map includes an identifier for a triangle of a mesh of which the pixel has membership. And for each pixel belonging to any triangle, the corresponding global and local deformation coordinates for that pixel are computed. The coordinates are used to sample the corresponding pixel and apply its value at the corresponding "normalized" position.

In another embodiment, spatial constraints are applied to points based on image correspondence strength and density resulting from the search for an image gradient. The points are sorted after the motion estimation is made based on some standard of image contrast residuals. The points are then filtered in prayer to a spatial density constraint.

In another embodiment, spectral spatial partitioning is used and small homogeneous spectral regions are merged with neighboring regions based on spatial affinity, similarity in their intensity and / or color. Homogeneous merging is then used to combine spectral regions together based on their overlap with regions of homogeneous texture (image gradient). Another embodiment uses center-peripheral points, which are small areas surrounded by larger areas as defining points of interest for supporting the vertices of the mesh. In another non-limiting embodiment, the point around the center is defined as an area within one pixel whose bounding box is 3 × 3 or 5 × 5 or 7 × 7 pixels in size, and the spatial image gradient for the bounding box is It is a corner shape. The center of the region can further be classified as a corner defining the position as a useful vertex position.

In another embodiment, horizontal and vertical pel finite difference images are used to classify the intensity of each mesh edge. If an edge has many finite differences that coincide with its spatial location, the edge and thus the vertices of the edge are considered very important in the regional deformation of the imaged phenomenon. If there is a large derivative difference between the means for summations of the finite differences of the edge, the region edge corresponds in most cases to the texture change edge and not to the quantization step.

In another embodiment, the spatial density model termination condition is used to optimize the processing of mesh vertices. If a sufficient number of points covering most of the spatial area of the start of the detection rectangle have been examined, the process may end. The end produces a score. Vertices and feature points entered in the processing are sorted by the score. If a point is too close spatially to an existing point, or if the point does not correspond to an edge in the image gradient, the point is discarded. Otherwise, the image gradient of the neighborhood of the point is descended, and the point is also discarded if the remainder of the gradient exceeds the threshold.

Regular Mesh Normalization (regular mesh normalization)

The present invention extends the above mentioned regional normalization utilizing regular meshes. The mesh is constructed without consideration for the underlying pixel but positioned and sized corresponding to the detected object.

Given the detected object area, the scale and spatial frame position indicating the size of the face creates a regular mesh over the beginning of the face area. In a preferred embodiment, a non-overlapping set of tiles is used to outline the rectangular mesh and a diagonal partitioning of the tiles is performed to yield a regular mesh with triangular mesh elements. In another preferred embodiment, the tiles are proportional to the tiles used in conventional video compression algorithms (eg MPEG-4 AVC).

In a preferred embodiment, the vertices associated with the aforementioned mesh are prioritized through analysis of pixel regions surrounding the vertices in specific frames of video used for training. Analysis of the gradient of the regions provides confidence in the process associated with each vertex that may be dependent on a local image gradient (such as block-based motion estimation).

Correspondences of vertex positions of multiple frames are found through a simple descent of the image gradient. In a preferred embodiment, this is achieved through block-based motion estimates. In this embodiment, high confidence vertices consider high trust responses. Lower trust vertex correspondences are implicitly reached through resolving of obscure image gradients through inference from upper trust vertex correspondences.

In one preferred embodiment, the regular mesh is rebuilt at the beginning of tracking the rectangle. The tiles are created 16 × 16 to create a triangular mesh, and are cut diagonally. The vertices of the triangles are estimated motions. Motion estimation depends on the type of texture each point has. Textures are divided into classes that define the processing order of the three classes, corners, edges, and vertices. Corner vertices use neighboring vertex estimation, that is, motion estimates of neighboring points (if available) are used for predictive motion vectors, and motion estimation is applied for each corner vertex. The motion vector that provides the lowest mad error is used as the vertex motion vector. The search strategy used for the corners is all (wide, small, origin). In the case of edges, the nearest neighbor motion vectors are again used as predicted motion vectors, and the one with the least amount of error is used. The search strategy for the edges is small and origin. For homogeneous, neighboring vertices are searched and motion estimates with the lowest error are used.

In one preferred embodiment, the image gradient for each triangle vertex is computed and aligned based on class and size. So, the corners are before the edges that are allogeneic. In the case of corners the strong corners are before the weak corners, and in the case of the edges the strong edges are before the weak edges.

In one preferred embodiment, the local deformation for each triangle is based on a motion estimate associated with that triangle. Each triangle is estimated to be affinity for it. If the triangle is not topologically inverted or degenerate, then pixels that are part of the triangle are used to sample the current image based on the obtained estimate affine.

Division

Spatial discontinuities identified through the segmentation processes that are further described are efficiently encoded through geometric parameterization of their respective boundaries, referred to as spatial discontinuity models. The spatial discontinuity models may be encoded in a progressive manner that takes into account more concise boundary techniques corresponding to subsets of encoding. Progressive encoding provides a robust means of prioritizing spatial geometry while maintaining many important aspects of spatial discontinuities.

Preferred embodiments of the present invention combine multi-resolution segmentation analysis with gradient analysis of spatial contrast fields, and also use temporal stability constraints to achieve robust segmentation.

As shown in FIG. 2, if the correspondences of the characteristics of the object are tracked over time 220 and modeled 224, then the adherence to the motion / deformation model divides the pixels corresponding to the object 230. Can be used for The process can be repeated for multiple objects 206 & 208 detected in video 202 & 204. The results of the process are divided object pixels 232.

One form of invariant characterization used by the present invention focuses on the identification of spatial discontinuities. The discontinuities are edges, shadows, occlusions, lines, corners, or any other visible feature that causes an interruption, and discernible separation between pixels in one or more imaged frames of the video. Specify as. In addition, subtle spatial discontinuities between similarly colored and / or textured objects may cause the pixels of the objects of the video frame to pass relatively coherent movements to the objects themselves rather than to relatively different movements with respect to each other. Can only be specified when The present invention utilizes a combination of spectral, texture and motion segmentation to robustly identify spatial discontinuities associated with the critical signal mode.

Time division

Temporal integration of finite-difference measurements of analytic motion vectors, or equivalent spatial intensity fields, into a higher-order motion model is a form of motion segmentation described in the prior art.

In one embodiment of the invention, a dense field of motion vectors representing finite differences in object motion of the video is created. The derivatives are spatially grouped through regular partitioning of tiles or by some initialization process such as spatial partitioning. Each group of "deriveds" is integrated into a higher order motion model by a linear least squares estimator. The resulting motion models are clustered as vectors in motion model space by the k-means clustering technique. Derivatives are classified based on which clusters best match them. Cluster labels are spatially clustered as the evolution of spatial partitioning. The process continues until space partitioning is stable.

In another embodiment of the invention, the motion vectors for a given aperture are interpolated at a series of pixel positions corresponding to the aperture. If the block defined by the interpolation lies at the pixel corresponding to the object boundary, the resulting classification is some exceptional diagonal partitioning for the block.

In the prior art, the least-squares estimator used to integrate derivatives is very sensitive to outliers. The sensitivity can produce motion models that highly bias the motion model clustering method for the points at which the iterations diverge widely.

In the present invention, motion segmentation methods identify spatial discontinuities through analysis of apparent pixel motion over two or more frames of video. The apparent motion is analyzed for consistency across the frames of the video and integrated into parametric motion models. Spatial discontinuities associated with the consistent movement are identified. Motion segmentation can also be referred to as time segmentation as time variations can be caused by motion. However, time changes can also be caused by some other phenomenon such as regional deformation, lighting changes, and the like.

Through the described method, an important signal mode corresponding to the normalization method can be identified and separated from the surrounding signal mode (background or non-object) by one of several background subtraction methods. Often, the methods statistically model the background as pixels representing the minimum amount of change in each time instance. The change can be characterized as a pixel value difference. Alternatively, motion segmentation can be achieved by given detected positions and scale of the critical image mode. The distance transform can be used to determine the distance of every pixel from the detected location. If the pixel values associated with the maximum distance are maintained, a reasonable model of the background can be resolved. In other words, the ambient signal is re-sampled in time by a signal difference metric.

Given a model of the ambient signal, the perfect critical signal mode can be differentiated at each time instance. Each of the differences can be re-sampled into spatially normalized signal differences (absolute differences). The differences are then aligned and accumulated relative to each other. Since the differences are spatially normalized relative to the critical signal mode, the peaks of the difference will mostly correspond to the pixel positions associated with the critical signal mode.

Decomposition of non-objects

Given a decomposed background image, errors between the image and the current frame can be spatially normalized and accumulated in time. Such decomposed background images are described in the "Background Decomposition" section.

As a result, the accumulated error is thresholded to provide an initial contour. The contour is delivered spatially to balance the error residual against the contour deformation.

gradient  Gradient segmentation

Texture division methods, or equivalently brightness gradient division, analyze a local gradient of pixels in one or more frames of video. Gradient response is a statistical measure that characterizes spatial discontinuities local to pixel locations in a video frame. One of several spatial clustering techniques is used to combine gradient responses into spatial regions. The boundaries for the regions are useful for identifying spatial discontinuities in one or more of the video frames.

In one embodiment of the invention, a summation area table resulting from computer graphics texture generation is used to facilitate the calculation of the gradient of the intensity field. Four lookups combined with four additional operations create a field of progressively summed values that facilitates summation of any rectangle of the original field.

Another embodiment uses a Harris response generated for the image, with neighbors of each pixel classified as homogeneous, edge or corner. A response value is generated from this information and indicates the degree of edge-ness or cornered-ness for each element of the frame.

Multi-scale gradient  analysis

Embodiments of the invention also constrain the image gradient supported by generating image gradient values over several spatial scales. The method may help define an image gradient such that spatial discontinuities at different scales are used to support each other, and the edge should be "important" while "edge" is distinguished at several different spatial scales. More defined image gradients will tend to correspond to more important characteristics.

In a preferred embodiment, a texture response field is first generated, and then the values of the field are quantized into several bins based on k-means binning / partitioning. The original image gradient values are processed progressively using each bin as the interval of values at which individual iterations can be applied to watershed segmentation. The benefit of this approach is that it is defined as a relative view with a homogeneous spatial bias.

Spectral  Division

Spectral segmentation methods analyze the statistical probability distribution of black and white, gray scale, or color pixels in a video signal. A spectral classifier is constructed by performing clustering operations on the probability distribution of the pixels. Classifiers are used to classify one or more pixels as belonging to a probability class. The resulting probability class and its pixels are given a class label. The class labels are spatially associated with regions of a pixel having distinct boundaries. The boundaries identify spatial discontinuities in one or more of the video frames.

The present invention utilizes spatial segmentation based on spectral classification to segment the pixels of video frames. In addition, the correspondence between regions can be determined based on the overlap of spectral regions having regions in preceding partitions.

If the video frames are roughly composed of consecutive color regions that are spatially connected to wider regions corresponding to the objects of the video frame, the identification and tracking of the colored (or spectral) regions is followed by the objects in the video sequence. It can be observed that partitioning can be facilitated.

Background split

The described invention includes a video frame background modeling method based on the temporal maximum of spatial distance measurements between each individual pixel and a detected object in each frame of video. Given the detected position of the object, the distance transformation is applied while generating a scalar distance for each pixel of the frame. A map of the maximum distance spanning all video frames for each pixel is maintained. If the maximum value is initially assigned or subsequently updated to a new and different value, the corresponding pixel for the video frame is kept in a "decomposed background" frame.

ocean modelling

A common goal of video processing is often to model and preserve the appearance of a sequence of video frames. The present invention allows for the use of preprocessing to provide constrained appearance modeling techniques in a manner that is robust and widely applicable. The registration, segmentation, and normalization described above is apparent for this.

The present invention describes a means of appearance variance modeling. The main basis of appearance change modeling is the analysis of feature vectors to reveal a solid foundation using linear correlations in the case of linear models. Feature vectors representing spatial contrast field pixels can be combined into an appearance change model.

In an alternative embodiment, the appearance change model is computed from the divided subset of pixels. Also, the feature vector can be separated into feature vectors that are spatially non-overlapping. The spatial decomposition can be achieved by spatial tiling. Computational efficiency can be achieved by processing the temporal ensemble without sacrificing the dimensionality of the more global PCA method.

In generating an appearance change model, spatial intensity field normalization can be used to reduce PCA modeling of spatial transforms.

PCA

A preferred means of generating the appearance change model is by combining video frames as training matrix, or ensemble pattern vectors, and by applying principal component analysis (PCA) on the training matrix. If this extension is truncated, the resulting PCA transform matrix is used to analyze and synthesize subsequent frames of video. Based on the level of truncation, varying levels of quality of the original appearance of the pixel can be achieved.

Specific means for the construction and decomposition of pattern vectors are known to those skilled in the art.

Given the spatial division of the critical signal mode from the surrounding signal and the spatial normalization of the mode, the appearance of the pixel itself, or, as a result, the normalized signal, is directly between the bit-rate and approximation error for the representation of the pixel appearance. It can be factored into linearly correlated components using low rank parameterization to account for trade-offs.

As shown in FIG. 2, normalized object pixels 242 & 244 can be projected into vector space and the linear correspondences undergo a decomposition process 250 such as PCA to yield a dimensionally concise version of data 252 & 254. Can be modeled using.

Sequential PCA

PCA encodes patterns into PCA coefficients using a PCA transform. Better patterns can be represented by the PCA transform, and fewer coefficients are needed to encode the pattern. Recognizing that pattern vectors can be degraded over time between the patterns to be encoded and the acquisition of training patterns that update the transform can help counter act against degradation. . As an alternative to creating a new transformation, sequential updating of existing patterns is more computationally efficient in some cases.

Many modern compression algorithms predict the frame of video from one or more other frames. The predictive model is based on partitioning of each predicted frame into non-overlapping tiles that are matched to corresponding patches in another frame and are associated with an associated translational displacement parameterized by an offset motion vector. . The spatial replacement, optionally coupled with the frame index, provides a "predicted motion" version of the tile. If the error of prediction is below a certain threshold, the pixels of the tiles are suitable for residual encoding; There is a corresponding gain in compression efficiency. Otherwise, the pixels of the tiles are just encoded. Alternatively, this type of tile-based motion prediction method, called block-based, models video by interpreting tiles comprising pixels. If the imaged phenomenon of the video is true to this type of modeling, the corresponding encoding efficiency increases. The modeling constraint exists for the imaged objects that undergo motion to estimate a level of temporal resolution, or to follow an analytical estimate inherent in block-based prediction. Another requirement for the analytical model is that space replacement for constant time resolution should be limited; That is, the time difference between the frames from which the prediction is derived and the frame being predicted should be a relatively short amount of absolute time. The time resolution and motion constraints facilitate the identification and modeling of certain redundant video signal components present in the video stream.

Residual -based decomposition

In MPEG video compression, the current frame is constructed by applying a residual update for compensation blocks, followed by a motion that compensates for the preceding frame using motion vectors, and finally any blocks that do not have enough matches are new blocks. Are encoded as

The pixel corresponding to the residual blocks is mapped to the pixel of the preceding frame via the motion vector. The result is a temporal path of pixels through the video that can be synthesized through successive application of residual values. The pixel is identified as best able to be represented using PCA.

Closed-based decomposition

Another improvement of the invention determines if the motion vectors applied to the blocks cause any pixels from the preceding frame to be closed (covered) by pixel movement. For each closure event, we divide the pixel that we are closing into a new layer. The pixel will also be revealed without history. The exposed pixels are placed in any layer where they are combined in the current frame and a historical fit can be made for that layer.

The temporal continuity of the pixels is supported through grafting and fusing of pixels to different layers. Once the stable layer model is reached, the pixels of each layer can be grouped based on membership to the coherent motion models.

Part-time quantization band (sub-band temporal quantization)

An alternative embodiment of the present invention uses Discrete Cosine Transform (DCT) or Discrete Wavelet Transform (DWT) to decompose each frame into partial-band images. Principal Component Analysis (PCA) is then applied to each of the "partial-band" videos. The concept is that the partial-band decomposition of a frame of video reduces the spatial variation of any one sub-bands compared to the original video frame.

In the case of video of moving objects (persons), spatial changes tend to dominate the changes modeled by the PCA. Partial-band decomposition reduces the spatial variation in any decomposition video.

For DCT, decomposition coefficients for any one sub-band are spatially arranged in the partial-band video. For example, DC coefficients are taken from each block and arranged into partial-band video that looks like a stamped version of the original video. This is repeated for all other part-bands, and the resulting part-band videos are each processed by the PCA.

In the case of DWT, the partial-bands are already arranged in the manner described for the DCT.

In a non-limiting embodiment, truncation of the CPA coefficients is varied.

Wavelet

When data is decomposed by Discrete Wavelet Transform (DWT), multi-band-pass data sets cause subspace decompositions. The transformation process can be applied recursively to the derived data until only single scalar values are derived. Scalar elements in a decomposed structure are typically related to hierarchical parent / child approaches. The resulting data includes a multi resolution hierarchical structure and also finite differences.

When DWT is applied to spatial contrast fields, many naturally occurring phenomena of the image are represented with little perceptual loss by the first or second low band pass derived from the data structures due to the low spatial frequency. Truncating the hierarchical structure provides a concise representation whether high frequency spatial data is present or considered noise.

While PCA can be used to achieve accurate reconstruction with a small number of coefficients, the transformation itself can be quite large. To reduce the size of the "initial" transform, an embedded zero tree (EZT) configuration of wavelet decomposition can be used to build progressively more accurate versions of the transform matrix.

Subspace Classification (subspace classification)

As will be appreciated by those skilled in the art, ideally sampled phenomenon data and derived data may be represented as a series of data vectors corresponding to an algebraic vector space. The data vectors comprise the normalized appearance of the segmented object's pixel, motion parameters, and any structural positions of vertices or properties in two or three dimensions in a non-limiting manner. Each of the vectors is in vector space, and the geometric analysis of the space can be used to produce sampled, or parametric, concise representations of the vectors. Advantageous geometric conditions are represented by parameter vectors forming concise subspaces. When one or more subspaces are mixed, it may be difficult to identify a more complex single subspace, constituent subspaces. There are a number of partitioning methods that consider the separation of the subspaces by examining the data of the high-dimensional vector spaces generated through some interaction of the original vectors (such as inner products).

One way of partitioning the vector space is to entrain the projection of the vectors into the Veronese vector space, which represents polynomials. Such methods are known to those skilled in the art as generalized PCA or GPCA techniques. Through this projection, normals to polynomials can be found and grouped, and the original vectors associated with the normals can be grouped together. An example of the use of this technique is in the factor of the three-dimensional structural model and the two-dimensional spatial point correspondences tracked over time into the movement of the three-dimensional model.

The GPCA technique is incomplete when defined and applied, and yields results only when data vectors are generated with almost no noise. The prior art assumes supervisory user intervention to guide the GPCA algorithm. The constraints greatly limit the potential of the technique.

The present invention extends the conceptual basis of the GPCA method to robustly identify and subdivide multiple subspaces in the presence of noise or mixed co-dimensions. The innovation provides an unsupervised improvement of the techniques in current technology.

In the prior art, GPCA operates on the normal vectors of the polynomials of the Veronese map without considering the tangent space of the normal vectors. The method according to the invention extends the GPCA to find the tangent space orthogonal to the space of the normal vectors normally found in the Veronese map. The "tangent space", or subspace of the Veronese map, is used to factor the Veronese map.

Tangent space is identified through the application of legend wave transforms between tangents and positions of normals to plane wave expansions and tangent plane coordinates that reveal duality in the representation of geometric objects, in particular the polynomials of the Veronese map. . Discrete Legendre transforms are applied through convex analysis to define the constrained shape of the derivative corresponding to the normal vectors. This approach is used to divide the data vectors by the calculation of the normal vectors in the presence of noise. The convex analysis is integrated with GPCA to provide a more robust algorithm.

The present invention highlights an iterative factorization approach in GPCA application. In particular, the derivative-based implementations found in the prior art are extended to refine the ensemble of classified data vectors through the very same GPCA described herein. Applied repeatedly, the technique can be used to robustly find candidate normal vectors in Veronese mapping and can further be used to define the vectors using the extended GPCA technique. For the factoring step, the original data associated with the set of purified vectors is removed from the original data set. The remaining data set can likewise be analyzed by the innovative GPCA technique. The innovation is important for using the GPCA algorithm in an unsupervised manner. 11 shows a regression refinement of data vectors.

It can also be seen that the extension according to the invention of the GPCA technique has greater advantages in the case of multiple roots in the Veronese polynomial vector space. In addition, if the technique according to the prior art encounters a case of regression when the normals of the Veronese map are parallel to the vector space axis, the method according to the invention is not regressed.

10 illustrates a basic polynomial binding and differentiation method.

Mixed Space Normalized Compression

The present invention extends the efficiency of block-based motion predicted coding schemes through the addition of dividing a video stream into two or more "normalized" streams. The streams are separately encoded in order for the analytical motion estimation of conventional codecs to be valid. Upon decoding of the normalized streams, the streams are de-normalized to their proper location and synthesized to yield the original video sequence.

In one embodiment, one or more objects are detected in the video stream and the pixels associated with each individual object are subsequently split leaving the non-object pixels. Next, a global spatial motion model is created for object and non-object pixels. The global model is used to spatially normalize object and non-object pixels. This normalization effectively eliminated non-interpretive motion from the video stream and provided a series of videos with minimal closed interaction. These are all useful properties of the method according to the invention.

New videos of objects and non-objects, with their pixels spatially normalized, are provided as input to conventional block-based compression algorithms. In accordance with the decoding of the videos, global motion model parameters are used to de-normalize the decoded frames, and the object pixel is synthesized to apply an approximation of the original video stream and applied to the non-object pixel.

As shown in FIG. 6, the preceding detection object instances 206 & 208 for one or more objects 630 & 650 are each processed by a separate instance of the conventional video compression method 632. In addition, the non-object 602 derived from the segmentation 230 of objects is also compressed by conventional video compression 632. The result of each of the separate compressed encodings 632 is separate separate conventionally encoded streams for each 634 corresponding to each video stream separately. At some point, perhaps after delivery, the intermediate encoded streams 234 may be decompressed 636 with a combination of normalized non-object 610 and a number of objects 638 & 658. The synthesized pixel has their non-normalized versions 622 in order to accurately position the pixels spatially relative to each other such that the compositing process 670 can combine the object and the non-object pixel into an integration 672 of the entire frame. 642 & 662 may be de-normalized 640.

Hybrid codec integration

As described in the present invention, when combining a conventional block-based compression algorithm with a normalization-division scheme, there are several methods according to the present invention that result. Mainly, there are required communication protocols and specialized data structures.

Key data structures include global spatial transformation parameters and object partition specific masts. The main communication protocols are layers that contain global spatial transformation parameters and object segmentation specific masks.

Claims (10)

A computer device for generating video signal data in encoded form from a plurality of video frames, wherein: Means for identifying corresponding elements of the object between two or more frames; Means for modeling the correspondences to produce modeled correspondences; Means for resampling pixel data of the video frames associated with the object utilizing the modeled correspondences; And Means for recovering spatial locations of the resampled pixel data utilizing the modeled correspondences, The object is one or more objects, The resampled data is an intermediate form of the data, Video signal data generating device. The method of claim 1, The object is estimated by a tracking method, The device, Means for detecting any object in a video frame sequence; And Means for tracking the object through two or more frames of a sequence of video frames, The object detecting and tracking means comprises a Viola / Jones face detection algorithm, Video signal data generating device. The method of claim 1, The object is divided from the video frame by a segmentation method, The device, Means for dividing the pixel data associated with the object from other pixel data in the video frame sequence; And Means for assembling the recovered pixels with associated segmentation data to produce an original video frame, Said dividing means comprises time integration, Video signal data generating device. The method of claim 1, The corresponding models are factored into global models, The apparatus comprises means for integrating corresponding measurements into a model of global motion, The corresponding modeling means is adapted to robust sampling consensus for a solution of a two-dimensional affine motion model, and to finite differences generated from block-based motion estimation between two or more video frames in the sequence. Including a sampling population based on Video signal data generating device. The method of claim 1, The intermediate data is further encoded, The device, Means for decomposing the normalized object pixel data into an encoded representation; And Means for reconstructing the normalized object pixel data from an encoded representation, Said decomposition means comprises principal component analysis, and The reconstruction means comprises principal component analysis, Video signal data generating device. The method of claim 5, Non-object pixels of the frame are modeled as follows: The method includes that the object is a remaining non-object of the frame when other objects are removed; Video signal data generating device. The method of claim 5, The divided and resampled pixels are combined with a conventional video compression / decompression process, The device, Means for providing said re-sampled pixel as standard video data to a conventional video compression process; And Means for storing and transmitting model correspondence data along with corresponding encoded video data, The compression / decompression method may allow the conventional video compression method to increase compression efficiency. Video signal data generating device. The method of claim 1, The corresponding models are factored into regional deformation models, The device, Means for defining a two-dimensional mesh that overlays a pixel corresponding to the object; And Means for corresponding measurements into a model of local movement, The mesh defining means is based on a uniform grid of vertices and edges, and the corresponding measurements include vertex displacements based on finite differences generated from block-based motion estimation between two or more video frames in the sequence, Video signal data generating device. The method of claim 8, The vertices correspond to discrete image characteristics, The device, Means for identifying significant image characteristics corresponding to the object, The identification means is an analysis of the image gradient Harris response, Video signal data generating device. A computer device for separating data vectors in discrete linear subspaces, Means for performing subspace partitioning on the set of data vectors; Means for making subspace segmentation criteria a constraint through the application of tangent vector analysis in an implicit vector space, The subspace partitioning method is GPCA, The implicit vector space is a Veronese map, The tangent space constraint is a Legendre transform, Data vector separator.

Images