JP2015515806A - Context-based video encoding and decoding - Google Patents

Abstract

By applying higher-order modeling, the fundamental limitations of the inter-prediction process of a conventional codec are eliminated while maintaining the same general processing flow and general processing framework as a conventional encoder, and an improved inter Provide predictions. A method of processing video data detects at least one of a feature and an object in a target region in a frame using a detection algorithm, models at least one using a set of parameters, and Every instance of at least one is correlated across multiple frames, forming a track of correlated instances, associating the track with a particular block of video data to be encoded, and using the track information, Generate model-based predictions. Model-based prediction is stored as processed video data. [Selection] Figure 1A

Description

Related applications

  This application claims the benefit of US Provisional Patent Application No. 61 / 615,795, filed March 26, 2012, and US Provisional Patent Application No. 61 / 707,650, filed September 28, 2012. This application further claims the benefit of US patent application Ser. No. 13 / 725,940, filed Dec. 21, 2012. This U.S. Patent Application No. 13 / 725,940, filed December 21, 2012, is a PCT filed on Oct. 6, 2009, claiming the benefit of U.S. Provisional Patent Application No. 61 / 103,362, filed Oct. 7, 2008. This is a continuation-in-part of US Patent Application No. 13 / 121,904, filed October 6, 2009, which is a US transitional application of / US2009 / 059653. This US Patent Application No. 13 / 121,904, filed October 6, 2009, is a continuation-in-part of US Patent Application No. 12 / 522,322, filed January 4, 2008. This US Patent Application No. 12 / 522,322, filed January 4, 2008, claims the benefit of US Provisional Patent Application No. 60 / 881,966, filed January 23, 2007, and is filed on June 8, 2006. It is related to US Provisional Patent Application No. 60 / 811,890, and is a continuation-in-part of US Patent Application No. 11 / 396,010 filed on March 31, 2006. This U.S. Patent Application No. 11 / 396,010, filed March 31, 2006, is a continuation-in-part of U.S. Patent Application No. 11 / 336,366, filed January 20, 2006, and is now U.S. Patent No. 7,457,472. It is. U.S. Patent Application No. 11 / 336,366, filed January 20, 2006, is a continuation-in-part of U.S. Patent Application No. 11 / 280,625, filed Nov. 16, 2005, and is currently U.S. Pat.No. 7,436,981. It is. U.S. Patent Application No. 11 / 280,625, filed November 16, 2005, is a continuation-in-part of U.S. Patent Application No. 11 / 230,686, filed September 20, 2005, and is now U.S. Patent No. 7,457,435. It is. U.S. Patent Application No. 11 / 230,686, filed September 20, 2005, is a continuation-in-part of U.S. Patent Application No. 11 / 191,562, filed July 28, 2005, and is now U.S. Patent No. 7,426,285. It is. US patent application Ser. No. 11 / 191,562, filed July 28, 2005, is now US Pat. No. 7,158,680. The above U.S. Patent Application No. 11 / 396,010 further includes U.S. Provisional Patent Application No. 60 / 667,532 filed on March 31, 2005 and U.S. Provisional Patent Application No. 60 / 670,951 filed on Apr. 13, 2005. Insist on profit. This application is further related to US Provisional Patent Application No. 61 / 616,334, filed March 27, 2012.

  The entire teachings of the above patent applications and patents are incorporated herein by reference.

  Video compression (video compression) can be said to be a process of expressing digital video data in a format that can be stored or transmitted with a small number of bits. Video compression algorithms can achieve compression by taking advantage of spatial, temporal, or color space redundancy or irrelevance of video data. Typically, a video compression algorithm divides video data into parts such as a frame group or a pel group, identifies a redundant part included in the video, and determines the redundant part from the original video data. Can be expressed with a small number of bits. By reducing such redundancy, greater compression can be achieved. An encoder is used when converting the video data into the encoding format. Then, by using the decoder, the encoded video is converted into a format almost comparable to the original video data. What implement | achieves an encoder / decoder is called a codec (encoder decoder).

  When encoding a video frame, a standard encoder divides one video frame into a plurality of coding units, that is, macro blocks (rectangular blocks including a plurality of adjacent pels) that do not overlap each other. Typically, macroblocks (MB) are processed in a left-to-right scan order or top-to-bottom scan order of the frame. Compression is performed when these macroblocks are predicted / encoded using previously encoded data. The process of encoding a macroblock using spatially adjacent previously encoded macroblock samples within the same frame is referred to as intra prediction. Intra prediction is intended to use spatial redundancy contained in data. The process of encoding a macroblock using similar regions from a previously encoded frame and a motion prediction model is referred to as inter prediction. Inter prediction is intended to use temporal redundancy contained in data.

  The encoder can generate a residual by measuring a difference between data to be encoded and prediction (prediction result). This residual can be the difference between the predicted macroblock and the original macroblock. The encoder may also generate motion vector information (eg, motion vector information indicating the position of the macroblock in the reference frame relative to the macroblock being encoded or decoded). By combining these predictions, motion vectors (for inter prediction), residuals and related data with other processes such as spatial transformation, quantization, entropy coding, loop filters, etc., efficient coding of video data is achieved. Can be generated. The quantized and transformed residuals are processed, added to the prediction, incorporated into a decoded frame, and stored in a frame store (means for storing or storing a frame). Details of such video encoding techniques are well known to those skilled in the art.

  H.264 / MPEG-4AVC (Advanced Video Encoding) is a codec standard that can express high-quality video at a relatively low bit rate using block-based motion prediction / compensation (hereinafter, “ H.264 "). H. H.264 is one of the encoding scheme choices used for major video distribution channels including video streaming over the Internet, video conferencing, cable television and direct satellite television as well as Blu-ray Discs. H. An H.264 encoding basic unit is a 16 × 16 macroblock. H. H.264 is the latest video compression standard that is widely used.

  The basic MPEG standard defines three types of frames (or pictures) according to the encoding method of macroblocks in a frame. One of them, an I frame (intra-encoded picture), is encoded using only data included in the frame. In general, an encoder that receives video signal data first generates an I frame, divides the video frame data into a plurality of macroblocks, and encodes each macroblock using intra prediction. Thus, an I frame is composed of only intra prediction macroblocks (or “intra macroblocks”). Since the I frame is encoded without using information from the encoded frame, the encoding cost is increased. The P frame (predicted picture) is encoded by forward prediction using data (also referred to as a reference frame) from the previously encoded I frame or P frame. A P frame may include both intra macroblocks and (forward) predicted macroblocks. The B frame (bi-predictive picture) is encoded by bi-directional prediction using data from both the previous frame and the subsequent frame. A B frame may include any of an intra macroblock, a (forward) prediction macroblock, and a bi-prediction macroblock.

  As described above, conventional inter prediction is based on block-based motion prediction and compensation (BBMEC). The BBMEC process searches for the best match between the target macroblock (the current macroblock to be encoded) and a region of the same size in the previously encoded reference frame. When the best match is found, the encoder may send a motion vector. The motion vector may include a pointer to the intra-frame position of the best match, as well as information regarding the difference between the best match and the target macroblock corresponding to the best match. While it is possible to exhaustively perform such searches across the “data cube” (height x width x frame index) of the video to find the best match for each macroblock, it is generally computationally intensive Becomes too big. Therefore, the BBMEC search process is limited, limited in time to reference frames to search, and spatially limited to neighboring regions to search. That is, the “best” match is not always found, especially for fast changing data.

A specific set of reference frames is referred to as a Group of Pictures (GOP). The GOP includes only the decoded pels in each reference frame and does not include information about how the macroblock or frame was encoded (I frame, B frame or P frame). In old video compression standards such as MPEG-2, one reference frame (past frame) is used for P frame prediction, and two reference frames (one previous frame and one frame after) are used for B frame prediction. ). In contrast, H. In the H.264 standard, a plurality of reference frames can be used for both prediction of P frames and prediction of B frames. Typically, a frame that is temporally adjacent to the current frame is used as a reference frame, but a frame other than a set of temporally adjacent frames can be designated as a reference frame.

  Conventional compression methods can predict the region of the current frame by blending multiple matches from multiple frames. Blending can be a linear mixture of multiple matches or a log-linear mixture. Such a bi-prediction method is effective when, for example, a fade with time is provided in transition from one image to another. The fade process is a linear blending of two images and may be efficiently modeled by bi-prediction. Some conventional standard encoders, such as the MPEG-2 interpolation mode, can synthesize bi-predictive models by interpolation of linear parameters over many frames.

  H. The H.264 standard further divides a frame into regions composed of one or more adjacent macroblocks, specifically, spatially independent regions called slices, thereby further coding. Provides freedom. Each slice in the same frame is encoded independently of the other slices (ie, decoded independently of each other). Then, like the three types of frames described above, I slices, P slices, and B slices are defined. Therefore, one frame can be constituted by a plurality of types of slices. Further, on the encoder side, in general, the order of processed slices can be freely determined. Thus, the decoder can process the slices that reach the decoder side in an arbitrary order.

  H. According to the H.264 standard, the codec can provide superior quality video with a small file size compared to older standards such as MPEG-2 and MPEG-4 ASP (Advanced Simple Profile). However, H. The “conventional” compression codec that incorporates the H.264 standard, which works on limited bandwidth networks and has limited memory (for example, smartphones and other mobile devices) and improves video quality and resolution When responding to demand for improvement, it has generally been struggling. In order to achieve satisfactory playback on such a device, the quality and resolution of the video often have to be compromised. Furthermore, since the resolution of the video is improved, the file size increases, which becomes a problem when the video is stored in the device or stored outside the device.

  The present invention recognizes the fundamental limitations of the inter-prediction process of conventional codecs and maintains the same general processing flow and general processing framework as conventional encoders by applying higher-order modeling (modeling). However, the above limitations are eliminated and improved inter prediction is provided.

  Higher-order modeling according to the present invention navigates (targets) more predictive search spaces (video data cubes) and provides better prediction than when using conventional block-based motion prediction / compensation Can be generated efficiently. First, a computer vision (computer vision) -based feature / object detection algorithm identifies a target area from a video data cube. The detection algorithm can be selected from a type of non-parametric feature detection algorithm. The detected features and objects are then modeled using a compact (small) set of parameters, and similar instances of the features / objects are correlated (associated) across multiple frames. The present invention further forms a track from the correlated features / objects, associates the track with a particular block of video data to be encoded, and uses this tracking information to model-based prediction for that block of data. Is generated.

  In each embodiment, the particular block of data that is encoded may be a macroblock. The formed tracks may associate features with corresponding macroblocks.

  The feature / object tracking configuration provides additional context to the conventional encoding / decoding process. Furthermore, since features / objects are modeled with a compact set of parameters, information about features / objects can be efficiently stored in memory, unlike expensive configurations that store entire pels of reference frames. Thereby, in the feature / object model, more video data cubes can be searched without requiring an unacceptable calculation amount or memory amount. The model-based prediction obtained in this way is derived from a larger number of prediction search spaces and is therefore superior to conventional inter prediction.

  In some embodiments, the compact set of parameters includes information about the feature / object and can be stored in memory. For a feature, the corresponding parameter may include a feature descriptor vector and the location of the feature. The corresponding parameter can be generated upon detection of the feature.

  After correlating feature / object instances over multiple frames, these correlated instances may be collected in an aggregate matrix (instead of forming a feature / object track). In this case, the present invention forms such an assembly matrix, summarizes the matrix using a subspace of important vectors, and uses this subvector space as a parametric model of the correlated features / objects To do. This makes it possible to realize extremely efficient encoding when those specific features / objects appear in the data.

  Computer based methods for processing video data, codecs for processing video data, and other computer systems and devices for processing video data that embody the principles of the invention described above may be provided.

  The foregoing will become apparent from the following detailed description of exemplary embodiments of the invention as illustrated in the accompanying drawings. Throughout the different figures, the same reference numerals refer to the same elements or components. The drawings are not necessarily to scale, but rather focus on showing embodiments of the invention.

It is a block diagram which shows the characteristic modeling concerning one Embodiment of this invention. It is a block diagram which shows the feature tracking (tracking of a feature) concerning one Embodiment of this invention. FIG. 5 is a block diagram illustrating a process of associating a feature with a neighboring macroblock and generating a good prediction of the macroblock using a track of the feature according to an embodiment of the present invention. FIG. 3 is a block diagram illustrating data modeling with multiple fidelity to achieve efficient encoding, in accordance with one embodiment of the present invention. It is a block diagram which shows the mode of the object specification by the correlation and aggregation of a feature model according to one Embodiment of this invention. It is a block diagram which shows the mode of the object specification by aggregation of the feature of a neighborhood, and aggregation of the macroblock of a neighborhood according to one Embodiment of this invention. It is the schematic which shows an example of a structure of the conversion base codec concerning one Embodiment of this invention. It is a block diagram which shows an example of the decoder for intra prediction macroblocks concerning one Embodiment of this invention. It is a block diagram which shows an example of the decoder for the inter prediction macroblock concerning one Embodiment of this invention. FIG. 3 is a schematic diagram illustrating an example of a configuration of a transform-based codec using feature-based prediction according to an embodiment of the present invention. FIG. 3 is a block diagram illustrating an example of a codec in a feature-based prediction framework according to an embodiment of the present invention. FIG. 6 is a block diagram illustrating a feature instance (feature instance) state extraction process according to one embodiment of the invention. It is a block diagram which shows an example of the component of the codec which uses parametric modeling (parametric modeling) concerning one Embodiment of this invention. FIG. 2 is a block diagram illustrating an example of components of a parametric model based adaptive encoder according to an embodiment of the present invention. It is a block diagram which shows the mode of the motion compensation prediction of the characteristic by the interpolation of the parameter of the characteristic model concerning one Embodiment of this invention. 1 is a block diagram illustrating an overview of an example cache architecture, according to an embodiment of the present invention. FIG. FIG. 6 is a block diagram illustrating processing associated with the use of local (short-term) cache data according to an embodiment of the present invention. It is a block diagram which shows the process accompanying utilization of the data of long-term cache concerning one Embodiment of this invention. It is the schematic which shows the computer network environment for implement | achieving embodiment. It is a block diagram which shows the computer node of the network of FIG. 8A. 4 is a screen shot of a feature-based compression tool in a specific example. It is the screenshot which attached | subjected the number to the facial feature and features other than a face concerning one Embodiment of this invention. It is a screenshot which shows the face specified by the eyelid tracker (face tracking means) of Drawing 8D concerning one embodiment of the present invention.

  The entire teachings of all patent publications, all patent publications, and all references cited in these publications are incorporated herein by reference. In the following, exemplary embodiments of the invention will be described.

  The present invention is applicable to various standard encoding methods and various encoding units (coding units). In the following, unless otherwise specified, terms such as “conventional” or “standard” (which may be used with terms such as “compression”, “codec”, “code”, “encoder”) are H.264. H.264, and “macroblock” refers to H.264 without loss of generality. It shall refer to the H.264 encoding basic unit.

<Feature-based modeling>

<Definition of features>
Components of the present invention may include a video compression process and a video decompression process that can optimally represent digital video data during storage or transmission. The process may comprise or interface with at least one video compression / coding algorithm that takes advantage of the spatial, temporal or spectral redundancy or disassociation of the video data. . Such effective utilization can also be done by the use and retention of feature-based models / parameters. Hereinafter, the terms “feature” and “object” are used interchangeably. An object can be defined as a “large-scale feature” without loss of generality. Both features and objects can be used to model the data.

  A feature is a group of pels that are close to each other and that indicates data complexity. Data complexity can be detected by various criteria as described below. From the viewpoint of compression, the characteristic of data complexity is, ultimately, “high coding cost”. The high coding cost means that the Pell code by the conventional video compression method exceeds a threshold that is considered to be “efficient coding”. When a conventional encoder allocates excessive bandwidth for a given region (when a traditional interframe search cannot find a good match for the given region in a conventional reference frame) Suggests that the region is “rich in features” and that the feature model-based compression method is likely to greatly improve the compression of the region.

<Feature detection>
FIG. 1A shows feature instances (feature instances) 10-1, 10-2,..., 10-n detected in at least one video frame 20-1, 20-2,. Has been. Typically, such features are based on multiple conditions based on structural information derived from pels, and moreover, conventional compression methods use excessive bandwidth to encode the feature region (feature region). It can be detected based on a complexity criterion that indicates that it must be utilized. Furthermore, each instance of the feature is represented by an “region” 30-1, 30-2, 30-2, 30-2,..., 20-n having a spatial extent or boundary within the frame 20-1, 20-2,. ..., 30-n can be spatially specified. Such regions (feature regions) 30-1, 30-2,..., 30-n of features can be extracted as simple rectangular regions composed of pel data, for example. In one embodiment of the present invention, the size of the feature region is H.264. It is the same 16 × 16 size as the H.264 macroblock.

  In the past literature, many algorithms have been proposed for detecting features based on the structure of the pel itself, including non-parametric feature detection algorithms that are robust against various transformations of pel data. ing. For example, scale invariant feature transformation (SIFT) [Lowe, David, 2004, "Distinctive image features from scale-invariant keypoints," Int. J. of Computer Vision, 60 (2): 91-110] Spotted features are detected by convolving function differences. The accelerated robust feature (SURF) algorithm [Bay, Herbert et al., 2008, "SURF: Speeded up robust features," Computer Vision and Image Understanding, 110 (3): 346-359] is also the determinant of the Hessian operator. Detect spotted features by using. In one embodiment of the invention, features are detected using the SURF algorithm.

  In another embodiment, based on the coding complexity (bandwidth) of a conventional encoder, as described in full in US patent application Ser. No. 13 / 121,904, filed Oct. 6, 2009, features Can be detected. The entire teaching content of this US patent application is incorporated herein by reference. As an example, encoding complexity may be determined by analyzing the amount of bandwidth (number of bits) required to encode a region in which features appear with a conventional compression method (eg, H.264). In other words, the operation is different if the detection algorithm is different, but in any embodiment, any detection algorithm is applied to the entire video frame sequence over the entire video data. As an example that does not limit the invention, A first encoding pass by the H.264 encoder is performed to generate a “bandwidth map”. With this bandwidth map, It is defined in which part of each frame the encoding cost according to H.264 is the highest, or its bandwidth map determines it.

  Typically, H.M. A conventional encoder such as H.264 divides a video frame into a plurality of uniform tiles (for example, 16 × 16 macroblocks, subtiles of the macroblocks, etc.) arranged so as not to overlap each other. In one embodiment, each tile is H.264. Based on the relative amount of bandwidth required to encode the tile at H.264, it can be analyzed as a feature candidate. As an example, H.C. The amount of bandwidth required to encode a tile at H.264 can be compared to a certain threshold. If the amount of bandwidth exceeds the threshold, the tile can be determined as a “feature”. This threshold value may be a predetermined numerical value. In that case, this predetermined numerical value can be stored in a database for easy access when detecting features. The threshold value may be a numerical value set as an average value of bandwidth amounts allocated to features encoded in the past. Similarly, the threshold value may be a numerical value set as a median value of the bandwidth amount allocated to the feature encoded in the past. Alternatively, the accumulated distribution function of the bandwidth of tiles over the entire frame (or the entire video) is calculated, and all tiles having bandwidth amounts that fall within the upper percentile of the bandwidth amount of all tiles are determined as “features”. Also good.

  In another embodiment, the video frame may be divided into tiles that overlap one another. This overlap sampling can be offset so that the intersection of the corners of the four tiles that overlap the tile is located at the center of the tile. Such excessive division increases the possibility that the feature can be detected at the first sampling position. In addition, as a more complicated division method, a topological division method can be cited.

  Analyzing small spatial regions detected as features and combining large small spatial regions based on given consistency criteria (coherency criteria) You may make it judge whether it can be made into an area | region. The size of the spatial region may vary from a small group of pels to a large part that may correspond to an actual object or part of an actual object. However, the detected feature does not necessarily have to correspond to a single entity that can be distinguished from each other, such as an object or a sub-object. One characteristic may include each element (component) of two or more objects, or may not include any element of the object. An important aspect of the feature according to the present invention is that the feature model-based compression method can efficiently compress a set of pels constituting the feature as compared with the conventional compression method.

  The consistency criteria when combining small areas into a large area may include motion similarity, appearance similarity after motion compensation, and encoding complexity similarity. Consistent motion can be found by higher order motion models. In one embodiment, the translational motion of each small area can be incorporated into the affine motion model. With this model, it is possible to approximate the motion model of each small area. For a small set of regions, if those movements can always be incorporated into the aggregate model, this suggests that the small regions are dependent and consistent. Such consistency can be effectively utilized by the aggregate feature model.

<Formation of feature model>
It is important to correlate multiple instances of the same feature after detecting the feature in multiple video frames. This process is referred to as “feature correlation” and is the basis for feature tracking (determining the location of a particular feature over time), as described below. However, to effectively perform this feature correlation process, it is first necessary to define a “feature model” that is used to distinguish similar feature instances from dissimilar feature instances.

  In one embodiment, features may be modeled using the feature pel itself. The feature pel region is two-dimensional and can be vectorized. Similar features can be identified by minimizing the mean square error (MSE) between pel vectors of different features or by maximizing the dot product between the pel vectors of different features. Problems of this configuration include that the feature pel vector is sensitive to small-scale changes in features such as translation, rotation, enlargement / reduction, and further to changes in illuminance of features. Since features frequently undergo such changes throughout the video, such changes need to be considered when features are modeled and correlated using a vector of feature pels. In one embodiment of the present invention, a very simple method of applying a standard motion prediction and compensation algorithm to account for the translational motion of features found in conventional codecs (eg, H.264, etc.) Consider the above-described changes in features. In other embodiments, more complex methods may be used to account for feature rotation, enlargement / reduction, and illumination changes between frames.

  In an alternative embodiment, the feature model is a compact representation of a feature (for a given type of transformation) that is “invariant” to small-scale rotation, translation, scaling, and possibly illumination changes of the feature. (Here, “compact” means that the dimension is lower than the dimension of the original feature pel vector). That is, even if the feature undergoes a small change between frames, the feature model in this case remains relatively constant. Such compact feature models are often referred to as “descriptors”. As an example, in one embodiment of the present invention, the length of the SURF feature descriptor is 64 based on the sum of the Haar wavelet transform response (as opposed to the length of the feature pel vector is 256). ). In another embodiment, a five bin color histogram is constructed from the feature pel color map, and the five component histogram serves as a feature descriptor. In yet another embodiment, the feature region is transformed by two-dimensional DCT. Then, the two-dimensional DCT coefficients are summed over the upper triangular portion and the lower triangular portion of the coefficient matrix. This sum constitutes an edge feature space and can serve as the feature descriptor.

  When features are modeled using feature descriptors, similar features can be obtained by minimizing the MSE between feature descriptors or by maximizing the inner product between feature descriptors (instead of a vector between pels of features) Can be identified.

<Feature correlation (feature association)>
The next step after detecting and modeling features is to correlate (match) similar features across multiple frames. Each feature instance that appears in each frame is a sample of the appearance of that feature. Multiple feature instances are considered to “belong” to the same feature by being correlated across multiple frames. A plurality of feature instances correlated to belong to the same feature may be aggregated to form a feature track, or may be collected in an aggregate matrix 40 (FIG. 1A).

  A “feature track” is defined as the feature location (x, y) relative to the video frame. In one embodiment, the newly detected instance of the feature is associated with the tracked feature (in the case of the first frame of the video, associated with the detected feature or a previously detected feature). Based on this, it is determined which of the feature tracks that have been constructed so far the feature instance in the current frame belongs to. Feature tracking is performed by associating feature instances in the current frame with previously constructed feature tracks (in the case of the first frame of a video, with detected features or previously detected features.

  FIG. 1B shows how features 60-1, 60-2,..., 60-n are tracked using feature tracking means (feature tracker) 70. FIG. The feature detection unit 80 (for example, SIFT, SURF, etc.) is used to identify the feature in the current frame. The feature instance detected in the current frame 90 is compared with the detected (or tracked) feature 50. In one embodiment, prior to the correlation process described above, Harris and Stephens' corner detection algorithm [Harris, Chris and Mike Stephens, 1988, "A combined corner and edge detector," in Proc. Of the 4th Alvey Vision Conference, pp. 147-151], the autocorrelation analysis (ACA) amount representing the feature strength based on the autocorrelation matrix of the feature by calculating the image gradient of the feature autocorrelation matrix by the differentiation of the Gaussian filter May be used to determine the rank in the set of feature detection candidates in the current frame. Feature instances with large ACA amounts are preferred as track extension candidates. In one embodiment, if a feature instance that is ranked low in the ACA ranking list is located within a given distance (eg, 1 pel) of feature instances that are ranked high in the list, Removed from feature set.

  In various embodiments, a feature descriptor (eg, SURF descriptor, etc.) or a feature pel vector may serve as a feature model. In one embodiment, tracked tracks (regions 60-1, 60-2,..., 60-n in FIG. 1B) are tracked one by one from the newly detected features in the current frame 90. (Follow-up of the tracking) is examined. In one embodiment, the most recent feature instance of each feature track is the focus of the search for track extensions in the current frame (ie, “target feature”). In the current frame, all feature detection candidates within a given distance (eg, 16 pels) of the target feature location are examined, and the candidate with the smallest MSE for that target feature is the feature track. Selected for extension. In another embodiment, feature candidates whose MSE for a target feature exceeds a given threshold are excluded as not eligible for track extension.

  In a further embodiment, if there are no feature detection candidates eligible for extension of a given feature track in the current frame, H. A limited search to find a matching region is performed using motion compensated prediction (MCP) in H.264 or general motion prediction and compensation (MEC). Both the MCP and the MEC perform a gradient descent search to search for matching regions in the current frame where the MSE for the target feature in the past frame is minimized (satisfying the MSE threshold). If no match for the target feature is found in the current frame from either the feature detection candidate or the MCP / MEC search process, the corresponding feature track is determined to be “invalid” or “finished”. .

  In a further embodiment, for two or more feature tracks, if each feature instance in the current frame matches more than a given threshold (eg, 70% overlap), those feature tracks Remove or exclude all but one of these from future consideration. With this deletion process, it is possible to maintain the feature track having the longest history and the largest total ACA amount that is the sum of all feature instances.

  In one embodiment of the present invention, as a combination of the above processes, SURF feature detection, ranking of ACA-based feature candidates, and feature correlation by minimizing MSE of feature candidates performed with assistance of MCP / MEC search method And apply. Hereinafter, such a combination is referred to as a feature point analysis (FPA) tracker (tracking means).

  In another embodiment of the present invention, macroblocks within a video frame are considered as features and H.264 MCP engine registers features / macroblocks, The feature / macroblock is correlated using an H.264 interframe prediction amount (such as the sum of absolute differences of conversion differences (SATD)). Hereinafter, such a combination is referred to as a macroblock cache (MBC) tracker (tracking means). This MBC tracker is standard in that it has different specific parameters (eg, it can perform a broader match search because the search boundary is disabled), and the specific configuration of the matching process is different. It is distinguished from simple interframe prediction. In the third embodiment, the SURF detection result is associated with a neighboring macroblock, and The macroblock is correlated and tracked using H.264 MCP engine and inter-frame prediction engine. Hereinafter, such a combination is referred to as a SURF tracker (tracking means).

  In an alternative embodiment, multiple feature instances are collected in an aggregate matrix for further modeling. Feature instances in the form of regions 30-1, 30-2,..., 30-n as shown in FIG. 1A are correlated and identified as representing the same feature. The pel data from these areas can then be vectorized and placed in the aggregate matrix 40. The entire aggregate matrix 40 represents the feature. By collecting a sufficient number of samples into a collection, it is possible to model the appearance of the feature not only in the frame in which the feature is sampled but also in the frame in which the feature is not sampled. It becomes possible. The number of dimensions of the “feature appearance model” is the same as the number of dimensions of the feature, and is different from the above-described feature descriptor model.

  The collection of areas may be spatially normalized (adapted to a given criterion by removing the cause of variation) around a single key area within the collection. In one embodiment, the area closest to the geometric centroid of the aggregate is selected as the key area. In another embodiment, a feature that exists early in the collection (a feature that has a long duration in the collection) is selected as the key area. US Patent No. 7,508,990, US Patent No. 7,457,472, US Patent No. 7,457,435, US Patent No. 7,426,285, US Patent No. 7,158,680, US Patent No. 7,424,157, US Patent No. 7,436,981, US Patent Application No. And US patent application Ser. No. 12 / 121,904, the deformations necessary to perform such normalization are collected as a deformation aggregate and the normalized image is modified. Collected as an appearance aggregate. The entire teachings of these patents and patent applications are incorporated herein by reference.

  In this embodiment, the appearance aggregate is processed to provide an appearance model, and the deformation aggregate is processed to provide a deformation model. A combination of the appearance model and the deformation model is a feature model of this feature. By using this feature model, features can be represented by a compact set of parameters. In one embodiment, the ensemble matrix is singular value decomposed (SVD) and a rank reduction method is applied to it to ensure that only a subset of singular vectors and the corresponding singular values are maintained. Is formed. In a further embodiment, the condition of the rank reduction method is that the aggregate matrix reconstructed by applying the rank reduction method is a complete set before reconstruction within an error threshold range based on a 2-norm of the aggregate matrix. It is assumed that a sufficient number of major singular vectors (and corresponding singular values) are maintained to approximate the body matrix. In an alternative embodiment, an orthogonal matching tracking (OMP) method [Pati, YC et al., 1993, “ Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition, "in Proc. Of the 27th Asilomar Conference, pp. 40-44]. Also in this case, a sufficient number of aggregate vectors (and corresponding OMP weights) can be maintained so that the reconstruction result after application of the OMP method satisfies an error threshold based on the 2-norm of the aggregate matrix. As will be described later, the feature appearance model and the deformation model thus formed may be used for feature-based compression.

  The collection of features can be improved by comparing the members of the collection with each other. In one embodiment, the collection is improved by thoroughly comparing each sampled area (each sampling area) (each vector of the collection) with other sampling areas. In this comparison, two tiles are registered. In the first registration, the first area is compared against the second area. In a second registration, the second area is compared against the first area. Such registration is performed for each image at the positions of the first and second regions in each image. The registration offset obtained in this way is kept together with the corresponding positional offset. These are called correlations. By analyzing this correlation, it is determined whether it is desirable to change the position of the sampling region in view of a plurality of registration results. If the changed position in the source frame results in a less error match for one or more areas in other frames, adjust the position of those areas to the changed position. To do. Thus, the selection of the position after the change when changing the area in the source frame is performed by linearly interpolating the position of the area in another frame corresponding to the time extension of the area in the source frame. Executed.

<Feature-based compression>
By using feature modeling (or general data modeling), compression can be improved over conventional codecs. In standard inter-frame prediction, block-based motion prediction / compensation is used to find a prediction of each coding unit (macroblock) from a limited search space of a decoded reference frame. If a thorough search is performed and a good prediction is made with all past reference frames, the computation load becomes too large. In contrast, detecting and tracking features through video can navigate more predictive search spaces without overloading the computation load, thus making it possible to generate better predictions. . Since the feature itself is a kind of model, hereinafter the terms “feature base” and “model base” are used interchangeably.

  In one embodiment of the invention, feature tracks are used to associate features with macroblocks. FIG. 1C illustrates this general process. A given feature track indicates the location of the feature over multiple frames. And the feature has movement over a frame. By using the position of the feature in the two most recent frames as viewed from the current frame, the position of the feature in the current frame can be inferred. A corresponding nearest macroblock exists at the estimated position of the feature. Such a macroblock is defined as a macroblock that most overlaps with the estimated position of the feature. Therefore, this macroblock (the target macroblock being encoded) is associated with a specific feature track. The estimated position of this particular feature track in the current frame is in the vicinity of the macroblock (step 100 of FIG. 1C).

  The next step is to calculate an offset between the target macroblock (target macroblock) and the estimated position of the feature in the current frame (step 110). By using this offset, as well as past feature instances in the associated feature track, predictions for the target macroblock can be generated. Such past instances are included in the local (nearby) cache 120 where the recent reference frame in which the feature appeared is stored, or the “old” reference frame 150 in which the feature appeared Included in stored long-term (distant) cache. By finding an area in the reference frame where the offset from the past feature instance in the reference frame is the same as the offset between the target macroblock and the estimated position of the feature in the current frame (steps 130 and 160), A prediction for the target macroblock can be generated.

<Generation of model-based primary and secondary predictions>
In one embodiment of the invention, feature-based prediction is performed as follows: (1) detecting features for each frame; (2) modeling detected features; (3) phase Correlate features in different frames to generate feature tracks; (4) use feature tracks to predict the location of features in the current “current” frame being encoded; (5) current Associating a macroblock that exists near the predicted position of that feature in the frame; and (6) a past location along the feature track of the associated feature (correlated feature) to the macroblock in (5) above Generate a prediction based on

  In one embodiment, features are detected using the previously described SURF algorithm and correlated and tracked using the previously described FPA algorithm. After feature detection, correlation and tracking, each feature track can be associated with the nearest macroblock as described above. In one embodiment, if multiple features can be associated with one macroblock, the feature that most closely overlaps the macroblock is selected as the feature associated with the macroblock.

  Given a target macroblock (the current macroblock being encoded), its associated features, and a feature track for that feature, a primary prediction (or key) for that target macroblock is given. Prediction) can be generated. The key prediction data (pel) is acquired from the most recent frame in which the feature appears (as viewed from the latest frame). Hereinafter, this latest frame is referred to as a key frame. Key predictions are generated after selecting a motion model and a pel sampling scheme. In one embodiment of the invention, the motion model is “zero order” which assumes that the feature is stationary between the key frame and the current frame, or the feature motion is the second most recent. May be assumed to be “primary”, which is assumed to be linear between the reference frame, the key frame, and the current frame. In either case, applying the feature motion to the macroblock in the current frame associated with the feature (in the reverse direction in time) provides a prediction for that macroblock in the key frame. In one embodiment of the present invention, the pel sampling scheme may be “direct” by rounding the motion vector to an integer (rounded to an integer) to directly extract the key prediction pel from the keyframe, or H.264. It may be “indirect” to derive motion compensated key predictions using a conventional compression method interpolation scheme such as H.264. That is, in the present invention, four different key predictions can be obtained according to the motion model (zero order or first order) and further according to the sampling scheme (direct or indirect).

  Key prediction can be improved by modeling local deformation components using a subtiling process. In the subtiling process, motion vectors are calculated for different local parts of the macroblock. In one embodiment, the subtiling process may be performed by dividing a 16 × 16 macroblock into four quadrants of 8 × 8 and calculating predictions for each separately. In another embodiment, the subtiling process may be performed by separately calculating predictions for the Y, U, and V color channels in the Y / U / V color space domain.

  In addition to the primary prediction / key prediction for the target macroblock, generate a secondary prediction based on the position of the feature associated with the target macroblock in a reference frame prior to the keyframe. Also good. In one embodiment, an offset from a target macroblock in a current frame to a feature location (estimated location) associated with the target macroblock is subtracted based on the feature location in a past reference frame. It can be used as a motion vector to find a target prediction. In this way, multiple secondary predictions (one for each frame in which the feature appears) for a given target macroblock with which the feature is associated can be generated. In one embodiment, the number of secondary predictions may be limited by limiting the number of past reference frames to be searched (for example, 25).

<Compound prediction>
After generating the primary prediction (key prediction) and the secondary prediction for the target macroblock, an overall reconstruction of the target macroblock can be calculated based on these predictions. In one embodiment, the reconfiguration is based on key prediction alone, following conventional codecs. Hereinafter, such a reconstruction is referred to as a key-only (KO) reconstruction.

In another embodiment, the reconstruction is a reconstruction based on a composite prediction that sums the key prediction and a weighted one of the secondary predictions. Hereinafter, such an algorithm is referred to as PCA-Lite (PCA-L). PCA-Lite includes the following procedures:
1. Generating a (one-dimensional) vector of target macroblocks (referred to as target vector t) and a (one-dimensional) vector of key predictions (referred to as key vector k);
2. Calculating a residual vector r by subtracting the key vector from the target vector;
3. Vectorize the set of secondary predictions to form vector s _i (assuming that these secondary vectors have unit norms without loss of generality). Next, a key subtraction set s _i -k is generated by subtracting the key vector from all the secondary vectors. This is like subtracting the projection of the key vector from the secondary vector;
4). 4. For each subvector, calculate the weighting factor c = r ^T (s _i −k); For each subvector, a composite prediction t ^{^} = k + c * (s _i -k) is calculated.

  In general, the above procedure for the PCA-Lite algorithm is similar to that of the well-known orthogonal matching tracking algorithm [Pati, 1993], but the above composite prediction is a redundant one from the primary and secondary predictions. It is intended not to include contributions. In another embodiment, in the PCA-Lite algorithm, the key vector in steps 3 to 5 described above is replaced with the average of the key vector and the secondary vector. Hereinafter, such a modified algorithm is referred to as PCA-Lite-Mean.

  The PCA-Lite algorithm described above can provide a different type of combined prediction than the bi-prediction algorithm found in some standard codecs (described in the “Background” section at the beginning). A standard bi-prediction algorithm blends multiple predictions based on the temporal distance between the reference frame used for each prediction and the current frame. In contrast, PCA-Lite mixes multiple predictions based on the “content” of each prediction to generate a composite prediction.

  Note that the composite prediction described above is possible without using feature-based modeling. That is, any prediction set can be used to generate a composite prediction for a given target macroblock. However, in feature-based modeling, the set of predictions for a given target macroblock is naturally related to each other. And by setting it as composite prediction, the information from those some prediction can be combined efficiently.

<Modeling data with multiple fidelity>
In the present invention, data can be modeled with multiple fidelity for model-based compression. FIG. 2A illustrates this embodiment. In FIG. 2A, four layers of modeling are depicted. The following table summarizes these four hierarchies. Hereinafter, these four hierarchies will be described in detail.

  The bottom hierarchy in FIG. 2A, referred to as the “macroblock” (MB) hierarchy, is a conventional compression that divides a frame into non-overlapping macroblocks (16 × 16 size tiles) or a finite set of subtiles. It is equivalent to the law. Conventional compression methods (e.g., H.264) basically do not model and use block-based motion prediction / compensation (BBMEC) from a limited search space in a decoded reference frame. Find the prediction 212 for each tile. At the decoder, the prediction 212 is combined with the macroblock (or subtile) residual code to synthesize the reconstruction of the original data (step 210).

  The second layer 202 in FIG. 2A is referred to as a “characterized by macroblock” (MBF) layer and corresponds to a compression method based on the MBC tracker described above (216 in FIG. 2A). In this hierarchy, macroblocks (macroblock subtiles) are treated as features by repeatedly applying the conventional BBMEC search method over a plurality of encoded frames. The same BBMEC as the MB layer is applied to find the conventional prediction for the target macroblock from the most recent reference frame in component 216. However, the second BBMEC application searches for the conventional further prediction relative to the conventional first prediction by searching for the second most recent reference frame in component 216. A “track” of the target macroblock (not identified as a feature) is generated by applying BBMEC repeatedly, stepping back past frames in component 216. A model 214 is generated by the MBC track, and a prediction 212 is generated by the model 214. At the decoder, this prediction 212 is combined with the macroblock (or subtile) residual code to synthesize the reconstruction of the original data (step 210).

  The third hierarchy 204 in FIG. 2A is called a “feature” hierarchy and corresponds to the feature-based compression method described above. As described above, features are detected and tracked regardless of the grid of macroblocks, these features are associated with macroblocks that overlap with the features, and the feature track is used to navigate the decoded reference frame 216. To find a good match for the overlapping macroblock. In an alternative embodiment, the codec directly encodes and decodes the feature without associating the feature with a macroblock, and features a “non-feature” background, such as by a conventional compression method of the MB layer. It can be processed separately. A prediction 212 is generated by the feature-based model 214. The decoder combines this prediction 212 with the corresponding macroblock (or subtile) residual code to synthesize the original data reconstruction (step 210).

  The top hierarchy 206 in FIG. 2A is referred to as an “object” hierarchy and corresponds to an object-based compression method. An object is essentially a large-scale feature that can contain multiple macroblocks and corresponds to something with a physical meaning (eg, face, ball, cell phone, etc.) or complex event 208. obtain. Object modeling (object modeling) can be modeled using a special basis function (step 214) if the object is expected to be of a particular type (eg, face). Parametric modeling (parametric modeling) can be performed. If an object contains multiple macroblocks or overlaps with multiple macroblocks, a single motion vector 212 can be calculated for all macroblocks corresponding to that object 216, which can be computationally intensive and encoded. Size can be saved. A prediction 212 is generated by the object-based model 214. The decoder combines this prediction 212 with the corresponding macroblock (or subtile) residual code to synthesize the original data reconstruction (step 210).

  In an alternative embodiment, an object may be identified by correlating and aggregating feature models 214 in the vicinity of the object. FIG. 2B is a block diagram showing a state of nonparametric or empirical object detection by aggregation of such feature models. A particular type of object 220 is detected by identifying features having the properties of that type of object (ie, features indicative of “object bias”) (step 222). Next, it is determined whether or not the feature set 222 exhibits the rigidity of the model state 224, that is, whether the features and the state of the features tend to correlate with time (step 224). If it is determined that each feature model has a correlation (thereby determining that an object has been detected (step 226)), a composite appearance model 228 with associated parameters and a complex deformation model with associated parameters 230 may be formed. By forming the composite appearance model and the composite deformation model, parameters are naturally reduced as compared to the case of the individual appearance model and the individual deformation model (step 232).

  FIG. 2C shows a configuration using both object-based parametric modeling and non-parametric modeling as a third embodiment of the “object” hierarchy 206 of FIG. 2A. The object is detected by a parametric model (step 240). The detected object 240 is processed to determine whether there is a feature that overlaps the object (step 250). Next, such overlapping feature sets may be examined to determine whether the features can be aggregated as described above (step 260). If it is determined that the overlapping features cannot be aggregated, the macroblocks overlapping with the object detected in the process 240 are examined, and the macroblocks are determined so as to have a common single motion vector as described above. It may be determined whether the data can be efficiently aggregated (step 270).

  In a processing architecture with multiple fidelity, the hierarchy 200, hierarchy 202, hierarchy 204, and hierarchy 206 may be combined as appropriate to achieve the best processing. In one embodiment, it is determined in which hierarchy the best (minimum amount) code for each macroblock to be encoded is obtained by examining all the layers of FIG. This “competition” will be described in detail later.

  In another embodiment, the hierarchy of FIG. 2A may be examined sequentially from the lowest hierarchy (the simplest hierarchy) to the highest hierarchy (the most complex hierarchy). If a lower layer solution is sufficient, the upper layer solution need not be examined. The criteria for determining whether a given solution is “good enough” will be discussed in detail later.

<Model-based compression codec>

<Conventional codec processing>
In the encoding process, the video data may be converted to a compressed format or an encoded format. Similarly, the decompression process may convert the compressed video to the format before it is compressed (ie, the original format). The video compression process and the video decompression process may be implemented by an encoder / decoder pair commonly referred to as a codec.

  FIG. 3A is a block diagram of a standard encoder 312. The encoder of FIG. 3A can be implemented in a software environment, a hardware environment, or a combination thereof. By way of example, such an encoder component may be implemented as code stored on a storage medium executable by at least one processor 820, as in FIG. 8A or FIG. 8B. Any combination of constituent elements may be used as the constituent elements of the encoder 312, such as an intra prediction unit 314, an inter prediction unit 316, a transform unit 324, a quantization unit 326, and entropy coding. The unit 328 and the loop filter 334 may be included, but are not necessarily limited thereto. The inter prediction unit 316 may include a motion compensation unit 318, a frame storage unit 320, and a motion prediction unit 322. The encoder 312 can further include an inverse quantization unit 330 and an inverse transform unit 332. The function of each component of encoder 312 in FIG. 3A is well known to those skilled in the art.

  The entropy encoding algorithm 328 of FIG. 3A may be an algorithm based on a probability distribution obtained by quantifying the probabilities of various numerical values of quantized transform coefficients. The coding size of the current coding unit (for example, a macroblock) depends on the current coding state (the numerical values of various quantities to be coded) and the degree of matching of the coding state with the probability distribution. . As will be described later, when the coding state changes, the coding size of the coding unit in the subsequent frame may be affected. Although it is possible to exhaustively search all the coding paths of video (that is, all possible coding states) in order to thoroughly optimize the video code, the computational load is too great turn into. In one embodiment of the invention, encoder 312 concentrates only on the latest (target) macroblock rather than considering a large range (ie, one slice, one frame or set of frames). To achieve optimization locally.

  FIG. 3B is a block diagram of a standard decoder 340 that decodes intra-prediction data 336, and FIG. 3C is a block diagram of a standard decoder 340 that decodes inter-prediction data 338. The decoder 340 can be realized in a software environment, a hardware environment, or a combination thereof. Referring to FIGS. 3A, 3B and 3C, exemplary encoder 312 receives video input 310 from the inside or outside, encodes the data, and stores the encoded data in decoder cache / buffer 348. The decoder 340 extracts the encoded data from the cache / buffer 348 and performs decoding and transmission. The decoder can access the decoded data via any available means such as a system bus or network interface. The decoder 340 may decode the video data and decompress the key frame and the prediction target frame (reference numeral 210 in FIG. 2A). The cache / buffer 348 may receive data related to the video sequence / bitstream and provide information to the entropy decoding unit 346. The entropy decoding unit 346 processes the bit stream to generate a quantized estimate of the transform coefficient of the intra prediction in FIG. 3B or a quantized estimate of the transform coefficient of the residual signal in FIG. 3C. The inverse quantization unit 344 generates an estimated value of the transform coefficient by performing inverse scaling (rescaling operation). By applying inverse transformation to the estimated values of these transform coefficients (step 342), the original video data pel intra prediction is synthesized in FIG. 3B, and the residual signal intra prediction is synthesized in FIG. 3C. In FIG. 3C, the synthesized residual signal is added to the inter prediction of the target macroblock to generate a complete reconstruction of the target macroblock. The inter prediction unit 350 of the decoder applies the inter prediction generated by the encoder by applying motion prediction (process 356) and motion compensation (process 354) to the reference frame included in the frame store (frame storage unit) 352. Duplicate. The inter prediction unit 350 of the decoder has the same configuration as that of the inter prediction unit 316 in FIG.

<Hybrid codec for model-based prediction>
FIG. 3D is a diagram illustrating an encoder of one embodiment of the present invention that performs model-based prediction. The codec 360 may encode the current (target) frame (step 362). Codec 360 may then encode each macroblock in the frame (step 364). Standard H.P. Using the H.264 encoding process, A basic (first) code that provides an H.264 coding solution is defined (step 366). In a preferred embodiment, the encoder 366 is an H.264 that can encode a GOP (a set of reference frames). H.264 encoder. Preferably, H.M. The H.264 encoder can be set so that various methods can be applied to encode pels in each frame. Examples of such methods include intra-frame prediction and inter-frame prediction. Multiple reference frames can be searched to find a good match for the macroblock being encoded. Preferably, the error between the original macroblock data and the prediction is transformed, quantized and entropy coded.

  Preferably, encoder 360 provides a context sensitive adaptive mechanism for context modeling by utilizing a CABAC entropy coding algorithm (step 382). Such context modeling is applied to binary sequences of video data syntax elements (eg, block types, motion vectors, quantized coefficients, etc.) using the binarization process of a given mechanism. obtain. Next, each element is encoded using an adaptive or fixed probability model. You may make it adjust a probability model suitably using a context value.

<Competition mode>
In FIG. The macroblock code according to H.264 is analyzed (step 368). In step 368, H.P. If the macroblock code according to H.264 is determined to be “efficient”, the H.264 H.264 solution is close to ideal and without further analysis, the target macroblock is Select H.264 encoding solution. In one embodiment, H.264. The encoding efficiency according to H.264 is H.264. H.264 encoding size (bits) can be determined by comparing with a threshold. Such a threshold can be derived from the percentile statistics of a previously encoded video or can be derived from previous percentile statistics of the same video. In other embodiments, H.264. The encoding efficiency according to H.264 is H.264. A “skip” macroblock, which can be determined by whether the H.264 encoder has determined that the target macroblock is a “skip” macroblock, is that the data inside and around it substantially requires additional encoding. A macroblock that is sufficiently uniform that it does not.

  In step 368, H.P. If the macroblock solution according to H.264 is not determined to be efficient, additional analysis is performed and the encoder transitions to a race mode 380. In this mode, a plurality of various predictions of the target macroblock are generated based on the plurality of models 378. Model 378 is generated by identifying the features detected and tracked in past frame 374 (step 376). Each time a new frame 362 is processed (encoded and decoded and stored in the frame store), it takes into account the detection of new features in the new frame 362 and the corresponding extension of the feature track. The model needs to be updated. The model-based solution 382 is the acquired H.264. It is ranked based on the coding size 384 with the H.264 solution. Thus, based on the degree of freedom that a given macroblock can be encoded by either basic coding (H.264 solution) or model-based coding, the codec according to the present invention is defined as a hybrid codec. Can be called.

  For example, in competitive mode, H.264 generates a code for the target macroblock and compares its compression efficiency (ability to encode data with fewer bits) to other models. For each coding algorithm used in the competition mode, perform the following steps: (1) generate a prediction based on the codec mode / algorithm used; (2) subtract the prediction from the target macroblock to obtain a residual Generate a signal; (3) transform the residual (target-prediction) using block-based two-dimensional DCT approximation; and (4) encode transform coefficients by entropy encoding. To do.

  In some aspects, H.264 (interframe) baseline prediction can be said to be a prediction based on a relatively simple limited model (H.264 is considered one of several algorithms used in competitive mode) . However, prediction based on a more complex model (a feature-based model or an object-based model) and corresponding tracking (tracking) may also be used for the prediction of the encoder 360. The encoder 360 operates on the assumption that a feature-based compression method will yield better results than a conventional compression method when a macroblock indicating data complexity is detected.

<Use of feature-based prediction in competitive mode>
As described above, first, for each target macroblock, H. Determine if the H.264 solution (prediction) is efficient (“good enough”). When this determination result is negative, the mode is shifted to the competition mode.

In the competition mode 380 of FIG. 3D, “entry” to the competition is determined by appropriately selecting various processing options (see the above description) when performing feature-based prediction. Each entry makes a different prediction for the target macroblock. In the feature-based prediction according to the present invention, the following processing options can be specified:
-Types of trackers (tracking means) (FPA, MBC, SURF),
-Motion model used for key prediction (zero order or first order)
-Sampling scheme used for key prediction (direct or indirect)
-Sub-tiling scheme used for key prediction (no sub-tiling, 1/4 division, Y / U / V)
A reconstruction algorithm (KO or PCA-L) and a reference frame used for secondary prediction (in the case of PCA-L).

  The search space for the macroblock solution for a given target includes H.264. In addition to the H.264 solution (the “best” inter-frame prediction in H.264), all kinds of feature-based prediction according to the present invention described above may be included. In one embodiment, the competition mode includes any combination of the above processing options (tracker type, motion model used for key prediction, sampling scheme used for key prediction, subtiling scheme and reconstruction algorithm). In another embodiment, the processing options in the competition mode are configurable and can be limited to a sufficient number of subset combinations to save computational effort.

  The solution candidates in the competition are evaluated one by one by the following four procedures (similar to the procedure described above): (1) generate a prediction; (2) subtract the prediction from the target macroblock. A residual signal; (3) transform the residual; and (4) encode the transform coefficient by entropy encoding. The output from step 382 of FIG. 3D is the number of bits associated with a given solution 384. As each solution is evaluated, the encoder is rolled back to the state prior to the current evaluation so that the next solution can be evaluated. In one embodiment, after evaluating all solutions, the “winner” of the competition is selected (step 370) by selecting the solution with the smallest coding size. The winner's solution is then sent back to the encoder as the final code for the target macroblock (step 372). As described above, since the winner's solution is a solution optimized only for the target macroblock, it can be said to be a locally optimal solution. In an alternative embodiment, the optimal solution is selected based on whether a wider trade-off can be mitigated. Such trade-offs may include, but are not necessarily limited to, the effects of contextual intra-frame prediction feedback, the effects of residual errors, etc. on subsequent frames.

  Information about the winner's solution is saved in the encoded stream (step 386) and transmitted / stored for future decoding. This information can include, but is not necessarily limited to, processing options used for feature-based prediction (eg, tracker (tracking means) type, key calculation, subtiling scheme, reconstruction algorithm, etc.).

  In some cases, the target macroblock is H.264. In addition to the encoder 360 determining that it is not efficiently encoded in H.264, it is also possible that no features that overlap the macroblock are detected. In such a case, the encoder uses H.264 as a last resort. The macroblock is encoded using H.264. In an alternative embodiment, feature-based prediction may be generated by extending the track of the feature tracking means (feature tracker) to generate a pseudo-feature that overlaps the macroblock.

  In one embodiment, movement between the four tiers of FIG. 2A is managed in the competitive mode.

<Decoding using feature-based prediction>
FIG. 4 is a diagram illustrating an example of a decoder according to an embodiment of the present invention that can implement model-based prediction within the Euclidean Vision codec by the applicant of the present application. The decoder 400 synthesizes an approximation of the input video frame that is the basis of the frame code 420 by decoding the encoded video bitstream. Frame code 420 may include a set of parameters that decoder 400 uses to reconstruct the corresponding video frame 418.

  The decoder 400 scans each frame in the same slice order as the order adopted by the encoder. The decoder scans each slice in the same macroblock order as the order adopted by the encoder. According to the process in the encoder, the decoder, for each macroblock 404, decides whether to decode the macroblock in the conventional manner (step 408) or whether to decode using the feature model and parameters (step 416). to decide. If the macroblock is encoded by model-based prediction according to the present invention, the decoder 400 may provide any feature information (feature track, feature reference frame [) necessary to reproduce the prediction in that solution (step 418). GOP], a feature motion vector). Also, the decoder updates the feature model at the time of decoding (steps 410, 412, 414) and synchronizes with the feature state on the encoder side for the frame / slice / macroblock being processed.

  In the conventional codec, due to memory limitations, the entire prediction context for the decoded frame cannot be held in the frame store 352 and the cache 348 in FIG. 3C, and only that frame (pel) is held. It was general. In contrast, the present invention allows the prediction context stored in the frame store 352 and cache 348 of FIG. 3C to be expanded by prioritizing retention of feature-based models and parameters.

  The entire set of parameters representing a feature model is called a feature state (feature state). In order to maintain the feature model efficiently, it is necessary to isolate this feature state. FIG. 5 is a block diagram illustrating a feature instance state extraction process 500 in one embodiment of the invention. This state extraction information may be associated with the target macroblock. This state extraction information may also include parameters corresponding to the related feature instance 502. Such parameters can be useful for encoding the target macroblock. Moreover, it is also possible to interpolate the predicted feature in subsequent video frames using this state extraction information. Each feature instance has a corresponding GOP 504. Each GOP includes corresponding state information (for example, corresponding boundary information). The state extraction information for each feature instance may further include state information for every object associated with the feature instance, state information 506 for the corresponding slice parameter, and state information 508 for the corresponding entropy state. Thus, the state information can provide a description of the GOP / slice / entropy parameter boundary of the feature instance and a description of the boundary extension to a new state and new context. By using the state information 506 and 508, it is possible to predict and interpolate the state of the predicted feature in subsequent frames.

  An extended prediction context is formed by macroblock data (pel) and feature state extraction information associated with the macroblock data. Extended contexts from multiple feature instances may be combined with decoded neighbors. The enhanced prediction context used by encoder 312 in FIG. 3A and decoder 340 in FIGS. 3B and 3C includes: (1) at least one macroblock; (2) at least one neighboring macroblock; (3) slice information; 4) a reference frame [GOP]; (5) at least one feature instance; and (6) object / texture information;

<Parametric model-based compression>

<Integration of parametric modeling into codec framework>
In the above hybrid codec aspect, the feature model is implicitly used to give the encoder a clue about good prediction of the macroblock. In contrast, the feature model can also be explicitly used in the codec framework. When a particular region in a target frame is represented by a given type of model (eg, a face model, etc.), the representation depends on the model parameters. Hereinafter, this type of explicit modeling is referred to as parametric modeling. On the other hand, the hybrid codec aspect described above uses non-parametric modeling (non-parametric modeling) or empirical modeling. Parametric modeling is done in anticipation of the existence of a particular type of feature or object (eg, face, etc.), so it usually consists of a set of basis vectors that span the space of every feature / object of that type. . The model parameter in this case is the projection of the target area onto the basis function.

  FIG. 6A is a block diagram illustrating an example of components of a codec 600 that implements parametric modeling in an alternative embodiment of the present invention. As shown in FIG. 6A, the codec 600 includes means 610 for performing adaptive motion compensated prediction and / or means 612 for performing adaptive motion vector prediction and / or means 614 for performing adaptive conversion processing and / or adaptive. Type entropy encoding means 616 may be included.

  The adaptive motion compensation prediction means 610 may select the reference frame 618 based on the fact that the feature instance is included. When compression efficiency is improved by modeling a feature, a frame from which the model is derived may be selected as a reference frame, and a corresponding GOP may be generated. Interpolation of motion vector offset 626 may be performed based on detected feature parameters. Thereby, a new data pel of the instance of the feature to be predicted can be constructed within a discrete set of known data points based on the detected feature. The result of the subtile division processing 612 used in the conventional encoder is supplemented by the restriction of the deformation change model 620. The conversion process 614 may be performed using the appearance change modeling 622 to constrain appearance change parameters. Entropy encoding process 616 may be supplemented by parameter range / scale analysis 624 and adaptive quantization 628 of codec 600 according to the present invention. The macroblock auxiliary data 630 obtained in this way is output by the codec 600.

<Improvement of hybrid codec by adaptive coding using parametric modeling>
In one variation, the prediction by the hybrid codec described above can be improved by using parametric modeling. In one embodiment, it is determined whether the prediction can be improved by applying an element of the parametric model to a prediction obtained in advance for the target macroblock (eg, the output of the competitive mode).

  FIG. 6B shows an example of an application of a parametric model based adaptive encoder 634. Adaptive encoder 634-1 may supplement the encoding performed by a conventional codec (eg, H.264, etc.) or a hybrid codec as described above. The pel residual 636 obtained by the conventional motion compensation prediction process is analyzed (step 638), and the deformation change and the appearance change of the residual can be modeled more efficiently with the parametric feature model (step 642). Determine whether. In one embodiment, the relative efficiency of the parametric model may be determined by whether or not the transformed difference absolute value sum (SATD) 640 between the prediction residual 636 and the parametric model 638 decreases. When the parametric model is determined to be an efficient expression, the target parameter (macroblock) is projected onto the feature model (appearance basis and deformation basis), and the feature parameter that functions as the sign of the residual signal is obtained. Can be obtained.

  In this embodiment, an additional rollback function is also provided that checks whether alternative residual modeling can be applied within the current GOP state, slice state, and entropy state. For example, in a series of video frame sequences, reference frames, GOPs and features (slices) 646 that are located far from the current frame being encoded can be considered as criteria for prediction. Such an approach is not practical with conventional encoding. Furthermore, if compression is improved with a feature model from another video file, it is possible to roll back to another video data such as such a video file.

<Feature-based prediction by interpolating parametric model parameters>
When multiple instances of the same feature appear in the video stream, it is desirable to maintain the invariant component of the feature model (a component that does not change between frames). In parametric feature modeling, specific parameters of the feature model (for example, coefficients representing weights of various basis functions) are invariant components. In general, in non-parametric (empirical) feature modeling, the feature pel itself is an invariant component. Maintaining the invariant component of the model when performing feature motion prediction / compensation may be a guideline principle of motion prediction / compensation (hereinafter referred to as “invariant principle”).

  FIG. 6C is a block diagram illustrating a state in which motion compensation prediction of a feature is performed by interpolation of a feature model parameter using the invariant principle as a guideline in an embodiment of the present invention. As shown in FIG. 6C, the motion compensated prediction process 668 begins with a normalization process that adjusts model parameters of multiple feature instances around an invariant instance of the parameters. Using a set of feature instances (“matched macroblocks”) 670 to generate multiple types of interpolation functions (674, 676, 678, 680) to normalize the instances around invariant instances can do. An invariant instance 682 of model parameters may be defined as a set of model parameter values at a key frame. Such invariant instances can represent most (if not all) predictions / patterns in a feature-based model. An invariant instance is similar in concept to the centroid of a vector space formed by a vector of instance appearance parameters.

  The invariant instance 682 can be a key pattern for extrapolating the target position 684 using one of the interpolation functions (674, 676, 678, 680). By using such an interpolation / extrapolation process, it is possible to predict the in-frame position, appearance change, and deformation change of the feature in the target frame. By combining such an invariant representation of features and a compact parameter form of feature instances, the amount of memory required to store the appearance and deformation of features contained in the reference source frame in a cache is reduced to that of conventional compression methods. It can be dramatically reduced in comparison. That is, with such a feature model, it is possible to succinctly collect data that is important and useful for compression out of frame data.

  As an alternative embodiment, use feature model parameters for at least two feature instances, given the time interval between the reference frame in which they appeared and the current (target) frame. Thus, the state of the target area can be predicted. In this case, based on a given state model and time step, the feature parameters of the target region can be predicted by extrapolating at least two feature parameters according to the invariant principle. The state model in this case may be a linear model or a higher-order model (for example, an extended Kalman filter).

<Cache organization and access of feature model information>
During generation of a feature model, multiple instances of the same feature are often found in the video. At this time, the feature model information can be efficiently stored or cached by organizing the feature model information before storing it in the cache. This approach can be applied to both parametric model-based compression schemes and non-parametric model-based compression schemes.

  For example, in FIG. 3C, the cache 348 (including the frame store 352) is configured to store prediction context information based on feature-based modeling when it is determined that compression efficiency is improved with prediction context information based on feature-based modeling. be able to. When the feature-based prediction context information is not stored in the cache, an attempt to access the feature-based prediction context may cause overhead, which may reduce the responsiveness and determination performance of the system. By storing the processed feature-based encoding prediction context in the cache, such overhead can be suppressed. With such a configuration, the frequency of access to data related to the feature-based prediction context can be reduced.

  As an example, the cache of the encoder 312 / decoder 340 (FIGS. 3A and 3C) may be a cache configured to improve the execution speed and efficiency of video processing. Even if the encoded video data is video data that is not spatially close to the frame from which the prediction data of feature-based encoding is derived, the feature-based encoding is performed in the vicinity of the encoded video data in the cache. Depending on whether the prediction data can be stored, the performance of the video processing can vary. The proximity of the cache can affect the access latency, operation delay, and data transmission time. For example, if feature data from a large number of frames is stored in a small amount of physical memory and can be accessed in that form, the frames from which those features are derived are stored in a permanent storage device and accessed there. Much more efficient than Also, the encoder 312 / decoder 340 (FIGS. 3A, 3C) allows the prediction data to easily access feature-based prediction context information in the cache / buffer / frame store when a macroblock or frame is decoded. May be included in the cache. A configurator (setting unit / setting unit) may be included.

  In a particular embodiment of the invention, first, by defining two types of feature correlations for the decoded frame, ie, locally decoded data to be stored in the cache and non-locally decoded data; By defining these two types, the cache can be expanded. A local cache may be a collection of decoded frames that are accessible in batch form (ie, in the form of a group of frames). The detected features determine the frames that make up such a group. The local cache is activated by the features detected in the current frame. Local cache is often used when there are few “strong” feature models (long history models) in the current frame / macroblock. The local cache processing is processing based on batch-type motion compensation prediction, and a group of frames is stored in a buffer of reference frames. FIG. 7A is a block diagram illustrating an overview of an example of a cache architecture 710-1 according to an embodiment of the present invention. Cache access architecture 710-1 includes local cache access 712 (716, 718, 720, 722, 724) and long-term (non-local) cache access 714 (726, 728, 730, 732). And a determination process 710. If most features are local (step 712) (eg, if there are few “strong” feature models in the current frame / macroblock), local cache processing is performed (step 718).

  FIG. 7B is a block diagram illustrating processing associated with the use of local (short-term) cache data 734. A local cache may be a collection of decoded frames that are accessible in batch form (ie, in the form of a group of frames). The detected features determine the frames that make up such a group. The local cache 734 of FIG. 7B groups only “short history” features, ie, features of feature tracks that only span a few frames. The aggregated set of frames encompassed by such “short history” features defines a joint frameset 738 for those features. The priority of the frames in the joint frame set 738 may be determined based on the complexity of the frame track of each frame. In one embodiment, such complexity is H.264. It can be determined by the cost of encoding the features by a basic encoding process such as H.264. 3B, 3C, 7A and 7B, the local cache may be stored / stored in the frame store 352 or the cache buffer 348. The locally stored frame is utilized in step 720. Next, a GOP / batch 742 based on the detected feature instance is utilized at 722. The GOP / batch 742 based on the detected feature instance can then be tested as a reference frame for the motion compensated prediction process (step 724). The motion compensated prediction performed in this way can be regarded as being “biased” in the feature tracking information because the motion compensation is performed using the frame in which the feature instance is detected as a reference frame. In addition, an additional rollback is provided (step 746) to see if residual modeling is possible within the GOP / batch state, slice state, and entropy state. This makes it possible to efficiently evaluate a reference frame located far from the current frame being encoded in the video frame sequence.

  Thus, in certain embodiments of the invention, past frames can be analyzed to determine the frame that has the highest probability of producing a match for the current frame. Further, the number of reference frames is much larger than the typical upper limit number of frames such as 1 to 16 in the conventional compression method. If there are a sufficient number of reference frames containing useful matches, depending on system resources, the number of such reference frames may reach the memory limit of the system. Furthermore, the amount of memory required to store the same number of reference frames can be reduced by the intermediate data generated in the present invention.

  Referring again to FIG. 7A, the majority of features 726 with long history are stored in a non-local / long-term cache. The non-local cache is a cache based on two types of cache access methods, “frame” and “hold”. Non-local cache "frame" access generates a feature model for encoding the current frame by accessing the frame directly. In “hold” mode, the decoded data is not directly accessed, but is stored as data previously derived from the decoded frame (the feature model in the decoded frame and the parameters of the instance in the feature model). Use feature models. As a result, even in the “hold” mode, the same data as in the “frame” mode can be synthesized. Specifically, the model of the feature instance is accessed (step 728). A reference frame is accessed (step 730). The optimal reference frame and model combination is marked (step 732). Intermediate feature information (including feature strength and feature bandwidth) of the feature model in each reference frame can be used as the criterion for whether or not it is optimal.

  The long-term cache 714 may be any data as long as it is decoded data (or encoded data), and is preferably accessible in a decoder state. Long-term cache 714 may include, for example, a reference frame / GOP. The reference frame / GOP is generally a plurality of frames preceding the current frame being encoded. In addition to such frame combinations, the decoder's long-term cache can store any combination of decoded frames that can be used to decode the current frame.

  FIG. 7C is a block diagram illustrating processing associated with long-term use of cache data. Long-term (non-local) cache 748 has a longer range cache architecture. If the detected feature instance occurs repeatedly multiple times and the corresponding relationship model of the feature can be repeatedly applied, and it is determined that the feature has a long history (step 752), the long-term cache is Initialized from the local cache (step 750). Next, the process determines which “hold” mode to use (step 754). There are two types of non-local cache modes: “retain” 760 and “non-retain” 756. “Non-retained” 756 compensates for the conventional motion compensated prediction process (as well as the use of implicit modeling in the hybrid codec described above) by prediction based on feature models. Therefore, in the “non-hold” mode 756, a valid prediction is obtained by accessing the reference frame (reference numeral 758). The “hold” mode differs from the “non-hold” mode in that it uses predictions explicitly obtained from the feature model (steps 762, 766). Therefore, in the “holding” mode, the prediction space is necessarily limited to only feature data that can be synthesized using the feature model. The feature model may also include instance parameters of feature instances in the past frame (equivalent to pels contained in the past frame). By interpolating such a function describing the parameters, the prediction is provided to the motion compensated prediction process to assist in frame synthesis (step 764).

  In the present invention, in some embodiments using a feature set (feature set), encoding is performed using feature information stored in a cache. In such an embodiment, a subset of the feature aggregate is used to represent (model) the entire aggregate. As described above, such a subset is selected by using, for example, SVD. The sub-space of the feature instance selected in this way becomes the basis of the aggregate and is cached so that the feature can be encoded each time a corresponding feature appears in a subsequent frame of the same video (or other video). Can be stored and used. With such a subset of feature instances, features can be modeled compactly and accurately.

<Digital processing environment and communication network>
Embodiments of the present invention can be implemented in a software environment, a firmware environment, or a hardware environment. In one embodiment, such an environment is shown in FIG. 8A. The at least one client computer / device 810 and the cloud (or server computer or group thereof) 812 may implement a processing function for executing an application program, a storage function, an input / output device, and the like. At least one client computer / device 810 is connectable to another computing device (including another client device / process 810 and at least one other server computer 812) via a communication network 816. Communication network 816 may be part of a remote access network, part of a global network (eg, the Internet, etc.), part of a worldwide collection of computers, part of a local area network, part of a wide area network, or It can be part of a gateway that communicates with each other using various protocols (TCP / IP, Bluetooth (registered trademark), etc.). Other electronic device / computer network architectures can also be used.

  FIG. 8B is a diagram illustrating the internal structure of a given computer / computing node (eg, client processor / device 810, server computer 812, etc.) in the processing environment of FIG. 8A. Each computer 810, 812 includes a system bus 834, which is a set of real or virtual hardware lines used for data transfer between components of the computer (or processing system). The bus 834 is like a shared pipe that connects different components (for example, a processor, a disk storage, a memory, an input / output port, etc.) of a computer system, and exchanges information between these components. to enable. An I / O device interface 818 for connecting various input / output devices (for example, a keyboard, a mouse, a display, a printer, a speaker, etc.) to the computers 810 and 812 is attached to the system bus 834. Computers 810, 812 can connect to various other devices attached to a network (eg, network 816 in FIG. 8A) via network interface 822. Memory 830 is a volatile memory that stores computer software instructions 824 and data 828 used to implement an embodiment of the present invention (eg, codec, video encoder / decoder, etc.). Disk storage 832 is a non-volatile storage that stores computer software instructions 824 (equivalent to “OS program” 826) and data 828 used to implement one embodiment of the present invention. The disk storage 832 can also be used for long-term storage of video in a compressed format. A central processing unit 820 for executing computer instructions is also attached to the system bus 834. Throughout this specification, “computer software instructions” and “OS programs” are equivalent to each other.

  In one embodiment, the processor routines 824 and data 828 are computer program products (generally designated 824) that provide at least some of the software instructions for the system according to the present invention. The computer program product 824 includes a computer readable medium that can be stored in the storage device 828. Computer program product 824 may be installable by any suitable software installation method known in the art. In other embodiments, at least some of the software instructions may be downloadable via cable and / or communication and / or wireless connection. In still another embodiment, the program according to the present invention is transmitted by a propagation signal (for example, a radio wave, an infrared wave, a laser wave, a sound wave, a radio wave propagated by a global network such as the Internet or other networks). A computer program propagated signal product 814 (FIG. 8A) is implemented. Such a carrier medium or carrier signal provides at least part of the software instructions for the routine / program 824, 826 according to the invention.

  In an alternative embodiment, the propagation signal is an analog carrier wave or digital signal carried by a propagation medium. For example, the propagated signal can be a digital signal carried by a global network (eg, the Internet, etc.), a telecommunications network, or other network. In one embodiment, the propagation signal is a signal transmitted by the propagation medium for a given period of time, for example, transmitted in packets by the network for a period of milliseconds, seconds, minutes or more. And instructions for software applications. In another embodiment, the computer readable medium of computer program product 824 is a propagation medium that can be received and read by computer system 810. For example, the computer system 810 receives a propagation medium and identifies a propagation signal embedded in the propagation medium, as in the case of the computer program propagation signal product described above.

<Feature-based display tool>
FIG. 8C is a screen shot 840 of a feature-based display tool in one implementation. Screenshot 840 depicts a frame of video with the features identified in box 842. A video frame sequence context relating to this frame is specified by reference numeral 844. Features 842 are tracked across a plurality of frames 844 to generate a plurality of feature sets that are displayed in a section 846 of the display. One feature set 846 includes a plurality of feature members (feature instances). In the data area, a feature bandwidth (Bandwidth) 852 that is the number of bits required when a given feature is encoded by a conventional compression method is displayed. A feature detection process is further displayed in the same data area (reference numeral 850). This tool can display all the features and feature tracks identified in the video of interest.

  Face detection may be supported by using a face tracker (face tracking means) biased to the face (focusing on the face). A plurality of features may be grouped by detecting a face. FIG. 8E is a screenshot 860-02 in which the face 864 is designated by the face tracker. FIG. 8D is a screenshot 860-01 showing both facial features and non-facial features as numbers 862. In this example, the numbers in FIG. 8D represent the length of feature tracking across multiple frames. By grouping features based on bias to the face, a model can be generated that can be used to encode multiple macroblocks that overlap the face.

  Strictly H. Instead of using the H.264 encoder process, the face model described above may be used to encode all pels / all pixels in the region of interest. By applying the heel model directly, there is no need to perform additional biasing, and A past reference frame can be selected without using H.264. After generating wrinkles based on the feature correspondence model, the residual is encoded by lower-order processing.

<Digital Rights Management>
In some embodiments, the model according to the present invention can be used to control access to encoded digital video. For example, without an associated model, the user cannot play a video file. One example of this approach is described in US patent application Ser. No. 12 / 522,357, filed Jan. 4, 2008. The entire teaching content of this US patent application is incorporated herein by reference. The model can be used to “lock” the video (“lock” the video). The model can be used as a key for accessing video data. The reproduction operation of the encoded video data may depend on the model. With such an approach, it is possible to read out the encoded video data without access to the model.

  By controlling access to the model, access to content playback can be controlled. This scheme can be a user-friendly and developer-friendly and efficient solution for restricting access to video content.

  Further, the content may be unlocked in stages (unlocking the content) using a model. Some versions of the model may allow the code to be decoded only to a certain level. By completing the model in stages, the entire video can be finally unlocked. In the initial unlock state, only the video thumbnail may be unlocked, and the user may be given an opportunity to decide whether or not the entire video is desired. If the user wants a standard quality version, he gets the model of the next higher version. If the user desires high definition quality or cinema quality, a more complete version of the model may be downloaded. The model is encoded without redundancy so that video quality corresponding to the encoding size and encoding quality can be realized in stages.

<Flexible macroblock ordering and scalable video coding>
In an exemplary embodiment of the invention, a conventional encoding / decoding process may be extended to improve the encoding process and benefit from compression. In one embodiment, the present invention provides a basic H.264. Flexible macroblock ordering (FMO) and scalable video coding (SVC), which are extensions of the H.264 standard, can be applied.

  The FMO assigns a macroblock of an encoded frame to one type among a plurality of types of slice groups. This allocation is determined by the macroblock allocation map, and macroblocks in the same slice group may not be adjacent to each other. FMO is advantageous in terms of error tolerance because it decodes slice groups independently of each other. Specifically, even if one slice group is lost during bitstream transfer, a macroblock assigned to that slice group is reconstructed from macroblocks that are assigned to other slices and adjacent to that slice group. can do. In one embodiment of the invention, feature-based compression is incorporated into the FMO “foreground and background” macroblock allocation map type. Macroblocks associated with features constitute a foreground slice group, and all other macroblocks (macroblocks not associated with features) constitute a background slice group.

  The SVC can provide video data codes at different bit rates. The base layer is encoded at a low bit rate, and at least one enhancement layer is encoded at a high bit rate. The decoding of the SVC bitstream may involve only the base layer (low bit rate / low quality application) or may additionally involve some or all enhancement layers (high bit rate / high quality application). . Since the SVC bitstream substream itself is also a valid bitstream, by using SVC, the SVC bitstream can be decoded (with different qualities depending on the capabilities of the device) by multiple devices, and In addition, the degree of freedom of an application scenario is improved including decoding in an environment where channel throughput changes such as Internet streaming.

  Generally, there are three types of scalability in SVC processing: temporal scalability, spatial scalability, and quality scalability. In one embodiment of the invention, feature-based compression is incorporated into the quality scalability configuration by including feature-based primary prediction in the base layer ("Generate model-based primary and secondary predictions"). (See the preceding description section). Then, by using the encoded frame in the base layer as a reference frame in the enhancement layer, feature-based secondary prediction can be realized in the enhancement layer. As a result, feature-based prediction information can be added step by step rather than all at once. As a modified example, all feature-based predictions (primary prediction and secondary prediction) may be transferred to the enhancement layer, and only the conventional prediction may be used in the base layer.

  The illustrated data path / execution path and components are merely examples, and the operation and configuration of each component, the data flow from each component, and the data flow to each component are the embodiment and the type of video data to be compressed It can be understood by those skilled in the art. That is, it is possible to employ data modules / data paths having any configuration.

  While the invention has been illustrated and described with reference to illustrative embodiments, workers skilled in the art will recognize that the invention is capable of form and detail without departing from the scope of the invention as encompassed by the appended claims. It will be understood that detailed modifications of are possible.

Claims (37)

A method of processing video data,
Using a detection algorithm to detect at least one of features and objects in the region of interest within at least one frame;
Modeling the detected at least one of features and objects using a set of parameters;
Correlating every instance of said detected at least one of features and objects across multiple frames;
Forming at least one track of the correlated instances;
Associating the at least one track with at least one block of video data to be encoded;
Generating a model-based prediction for the at least one block of video data using the associated track information, comprising storing the model-based prediction as processed video data; ,
A method for processing video data.   2. The video data processing method according to claim 1, wherein the detection algorithm is included in a type of non-parametric feature detection algorithm.   2. The method of processing video data according to claim 1, wherein the set of parameters includes information about the at least one of features and objects and is stored in a memory.   4. The video data processing method according to claim 3, wherein the feature parameters include a feature descriptor vector and a position of the feature.   5. The video data processing method according to claim 4, wherein the parameter is generated when the feature is detected.   2. The video data processing method according to claim 1, wherein the at least one block of the video data is a macroblock, and the at least one track associates a feature with the macroblock. A method of processing video data,
Detecting at least one of features and objects in the area of interest;
Modeling the at least one of features and objects using a set of parameters;
Correlating every instance of said at least one of features and objects across multiple frames;
Forming at least one matrix of the correlated instances;
Associating said at least one matrix with at least one block of video data to be encoded;
Generating a model-based prediction for the at least one block of video data using the associated information of the matrix, the method comprising storing the model-based prediction as processed video data; and ,
A method for processing video data.   8. The method of processing video data according to claim 7, wherein the set of parameters includes information about the at least one of features and objects and is stored in a memory.   9. The video data processing method according to claim 8, wherein the feature parameters include a feature descriptor vector and a position of the feature.   10. The video data processing method according to claim 9, wherein the parameter is generated when the feature is detected. The video data processing method according to claim 7, further comprising:
Using at least one subspace of a vector space to organize the at least one matrix as a parametric model of the at least one of features and objects correlated;
A method for processing video data, including: A codec for processing video data,
Feature-based detection means for identifying an instance of a feature in at least two video frames, wherein the identified instance of the feature is more data than other pixels in the one or more video frames. Feature-based detection means having a plurality of pixels indicative of complexity;
Modeling means operatively connected to the feature-based detection means for generating a feature-based correspondence model that models the correspondence of the instances of the feature in two or more video frames Modeling means to
When it is determined that encoding the instance of a feature using the feature-based correspondence model improves compression efficiency than encoding the instance of the feature using a first video encoding process A cache that prioritizes the use of the feature-based correspondence model;
A codec.   13. The codec according to claim 12, wherein the data complexity is determined when the encoding of the pixel by a conventional video compression method exceeds a predetermined threshold.   13. The codec according to claim 12, wherein the data complexity is determined when an amount of bandwidth allocated when the feature is encoded by a conventional video compression method exceeds a predetermined threshold.   15. The codec according to claim 14, wherein the predetermined threshold is a predetermined numerical value, a predetermined numerical value stored in a database, a numerical value set as an average value of bandwidth amounts allocated to features encoded in the past, and A codec, which is at least one of numerical values set as a median of the amount of bandwidth allocated to features encoded in the past.   The codec of claim 12, wherein the first video encoding process includes a motion compensated prediction process.   13. The codec of claim 12, wherein the preference for use is determined by comparing the coding costs of each solution candidate in a competitive mode, wherein the solution candidate is a tracking means, a key prediction motion model, a key prediction sampling. Codecs, including schemes, subtiling schemes, reconstruction algorithms (and (possibly) secondary prediction schemes).   18. The codec of claim 17, wherein when use of the feature-based modeling is prioritized, the level of data complexity of the instance of the feature is used as the threshold so that subsequent instances of the feature A codec in which the encoder automatically determines the start and use of feature-based compression for that subsequent instance of a feature when indicating a level of data complexity above a threshold.   The codec according to claim 12, wherein the feature-based detection means uses one of an FPA tracker, an MBC tracker and a SURF tracker. A codec for processing video data,
Feature-based detection means for identifying an instance of a feature in at least two video frames, wherein the identified instance of the feature is other in at least one video frame of the at least two video frames A feature-based detection means having a plurality of pixels exhibiting data complexity rather than
Modeling means operatively connected to the feature-based detection means for generating a feature-based correspondence model that models correspondences of identified instances of the features in the at least two video frames Modeling means;
Priority is given to the use of the correspondence model when it is determined that the compression efficiency of the specified instance of the feature is improved by a given feature-based correspondence model among a plurality of the feature-based correspondence models. Memory to
A codec.   21. The codec according to claim 20, wherein the compression efficiency of the specified feature is determined by encoding the instant of the feature in the case of using the first video encoding process and the compression efficiency stored in the database. A codec that determines an improvement in compression efficiency of the identified instance of the feature by comparing with one of the numbers. A method of processing video data,
Modeling a feature by vectorizing at least one of a feature pel and a feature descriptor;
(A) minimizing the mean square error (MSE) between different vectors of feature pels or between different feature descriptors; and (b) maximizing the inner product between different vectors of feature pels or between different feature descriptors. Identifying similar features by at least one of the following:
A process of applying a standard motion prediction / compensation algorithm, thereby taking into account the translational motion of the feature and obtaining processed video data;
A method for processing video data. A method of processing video data,
Implementing model-based prediction by configuring a codec to encode a target frame;
Encoding a macroblock in the target frame using a conventional encoding process;
In the process of analyzing the encoding of the macroblock, when the conventional encoding of the macroblock is determined to be at least one of efficient and inefficient, and the conventional encoding is determined to be inefficient, The encoder is analyzed by generating a plurality of predictions for a macroblock based on a plurality of models, and the evaluation of the plurality of predictions for the macroblock is based on a coding size;
Ranking the predictions of the macroblocks together with the macroblocks according to the conventional coding;
A method for processing video data.   24. The video data processing method according to claim 23, wherein the conventional encoding of the macroblock is efficient when the encoding size is smaller than a predetermined size threshold.   24. The video data processing method according to claim 23, wherein the conventional encoding of the macroblock is efficient when the target macroblock is a skip macroblock.   The video data processing method according to claim 23, wherein the conventional encoding of the macroblock is inefficient when the encoding size is larger than a threshold.   24. The video data processing method according to claim 23, wherein when it is determined that the conventional encoding of the macroblock is inefficient, a plurality of encodings for the macroblock are generated in a competition mode and the compression efficiency of each other is increased. A method for processing video data. The video data processing method according to claim 27, wherein the competition mode encoding algorithm is:
Subtracting the prediction from the macroblock, thereby generating a residual signal;
A step of transforming the residual signal using block-based two-dimensional DCT approximation, and a step of encoding transform coefficients using an entropy encoder;
A method for processing video data.   24. The video data processing method of claim 23, wherein the encoder analyzed by generating a plurality of predictions generates a composite prediction that sums a primary prediction and a weighted secondary prediction. Processing method. A method of processing video data,
A process for modeling data with a plurality of fidelity for model-based compression, wherein the plurality of fidelities are at least one of a macroblock hierarchy, a macroblock hierarchy as a feature, a feature hierarchy, and an object hierarchy. A process involving one,
With
The macroblock hierarchy uses a block-based motion prediction and compensation (BBMEC) application to find a prediction for each tile from a limited space in the decoded reference frame;
The featured macroblock hierarchy (i) finds a first prediction of the target macroblock from the most recent reference frame using the same first BBMEC application as the macroblock hierarchy; (ii) 2 Find the second prediction for the first prediction by searching for the second most recent reference frame using the second BBMEC application, and (iii) gradually apply the BBMEC application over the past frames To generate a track of the target macroblock,
The feature hierarchy detects and tracks a feature regardless of the grid of macroblocks, associates the feature with a macroblock that overlaps the feature, and navigates the decoded reference frame using the feature track Finds a good match for the overlapping macroblock, and if multiple features overlap with a target macroblock of interest, the feature with the greatest overlap models the target macroblock. Selected
In the object hierarchy, when an object includes a plurality of macroblocks or overlaps with a plurality of macroblocks, a single motion vector can be calculated for all macroblocks corresponding to the object. And save coding size,
Video data processing method.   31. The video data processing method according to claim 30, wherein the plurality of fidelities are sequentially examined.   31. The video data processing method according to claim 30, wherein the plurality of fidelities are examined in a competition mode. A computer program product comprising program code means,
The computer program product for controlling the computer to execute the processing method according to claim 1 by being loaded into the computer. A computer program product comprising program code means,
8. A computer program product for controlling the computer to execute the processing method according to claim 7, wherein the program code means is loaded into the computer. A computer program product comprising program code means,
23. A computer program product for controlling the computer to execute the processing method according to claim 22, wherein the program code means is loaded into the computer. A computer program product comprising program code means,
24. A computer program product for controlling the computer to execute the processing method according to claim 23, wherein the program code means is loaded into the computer. A computer program product comprising program code means,
32. A computer program product, wherein the program code means controls the computer to execute the processing method according to claim 30 by being loaded into the computer.