TW201342935A - Video compression repository and model reuse - Google Patents

Abstract

Systems and methods of improving video encoding/decoding efficiency may be provided. A feature-based processing stream is applied to video data having a series of video frames. Computer-vision-based feature and object detection algorithms identify regions of interest throughout the video datacube. The detected features and objects are modeled with a compact set of parameters, and similar feature/object instances are associated across frames. Associated feature/objects are formed into tracks, and each track is given a representative, characteristic feature. Similar characteristic features are clustered and then stored in a model library, for reuse in the compression of other videos. A model-based compression framework makes use of the preserved model data by detecting features in a new video to be encoded, relating those features to specific blocks of data, and accessing similar model information from the model library. The formation of model libraries can be specialized to include personal, ''smart'' model libraries, differential libraries, and predictive libraries. Predictive model libraries can be modified to handle a variety of demand scenarios.

Description

Video compression repository and model reuse

The invention relates to a video compression repository and model reuse.

Related application

This application claims the benefit of U.S. Provisional Application No. 61/650,363, filed on May 22, 2012, and U.S. Provisional Application No. 61/616,334, filed on March 27, 2012. The present application also claims priority to U.S. Patent Application Serial No. 13/772,230, filed on February 20, 2013. U.S. Patent Application Serial No. 13/772,230, filed on Feb. 20, 2013, which is incorporated herein by reference in its entirety, in its entirety, the entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire content 13/121,904 is the US national phase of International Patent Application No. PCT/US2009/059653, which was filed on October 6, 2009. The designated country is the United States and is published in English and claims to be submitted on October 7, 2008. US Provisional Application No. 61/103,362. Application No. 13/121,904 is also a continuation of U.S. Patent Application Serial No. 12/522,322, filed on Jan. 4, 2008, which is incorporated herein by reference. In the US national phase of International Patent Application No. PCT/US2008/000090, the designated country of the case is the United States and is published in English. The case claims the benefit of US Provisional Application No. 60/881,966, which was filed on January 23, 2007. And in relation to U.S. Provisional Application No. 60/811,890, which was filed on June 8, 2006, and which is filed on March 31, 2006, U.S. Application No. 11/396,010 (already on November 25, 2008) Department awarded as US Patent No. 7,457,472) The continuation case is a continuation of the US application No. 11/336,366 (issued as US Patent No. 7,436,981 on October 14, 2008), which was filed on January 20, 2006. Part of the continuation of US Application No. 11/280,625 (issued as US Patent No. 7,457,435 on November 25, 2008), which was filed on November 16, 2005, which claims November 17, 2004. U.S. Provisional Application No. 60/628,819, issued to Japan, and U.S. Provisional Application No. 60/628,861, filed on November 17, 2004, and the U.S. Application No. 11 filed on September 20, 2005 Partial succession of No. 230,686 (issued as US Patent No. 7,426,285 on September 16, 2008), filed on July 28, 2005, US Application No. 11/191,562 (already Part of the continuation of the US Patent Case No. 7,158,680 on January 2, 2007, which claims the benefit of US Provisional Application No. 60/598,085, which was filed on July 30, 2004. US Application No. 11/396,010 also claims priority to U.S. Provisional Application No. 60/667,532, filed on March 31, 2005, and U.S. Provisional Application No. 60/670,951, filed on April 13, 2005. .

This application is also related to U.S. Patent Application Serial No. 13/725,940, filed on Dec. 21, 2012, which is incorporated herein by reference to U.S. Provisional Application Serial No. 61/707,650 and No. Benefits of US Provisional Application No. 61/615,795, which was filed on the 26th of the month.

The complete teachings of the above application are hereby incorporated by reference.

Video compression is considered a method of representing digital video material in the form of using fewer bits when storing or transmitting. The video compression algorithm uses the redundancy and irrelevancy of video data in space, time, or color space to achieve compression. Video compression algorithms usually cut video data into multiple parts, for example, multiple frame groups and multiple An image point (pel) for identifying a redundant area in the video that can be represented by fewer bits than the original video material. When the redundancy in the data is reduced, a greater compression effect is achieved. The encoder is used to convert the video data into an encoded format, and the decoder is used to convert the encoded video back to the form equivalent to the original video material. The implementation of the encoder/decoder is called a codec.

Standard encoder splits a given video frame into multiple non-overlapping codes A coding unit or a macroblock (a rectangular area consisting of a plurality of consecutive image points) for encoding, the macroblocks are usually in a horizontal order from left to right and the message The box is handled from top to bottom. Compression is achieved when data from previously existing code is used to predict and encode macroblocks. The method of encoding a macroblock using spatial proximity sampling of a macroblock of a previously existing code in the same frame is called intra-prediction. Intraframe prediction attempts to exploit spatial redundancy in the data. Encoding a macroblock using a similar area of a previously existing code frame, together with a motion estimation model, is called inter-prediction. Inter-frame prediction attempts to exploit temporal redundancy in the data.

The encoder can measure the difference between the data to be encoded and the predicted value. Generate residual (residual). This residual value can provide a difference between the predicted macroblock and the original macroblock. The encoder generates motion vector information, for example, which details the location of the macroblock in a reference frame relative to the macroblock being encoded or decoded. The predictors, motion vectors (for inter-frame prediction), residual values, and related data are combined with other processing (eg, spatial transforms, quantizers, entropy encoders, and loop filters). A valid code used to generate the video material. The quantized and transformed residual values are processed and added back to the prediction The values are aggregated into decoded frames and stored in a frame store. Those skilled in the art will be familiar with the details of such encoding techniques for video.

H.264/MPEG-4 Part 10 AVC (advanced video code) Coding)), hereinafter referred to as H.264, is a codec standard for video compression that uses block-based motion prediction and compensation and achieves high quality at a relatively low bitrate. Video representative. This standard is used for Blu-ray disc creation and is one of the coding options in the main video distribution channel (which includes video streaming over the Internet, video conferencing, cable TV, and direct broadcast satellite TV). The basic code unit for H.264 is a 16X16 macroblock. H.264 is the most widely accepted standard in video compression recently.

The basic MPEG standard defines three types of frames (or images) depending on how the macroblocks in the frame are encoded. The I frame (the image with the code in the frame) is encoded using only the data that appears in the frame itself. Generally, when the encoder receives the video signal data, the encoder first generates an I frame and cuts the video frame data into a plurality of macroblocks each encoded by the intraframe prediction. Therefore, the I frame consists only of intra-frame prediction macro blocks (or "intra-frame macro blocks"). The encoding of the I-frame is very costly because the encoding does not benefit from the information from the previously decoded frame. The P-frame (predicted image) is encoded by forward prediction, using data from previously decoded I-frames or P-frames (also known as reference frames). The P frame may contain intra-frame macroblocks or (forward) predictive macroblocks. The B-frame (bidirectional predictive image) is encoded by bi-directional prediction, using data from both the previous frame and the subsequent frame. The B frame may contain macroblocks within the frame, (forward) predicted macroblocks, or bidirectionally predicted macroblocks.

As described above, conventional inter-frame prediction is based on Block Based Motion Estimation and Compensation (BBMEC). BBMEC The method searches for the best match between the target macroblock (the current macroblock being encoded) and the similarly sized area in the previously decoded reference frame. When a best match is found, the encoder may transmit a motion vector. The motion vector may include a pointer to the best matching frame position and information related to the difference between the best match and the corresponding target macro block. In this way, anyone can perform a exhaustive search in the entire video "datacube" (height X width X frame) in a trusted manner to find the most of each macro block. Good matches may be possible; however, exhaustive search usually has computational prohibitions. Therefore, the BBMEC search method is limited in terms of the time in which the reference frame is searched and in the vicinity of the searched neighborhood. This means that it is not always possible to find "best possible" matches, especially for fast-changing data.

There is a special set of reference frames called Group Of Pictures. GOP). The GOP only contains the decoded image points in each reference frame and does not contain how the macro blocks or frames themselves are encoded (I-frame, B-frame, or P-frame). News. Earlier video compression standards (eg, MPEG-2) used a reference frame (previous frame) to predict the P frame and used two reference frames (a past frame, a future frame) to predict B messages. frame. Conversely, the H.264 standard allows the use of multiple reference frames for P-frame prediction and B-frame prediction. The reference frames are typically temporally adjacent to the current frame; however, they are also adapted to specify a reference frame from outside the set of adjacent frames at that time.

Conventional compression allows for blending multiple matches in multiple frames for Predict the area in the current frame. This blending is often a linear combination of the matching linear or logarithmic scales. One example of this bi-directional prediction method is that it fades from one image to another over time. The process of fading is a linear blend of two images, and the process can sometimes be utilized Bidirectional prediction for effective simulation. Some legacy standard encoders (eg, MPEG-2 interpolation mode) allow interpolation of linear parameters to synthesize the bi-predictive model in a number of frames.

The H.264 standard also consists of one or more continuous giants by dividing the frame into slices. The additional spatial flexibility is introduced by the different spatial regions of the block. Each slice in a frame is encoded in a manner independent of the other slices (and thus decoded in an independent manner). The I slice, the P slice, and the B slice are then defined in a manner similar to the frame type described above, and the frame is composed of a plurality of slice types. In addition to this, the manner in which the encoder sorts the processed slices is generally flexible, so the decoder processes them in any order in which the slices arrive at the decoder.

Model-based compression techniques have been proposed in history to avoid Limitations of BBMEC predictions. Such model-based compression techniques (most well known as the MPEG-4 Part 7 standard) rely on detecting and tracking objects or features in the video and encoding these features in a manner separate from the rest of the video frame. / object method. However, such model-based compression techniques suffer from the difficulty of cutting video frames into object areas and non-object areas (feature areas and non-feature areas). First, because objects may be of any size, their shape must be encoded in addition to their texture (color content). Second, tracking multiple moving objects can be quite difficult, and inaccurate tracking can result in incorrect cutting, which often results in poor compression performance. The third problem is that not all video is composed of objects or features, so when the object/feature does not exist, the backup coding technique is needed.

The H.264 standard allows codecs to be lower than previous standards (for example, MPEG-2 and MPEG-4 ASP (Advanced Simple Profile (ASP)) file size to provide better quality video; however, the implementation of the H.264 standard "known" compression Codecs typically struggle to maintain greater video quality and resolution requirements in devices with limited memory (eg, smart phones and other mobile devices) operating on limited bandwidth networks. Video quality and resolution are often sacrificed to achieve unsatisfactory playback on such devices. Furthermore, as video resolution increases, file size increases, and storing video on and off such devices becomes a potential problem.

U.S. Application Serial No. 13/725,940 (herein referred to as "Application No. 940") in the co-pending application of the Applicant proposes a model-based compression technique which avoids the cutting problem mentioned above. The Model-Based Compression Framework (MBCF) in the applicant's co-application No. 940 also detects and tracks objects/features in order to identify important areas to be encoded in the video frame. Try to explicitly encode these objects/features. Rather, the objects/features are associated with neighboring macroblocks, and the macroblocks that are encoded are as in the "known" codec. This implicit use of analog information can mitigate the cutting problem in two ways: maintaining a fixed code unit (macroblock size) (thus eliminating the need to encode objects/feature shapes) and reducing the effects of inaccurate tracking ( Because tracking helps, but does not specify the mobile prediction step). In addition, the MBCF in the 940 application in the co-application also simulates video data with multiple fidelity, including a backup method using a conventional compression method when the object/feature does not exist; such hybrid coding Technology ensures that analog information is used only when needed, and is not applied incorrectly when not needed.

U.S. Patent No. 6,088,484 to Mead, which proposes an extension of the standard model-based compression technique in which objects detected in one of the videos are stored and then reused to help compress similar objects in another video. . However, this Mead specializes The model reuse compression technique involves explicit or direct encoding of objects/features in new video, and thus encounters the same cutting problems mentioned above (ie, precise cutting of objects/features from non-object/non-features) problem). The present invention proposes a model reuse compression technique in the framework of the 940 application in the co-application, the implicit use of the object/feature model to represent the important macroblock to be encoded can avoid the cutting problem while retaining the simulation Most of the benefits in order to improve encoder prediction.

The invention finds the basics in the inter-frame prediction method of the conventional video codec Limiting and applying higher order simulations overcomes these limitations and provides improved inter-frame prediction results while maintaining the same general processing flow and architecture as conventional encoders.

The present invention is based on the model proposed in the 940 application of the co-application. In the compression mode, the features in the video are detected, simulated, and tracked, and the feature information is used to improve the prediction and encoding of the data in the same video. This "online" based on application No. 940 is based on feature prediction (where feature information is generated and used to help encode subsequent video cuts in the same video) expanded to "offline" in the present invention. Based on feature prediction, feature information from one of the videos is retained or saved to the model library for reuse to identify the target macroblock and thereby assist in encoding the data in another video. The way to achieve this is not to perform feature cutting in the target video. The standard compression technique and the on-line prediction in the 940 application attempt to exploit temporal redundancy in a single video; however, the offline prediction proposed by the present invention attempts to exploit redundancy across multiple video.

The four main components of the offline feature-based compression technique of the present invention are as follows: (i) Generating a feature model and related information from one or more input videos and saving the feature information; (ii) Reusing the saved feature information to improve the compression effect of another video (unlike the input video or independent of the input video), to avoid feature cutting in the video; (iii) from multiple input video Forming a feature model library without the feature information in the large set; (iv) using the feature model library to decode the unrelated or target video. The formation of the model library will be specialized to include personal "smart" model libraries, differential libraries, and predictive libraries. The predictive model library is modified to handle a wide variety of demand scenarios.

10-1~10-n‧‧‧Featured examples of detection

20-1~20-n‧‧‧ frame

30-1~30-n‧‧‧Area

40‧‧‧Overall matrix

50‧‧‧Features detected previously

60-1~60-n‧‧‧Characteristics

70‧‧‧Feature Tracker

80‧‧‧Feature Detector

90‧‧‧ Current frame

170‧‧‧Network

300‧‧‧ Flowchart

310‧‧‧Steps

312‧‧ steps

314‧‧‧Steps

316‧‧‧Steps

318‧‧‧Steps

320‧‧‧Steps

322‧‧‧Steps

324‧‧‧Steps

326‧‧‧Steps

328‧‧‧Steps

414‧‧‧Number of members

416‧‧‧ Results after clustering unique features based on characteristic spectral color maps

418‧‧‧ cluster members

420‧‧‧ Results after clustering unique features based on the SURF descriptor of the special number

422‧‧‧ cluster members

500‧‧‧ steps

510‧‧ steps

520‧‧‧Steps

530‧‧‧Steps

610‧‧‧Steps

612‧‧ steps

614‧‧‧Steps

616‧‧‧Steps

618‧‧ steps

620‧‧‧Steps

632‧‧‧Steps

634‧‧‧Steps

702‧‧‧Video data block

704A~704F‧‧‧ frame

706A~706F‧‧‧

708A~708F‧‧‧

802‧‧ steps

804‧‧‧ steps

806‧‧‧Steps

808‧‧‧Steps

810‧‧‧Steps

812‧‧‧ steps

814‧‧‧Steps

816‧‧ steps

818‧‧‧Steps

820‧‧‧Steps

822‧‧‧Flowchart

824‧‧‧Steps

826‧‧‧Steps

828‧‧ steps

830‧‧ steps

832‧‧‧Steps

834‧‧ steps

836‧‧ steps

838‧‧‧Steps

840‧‧‧Steps

850‧‧‧flow chart

852‧‧‧Steps

854‧‧‧Steps

856‧‧ steps

858‧‧‧Steps

860‧‧‧Steps

902‧‧‧Video Repository

904‧‧‧Coded video set

906A‧‧‧First Video Set

906B‧‧‧Second video set

906C‧‧‧ Third Video Set

906D‧‧‧Nth Video Set

908‧‧‧Customer device

910‧‧‧Streaming decoder module

911‧‧‧Decoded video

912A‧‧‧ memory

912B‧‧‧ display

912C‧‧‧ storage module

914‧‧‧Request generation module

916‧‧‧Video request

918‧‧‧Request receiving module

920‧‧‧Requested video inquiries

922‧‧‧Requested video

924‧‧‧Stream generation module

926‧‧‧ generated library

928‧‧‧Requested encoded video

952‧‧‧Customer version of the requested video library

954‧‧‧ version design module

956‧‧‧Customer version of the library requested to be video

958‧‧‧Different library generation module

960‧‧‧Video Coding Module

962‧‧‧Differential library

964‧‧‧Library storage module

966‧‧‧Program Configuration Module

970‧‧‧Combined library

980‧‧‧Characteristic model

982‧‧‧Characteristic model

1002‧‧‧ Predictive library generation module

1004‧‧‧User Behavior Description File Module

1006‧‧‧User behavior description file

1008‧‧‧ Predictively encoded video request

1010‧‧‧Video for encoding

1012‧‧‧ Predictively generated libraries

1110‧‧‧Customer computer/device

1112‧‧‧Cloud

1114‧‧‧Computer Program Communication Signal Products

1116‧‧‧Communication network

1118‧‧‧I/O device interface

1120‧‧‧Central Processor Unit

1122‧‧‧Internet interface

1124‧‧‧Computer Software Instructions

1126‧‧‧OS program

1128‧‧‧Information

1130‧‧‧ memory

1132‧‧‧ disc storage

1134‧‧‧System Bus

The foregoing will be understood from the following detailed description of the embodiments of the invention, The drawings are not necessarily to scale, emphasis is placed on illustrating the embodiments of the invention.

Figure 1 is a block diagram of a feature simulation in accordance with an embodiment of the present invention.

2 is a block diagram of feature tracking in accordance with an embodiment of the present invention.

A flow chart of a method for extracting and generating a feature model for a model embodiment of the system repository shown in FIG.

Figure 4A shows a screenshot of a feature-based compression tool based on an exemplary implementation.

Figure 4B shows a screenshot of a feature-based compression tool based on an exemplary implementation.

Figure 5 is a block diagram of a processing element that simulates a macroblock as a feature that aligns the boundaries of a macroblock.

Figure 6A is a flow diagram of a method of generating an index value used by a repository.

Figure 6B is a flow diagram of a method of using index values used by a repository.

Figure 7 is a block diagram of a normalized block. The normalized block is a collection of multiple associated tables.

A flowchart of an exemplary embodiment of a method for generating an index value is shown in FIG. 8A.

A flowchart of an exemplary embodiment of a method of querying features using index values is shown in FIG. 8B.

A flow chart of another method utilized by an exemplary embodiment of the system repository shown in FIG. 8C.

A flow chart of a method employed by an exemplary embodiment of the system repository shown in Figure 8D.

Figure 9A is a block diagram of an exemplary embodiment of a repository operatively coupled to a client device over a network.

Figure 9B is a block diagram of another exemplary embodiment of a repository for communicating with client devices over a network.

Figure 10 is a block diagram of an exemplary embodiment of operating a repository connected to a client device over a network.

FIG. 11A is a schematic diagram of a computer network environment in which an embodiment of the present invention is deployed.

Figure 11B is a block diagram of a computer node in the network of Figure 11A.

The teachings of all patents, published applications, and citations cited herein are hereby incorporated by reference. An illustration of an exemplary embodiment of the present invention is as follows.

The invention can be applied to a variety of standard coding and code units. In the following, unless otherwise mentioned, terms such as "conventional" and "standard" (sometimes with "compression", "codec", "encoding", or "encoding" "Used together" means H.264; and "macroblock" generally represents the underlying H.264 code unit.

Generate and save feature models

Definition of characteristics

Exemplary components of the present invention include the ability to be stored or transferred in an optimal manner Represents video compression and decompression processing of digital video data. Such processing may involve interfacing one (or more) video compression/encoding algorithms to exploit spatial, temporal, or spectral redundancy and independence in the video material. This application can be achieved by using and retaining the feature model/parameters. Then, words such as "feature" and "object" are used interchangeably. Objects are generally defined as "large features." Both features and objects can be used to simulate data.

The feature is composed of multiple image points that are very close in the complexity of the data. Group. As detailed below, data complexity can be detected by various criteria; however, from a compression point of view, the ultimate feature of data complexity is "very high coding cost", which means that these images are encoded by conventional video compression. The image point will exceed the critical point that is considered "effective coding." When the conventional encoder assigns a disproportionate amount of bandwidth to a particular area (because the inter-frame search cannot find a good match for them in the conventional reference frame), the area is likely to be "feature-feature- Rich), and the feature model based compression method will significantly improve the compression effect in these areas.

Feature detection

The feature examples 10-1, 10-2, ..., 10-n shown in Fig. 1 are already in the video One or more frames 20-1, 20-2, ..., 20-n are detected. In general, this feature can be detected using several criteria based on the following: structural information inferred from image points and complexities that indicate that conventional compression uses a disproportionate number of bandwidths to encode feature regions. Sexual criteria. Each feature instance will be in its frame 20-1, 20-2, ... by the corresponding spatial extent shown by "area" 30-1, 30-2, ..., 30-n in Figure 1. Further spatial identification in 20-n. For example, such feature regions 30-1, 30-2, ..., 30-n are extracted as simple rectangular regions of image point data. In one embodiment of the invention, the size of the feature regions is 16X16, the same size as the H.264 macro block.

In the literature that detects features based on the structure of the image points themselves, A number of algorithms have been proposed containing non-parametric feature detection algorithm classes suitable for different transformations of image point data. For example, Scale Invariant Feature Transform (SIFT) [Lowe, David, published in Int. J. of Computer Vision, 60(2): 91-110, 2004. Distinctive image features from scale-invariant keypoints] use convolution of the Gaussian difference function of the image to detect blob-like features. Speeded-Up Robust Features (SURF) algorithm [Bay, Herbert et al., 2008, Computer Vision and Image Understanding, 110(3): 346-359, "SURF: Accelerated Robust Features"] The Hessian operator's determinant is also used to detect the class characteristics. The SURF algorithm is used to detect features in one of the embodiments of the present invention.

Other feature detection algorithms are designed to find specific types of features, such as For example, face. In another embodiment of the invention, Haar-like features are detected as part of the forehead and facial contour detection [Viola, Paul and Jones, Michael in 2001 IEEE Conf. on ComputerVision and Pattern Recognition, "Rapid object detection using a boosted cascade of simple features", published in 1: 511-518.

In another embodiment, 2009 is fully incorporated herein by reference. </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; For example, coding complexity can be required by analyzing conventional compression methods (for example, H.264) to encode regions in which features appear. The bandwidth (the number of bits) is determined. Again, the different detection algorithms operate differently; however, each applies a complete sequence of video frames throughout the video data in the embodiment. In a non-limiting example, the first encoding operation is done using an H.264 encoder and produces a "bandwidth map." It then defines or determines where the H.264 encoding is the most expensive in each frame.

In general, conventional encoders (eg, H.264) cut a video frame into multiple uniform tiles arranged in a non-overlapping pattern (for example, 16×16 macroblocks and their Sub-brick). In one embodiment, each tile is analyzed as a potential feature based on the relative bandwidth required for H.264 encoding the tile. For example, the bandwidth required to encode a tile through H.264 can be compared to a fixed threshold. If the bandwidth exceeds the threshold, the tile will be declared a "feature." The threshold may be a predetermined value. The pre-set values can be stored in a database for easy access during feature detection. The threshold may be a value that is set to the average number of bandwidths assigned to previously encoded features. Similarly, the threshold may also be a value that is set to the number of bit widths assigned to the previously encoded feature. Alternatively, you can calculate the cumulative distribution function of the tile widths in the entire frame (or the entire video) and drop any bandwidth in the top percentile of all tile bandwidths. It is a "feature".

In another embodiment, the video frame is cut into a plurality of overlapping tiles. The overlapping samples may be offset such that the center of the overlapping tiles will appear at the intersection of the corners of every four lower tiles. This overcomplete partitioning is intended to increase the likelihood that the initial sampling position will result in a detected feature. Other potentially more sophisticated top cutting methods can also be used.

Small spatial areas that are detected as features are analyzed to determine if they are It can be combined into a larger spatial region based on specific coherency criteria. The size of the spatial region may vary from a small group of image points to a larger region that may correspond to a real object or a portion of the object. However, it is important to note that the detected features do not necessarily correspond to unique and separable entities, such as objects and sub-objects. A single feature may contain elements of two or more objects or no elements at all. For the purposes of the present invention, a key feature of a feature is the ability to effectively compress (relative to conventional methods) the set of image points that make up the feature with a feature model based compression technique.

Coherence criteria for combining small areas into larger areas may include Includes: mobile similarity, appearance similarity after motion compensation, and coding complexity similarity. Coherent motion can be found via higher order motion models. In one embodiment, the translational movement of each individual small area is integrated into an affine movement model that approximates the movement model of each of the small areas. If the movement of a small set of regions can be integrated into a summary model on a consistent basis, this implies a dependency between the regions, which may indicate that the coherence between the small regions can be applied via a summary feature model. Sex.

Feature model formation

After detecting the feature(s) in multiple frames of a video, the important system will Multiple instances of the same feature are associated together. This method is called feature association bonding and is the basis for feature tracking (to determine the location of a particular feature over time), as explained below. However, in order to function, the feature association method must first define a feature model that can be used to distinguish similar feature instances from dissimilar feature instances.

In one embodiment, the feature image points themselves are used to simulate a special Sign. Two-dimensional feature image point regions are vectorized, and by minimizing the Mean-Squared Error (MSE) between different feature image point vectors or maximizing between different feature image point vectors The inner product can identify similar features. The problem with this approach is that the feature image point vectors are subject to small changes in the features, such as translation, rotation, scaling, and changing the illumination of the features. Features often change in this way throughout the video, so using the feature image point vector itself to simulate and correlate the features needs to take into account these changes. In one embodiment, the present invention is simplest by applying a standard motion prediction and compensation algorithm for the translational movement of the feature found in a conventional codec (for example, H.264). Ways to consider these feature changes. In other embodiments, more complex techniques can be used to consider the rotation, scaling, and illumination variations of the features frame by frame.

In an alternative embodiment, the feature model is unchanged from small rotation, translation, scaling, And a reduced representation of the feature of the possible illumination changes of the feature ("compact" means "dimension is lower than the original feature image point vector"), meaning that if the feature changes slightly from frame to frame The feature model will still remain relatively constant. This type of reduced feature model is often referred to as a "descriptor." In one embodiment of the present invention, for example, the SURF feature descriptor has a length of 64 (contrared to a feature image point vector of length 256) and is based on the sum of multiple Haar wavelet transform responses. . In another embodiment, a color map of the five color bins is constructed by using a color map of the feature image points, and the 5-component color strip map serves as a feature descriptor. . In an alternative embodiment, the feature area is converted by a 2D Discrete Cosine Transform (DCT). The 2D DCT coefficients are then added in the upper and lower triangles of the coefficient matrix. These sums then form an edge feature space and act as feature descriptors.

When using feature descriptors to simulate features, by minimizing these feature descriptions Similar features can be identified by MSE between symbols (rather than feature image point vectors) or by maximizing the inner product between the feature descriptors.

Feature association and tracking

Once the feature is detected and simulated, the next step will be multiple messages. Similar features in the box are associated with each other. Each instance of a feature that appears in multiple frames is a sample of the appearance of the feature, and multiple instances of the feature that are combined across multiple frames are considered to be "belonging" to the same feature. Once the associations are combined, multiple feature instances belonging to the same feature can be aggregated to form a feature track or aggregated into an ensemble matrix 40 (FIG. 1).

The definition of the feature trajectory is the function of the (x, y) position of the feature and the frame in the video. relationship. In one embodiment, the newly detected feature instance is associated with the previously tracked feature association (or, in the case of the first frame of the video, it is associated with the previously detected feature association) as a decision as to which of the current frames. The feature instance is the basis for the extension of the previously established feature track. Using a previously created feature track (or, in the case of a first video frame, utilizing previously detected features) to identify a feature instance in the current frame constitutes tracking of the feature.

Figure 2 illustrates the use of feature tracker 70 to track features 60-1, 60-2, ..., 60-n. Feature detector 80 (for example, SIFT or SURF) is used to identify features in the current frame. The feature instances detected in the current frame 90 will match the previously detected (or tracked) features 50. In one embodiment, an Auto-Correlation Analysis (ACA) indicator (which will be based on an automatic association matrix of features) is used prior to the association step. The feature intensity is used to sort the candidate feature detection set in the current frame, and the Gaussian derivative filter is used to calculate the image gradient in the automatic correlation matrix, and the Harris-Stephens corner detection algorithm is used [Harris, Chris, and Mike Stephens in 1988). It can be found in the "A combined corner and edge detector" published on pages 147 to 151 of the 4th Alvey Vision Conference. A feature instance with a high ACA value will be given priority as a candidate feature instance for the track extension. In one embodiment, the lower-level feature instances in the ACA sort list are deleted from the candidate feature set if they fall within a particular distance (eg, an image point) of a higher-level feature instance in the list.

In different embodiments, feature descriptors (for example, SURF descriptors) Or the feature image points themselves can serve as feature models for the purpose of determining the trajectory extension. In one embodiment, whether the previous tracking feature shown by the regions 60-1, 60-2, ..., 60-n in FIG. 2 is tested in the new detection feature in the current frame 90 is performed one at a time. Track extension. In one embodiment, the most recent feature instance of each feature trajectory acts as a focus (or "target feature") when searching for the trajectory extension in the current frame. All candidate feature detections within a particular distance (for example, 16 image points) at the location of the target feature are currently tested, with a minimum MSE relative to the target feature (in the image point space) Candidate features in the middle or in the descriptor space are then selected as extensions of the feature trajectory. In another embodiment, a candidate feature is disqualified as a track extension if the MSE of the target feature is greater than a certain threshold.

In a further embodiment, if there is no candidate feature check in the current frame If you qualify for an extension of a given feature trajectory, you can use the Motion Compensated Prediction (MCP) algorithm in H.264 or the general motion prediction and The Motion Estimation and Compensation (MEC) algorithm performs a limited search for matching regions in the current frame. Both the MCP and the MEC perform a gradient descent search for the matching region of the current frame, which minimizes the MSE relative to the target feature in the previous frame (and meets the MSE threshold). If any match of the target features is not found in the current frame (from the candidate feature detection or from the CP/MEC search process), the corresponding feature track will be declared as "dead" or " Terminated.

In a further embodiment, if two or more feature tracks are in the current frame If the feature instance is coined by a certain threshold (for example, 70% overlap), then only one feature track will be left, and all other feature tracks will be deleted or not further considered. This deletion process keeps the feature track the longest history and maintains the maximum total ACA, the sum of all feature instances.

In another embodiment, the midpoint normalization is performed on the feature trajectory, which is calculated first. The "smooth" set of track positions and then adjusts the position of the "away" feature of the normalized midpoint (this process is called center adjustment).

Summarizing the above, the following steps are common to the various embodiments of the present invention: Sign detection (SURF or face); feature simulation (SURF descriptor, spectral strip profile); ACA-based candidate feature ranking; and feature association combining and tracking among multiple candidate features through MSE minimization It will be supplemented by the MCP/MEC search trajectory extension and the center adjustment of the trajectory. The processing stream is referred to as a SURF tracker if the SURF descriptor is used for tracking. If the color strip map is used for tracking, the processing stream is called a spectrum tracker.

Figure 3 shows the feature-based streaming described above (Feature-based Flowchart 300 of the basic steps in Processing Stream, FPS). Assuming a special video, the FPS will start from 310 detecting the features in the video. Next, the FPS associates (associates) instances 320 of the detected features, which may use a reduced feature model rather than the feature image point itself. The FPS then tracks the detected and associated features 312. The FPS also determines the similarity 322 in the associated instances of the features and implements midpoint normalization 324 for similar associated instances of the features. The FPS will adjust the center 326 of the associated features based on the normalized midpoints and then forward the feed into the tracker 312. The FPS will aggregate the feature set 314 consisting of the associated combined feature instances based on at least one of the tracked feature 312, the associated instance 320 of the feature, and the associated instance 326 having the characteristics of the adjusted center. In a further embodiment, the FPS may split or merge feature sets 316 based on different criteria. The FPS also sets the state of the model 318 (for example, set to isolation). In one embodiment, the isolation model is a model that should not be edited again because it has ended. In another embodiment, the repository also analyzes the complexity of the video 328 during the generation of the model. Those skilled in the art will appreciate that the methods described above can be performed in any order and need not be performed in the order described above.

Unique features and feature clusters

The above paragraph outlines how to go through video (for clarity, this article is called "losing" Into the video frame to detect, simulate, correlate, and track features. The present invention seeks to preserve or "renew" all of the feature information in the input video that can be used to improve the effect in another "target" video (defined as the video to be encoded). In one embodiment, the feature information is stored in the archive. In other embodiments, the feature information can be stored in a relational database, an object database, a NoSQL database, or other data structures. More details regarding the storage of feature information are described below. However, for Effectively improving the compression effect in another video, the feature information from the input video must capture the feature content of the input video widely but succinctly.

Feature detection step, simulation step, association combining step, and tracking step After that, the feature information in the input video will be incorporated into the feature track set. To simplify this information into a convenient and representative form, the first step is to select representative or characteristic features of each feature trajectory. In one embodiment, the unique feature of a given feature trajectory is the first (frontmost) feature instance in the trajectory. In another embodiment, the unique feature of a given feature trajectory is the arithmetic mean of all feature instances in the trajectory. Selecting the unique feature of each feature track will simplify the feature information of the input video from the feature track set to the unique feature set.

Simplify the feature information in the input video into a convenient and representative form The next step is to bring together similar unique features. Unique features are gathered or clustered together using techniques well known in the art. In an embodiment of the tracker detailed above in the tracker, the cluster is based on the spectral color map of the unique features. The "U" and "V" (chrominance) components from the YUV color space data are treated as two-component vectors. The different values of the U/V components correspond to different colors in a spectral color map. A long profile will be generated from the color map and may contain any number k of boxes that summarize the full range of U/V component values. In one exemplary embodiment, k=5. In another embodiment of the SURF tracker detailed above in the tracker, the cluster is based on the length 64 SURF feature descriptor vector of the unique features. Once the feature model fields for the cluster (for example, the color strip map or the SURF descriptor in the above example) are established, any standard cluster algorithm can be applied to implement the cluster. In a preferred embodiment, the clustering is accomplished using a k-means clustering algorithm. The k-means algorithm divides all the unique features in the input video. Assigned to one of the m clusters. In one exemplary embodiment, m=5. For each cluster, the k-means algorithm computes the centroid representing the arithmetic mean of the cluster members.

Figures 4A and 4B are based on the spectrum tracker (Figure 4A) and SURF A screenshot of the feature-based display tool for the example implementation of the tracer (Fig. 4B). The result shown in the upper left display of each figure is the result of clustering the unique features based on the characteristic spectral color map (416 in Figure 4A) or the SURF descriptor of the feature (420 in Figure 4B). . Each unique feature represents one or more feature members (24 of 414). Each cluster (ten in 416 and twelve in 420) is the image point from the cluster centroid and the corresponding feature model of the centroid (the spectral color map in 416 or the SURF description in 420) Symbol) to indicate. Each cluster has a specific number of unique feature members. There are eight unique feature members 418 for the example color spectrum cluster 418 of Figure 4A and twenty more unique feature members 422 for the example SURF cluster 420 of Figure 4B.

It should be noted that if there are too many members in these m clusters, Apply a second layer of sub-clusters. In one exemplary embodiment, each of the m color spectral clusters is divided into 1 sub-cluster, where m is 5 and 1 ranges from 2 to 4.

After forming the initial set of m clusters, the features in the video will be input. The final step in simplifying the information into a neat, yet representative form is to select a subset of n cluster members to represent each cluster. The reason for this step is that a cluster may have many cluster members, and the number n of representative cluster elements must be very small for efficient use in compression; in one exemplary embodiment, n=5. The choice of representative cluster elements is often based on cluster centroids. In one embodiment, orthogonal matching tracking is used in a minimally redundant manner. The (Orthogonal Matching Pursuit, OMP) algorithm is used to select the n cluster members that most closely approximate the cluster centroid. In another embodiment, the n cluster members are selected to have the largest inner product with the cluster centroid; the cluster members selected in this manner will have greater redundancy than the cluster members selected using OMP.

Once you have selected the most representative cluster members of each cluster, you are ready to save the feature information for the input video. The saved feature information is composed of m clusters among n cluster members, each cluster member is a unique feature in a special feature track, and each unique feature has a associated with its corresponding feature region. The set of image points (16X16 in one of the embodiments of the present invention). Each cluster will have a centroid (the image points will be saved) and associated feature models (for example, a color strip map or a SURF descriptor). In addition, because the present invention uses the saved feature information to encode (for "offset processing, see below for details"), the saved feature information is also included in each cluster. The image points of the surrounding area of the feature area of the member. In one embodiment, the "peripheral area" is defined as an area that falls within a 16X16 macroblock in any direction, so that a feature area and its surrounding area comprise a 48X48 super area (super- Region). Therefore, the saved feature information is composed of image points of m clusters in n super regions, and image points and feature models of the m cluster centroids are added.

The feature-based processing stream (feature detection, simulation, association combining, tracking, unique feature selection, clustering, cluster member selection, and preservation of feature information) outlined above can be extended from one input video to multiple input video. In the case of more than one input video, the unique features representing the characteristic trajectories of all input video are used to create the clusters.

Reusing feature models for offline feature-based compression

Model-based compression architecture

Once the feature-based stream (300 in Figure 3) outlined above is applied to one When inputting video (or multiple input video), the saved feature information will be reused to improve a "target" video (the video to be encoded is likely to be different from the input video). The compression effect. Reusing feature information for compression appears in the Model-Based Compression Architecture (MBCF) outlined in the co-pending Application No. 940, which contains the relevant components of the case (and is typically indicated by 924).

The MBCF begins with a similar step to the feature processing stream outlined above: features are detected, simulated, and associated, but based on the target video. In a preferred embodiment, the features are detected using a SURF detection algorithm and the SURF descriptors are used to simulate and correlate the combinations.

Next, MBCF uses the feature trajectory to make the feature and the macro block have a relationship, as shown in Figure 5. A given feature trajectory will indicate the location of a feature in multiple frames, and there will be associated movement of the feature across multiple frames. The location of the feature can be projected into the current frame using the location of the feature in the two nearest frames preceding the current frame. Next, the projected feature location has an associated macroblock that is defined as a macroblock that has the greatest overlap with the projected feature location. This macroblock (now the target macroblock being encoded) has been associated with a particular feature trajectory adjacent to the macroblock's projected position in the current frame. (500 in Figure 5). A single macroblock may be associated with multiple features, so one embodiment of the MBCF will select the feature with the greatest overlap with the macroblock as the associated binding feature of the macroblock.

Next, MBCF calculates the projection features of the target macroblock and the current frame. Offset 510 between positions. When MBCF operates in line mode (completely generating prediction values from early decoded image points in the same video), this offset generates a target macro by using early feature instances in the trajectory of the associated feature. The predicted value of the block. The on-line type of the target macroblock can be generated by using the same offset in the early feature instance and the offset between the target macroblock and the projected feature position in the current frame to find the regions in the reference frame. Predicted value (520, 530).

Assuming that a target macroblock (the current macroblock being encoded), its associated features, and the feature trajectory of the feature are known, the MBCF will generate a primary or critical predictor of the target macroblock. The key predictive data (image points) comes from the most recent frame where the feature appears (before the current frame), hereinafter referred to as the key frame. Key predictive values are generated after selecting the moving model and image point sampling techniques. In one embodiment of the MBCF, the motion model may be "0th order" (assuming the feature is stationary between the key frame and the current frame) or "1st order" (assuming the feature moves in the second closest) The reference frame, key frame, and current frame are linear). In either case, the movement of the feature is applied (in the backward time direction) to the macroblock associated with the current frame to obtain the predicted value of the macroblock in the key frame. In one embodiment of MBCF, the image point sampling technique may be "direct" (where the motion vector is rounded to the nearest integer, and the image points of the key prediction values are taken directly from the key frame) , or "indirect" (where the interpolation technique of the conventional compression method (for example, H.264) is used to infer the key value of the motion compensation). Therefore, the MBCF invention has four different types of key predictions depending on the motion mode (0th or 1st order) and the sampling technique (direct or indirect).

MBCF also produces refined key predictions by simulating local deformations by a subtiling process. Different parts of the macroblock are calculated during the miniaturization process Part of the different moving vectors. In one embodiment of the MBCF, refinement can be achieved by dividing the 16×16 macroblock into four 8×8 quadrants and separately calculating the predicted values for each. In another embodiment, the refinement can be performed in the Y/U/V color space domain by separately calculating the predicted values of the Y, U, and V color channels.

In addition to the main/critical predictions of the target macroblock, MBCF will The secondary predicted value is generated based on the position of the associated feature in the reference frame before the key frame. In one embodiment, the offset from the target macroblock to the (projected) position of the associated feature in the current frame represents a motion vector that can be used to find the location of the feature in the past reference frame. To predict the value. In this manner, a given target macroblock with an associated feature may generate a large number of secondary predictions (one secondary prediction for each frame that has previously occurred). In one embodiment, the number of secondary predictions is limited by the ability to limit the search to a specific reasonable number (for example, 25) of past reference frames.

Once the primary (critical) predictions and secondary predictions for a target macroblock are generated After the value, the overall reconstruction of the target macroblock can be calculated based on the predicted values. In one embodiment of the MBCF, a conventional encoder is followed that is based only on critical predictive values and is therefore referred to as a Key-Only (KO) reconstruction.

In another embodiment of MBCF, the reconstruction is to add key predictions and The composite prediction value of the weighted version of one of the secondary prediction values is based. This algorithm is hereinafter referred to as PCA-Lite (PCA-L) and includes the following steps: generating a vector (1-D) version of the target macroblock and key prediction values. Next, these results are represented by the target vector t and the key vector k.

The key vector is subtracted from the target vector to calculate the residual vector r.

The secondary set of predictive values is vectorized to form a vector si (generally, it assumes that these secondary vectors have a unit norm). The key vectors are then subtracted from all of the secondary vectors to form a key deduction set si-k. The approximation effect is that the secondary vectors are projected onto the key vector.

Calculate the weight of each secondary vector c=r^T(s_i-k).

Calculate the composite predicted value of each secondary vector t^=k+c. (s_i-k).

In general, the steps in the PCA-Lite algorithm approximate the well-known orthogonal matching pursuit algorithm [Pati, YC et al., 1993, at the 27th Asilomar Conference, pages 40-44. Orthogonal matching tracking: Orthogonal matching pursuit (Recursive function approximation with applications to wavelet decomposition)], the meaning of the composite prediction value is that the main prediction value and the secondary prediction value have Non-redundant contribution. In another embodiment, the PCA-Lite algorithm described above is modified to replace the key vectors in steps 3 through 5 above by the average of the key vector and the secondary vector. This modified algorithm is hereinafter referred to as PCA-Lite-Mean.

The PCA-Lite algorithm provides different types of composite prediction values for bidirectional prediction algorithms found in some standard codecs. The standard bi-predictive algorithm uses a blended value of a plurality of predicted values based on the temporal distance between the reference frame of the individual predicted values and the current frame. Conversely, PCA-Lite blends multiple predictive values into a composite predictive value based on the content of individual predicted values.

Note that forming a composite prediction value as described above does not require feature-based simulation; the composite prediction value can be formed from any set of multiple prediction values for a given target macroblock. However, feature-based simulation provides multiple blocks from a given target macroblock. A set of natural associations of predicted values, and composite predictors provide an efficient way to combine information from such multiple predictors.

Model reuse for offline streaming in model-based compression architecture

The Model Based Compression Architecture (MBCF) can also operate in offline mode, utilizing feature information generated and stored based on the feature processing stream as outlined above.

In one embodiment, the offline mode MBCF uses the SURF detection algorithm to detect features in the target video, the SURF descriptor to simulate the detected features, and the 0th-order mobile mode (assuming the feature is critical) Key predictive values are generated under conditions of static and direct interpolation between the frame and the current frame.

Next, the offline mode MBCF reads the appropriate feature information from the input video(s) that have been stored by the feature-based processing stream. (Recall that the saved feature information is composed of image points of m clusters in n super regions, plus image points and feature models of the m cluster centroids.) The MBCF reads the cluster elements from the cluster of SURF descriptors whose SURF descriptors are closest to the feature associated with the target macroblock (the current macroblock being encoded).

Once read into the special cluster, the offline mode MBCF will then deviate from each of the super-regions in the cluster from the center of the super-region (the manner and the target macroblock deviate from their associated features in the target video) The same) extracts image points to produce secondary predictions. In this way, n secondary prediction values are generated, one for each cluster member.

In one embodiment, the secondary prediction values generated by the offline mode MBCF are then combined with key prediction values using a PCA-Lite algorithm or a PCA-Lite-Mean algorithm as described above.

In another embodiment, the secondary predicted values can be considered as the primary predicted value, if If they produce lower errors or coding costs, they can replace the key predictions within the video. In this embodiment (the primary predictive value may come from an offline source (outside the target video)), a normalization step (for example, assumed to be an affine movement model) can be applied to the offline prediction values to ensure more Closely match the target macro block.

In summary, offline mode MBCF reuses the feature model by following the steps below. Compressing: (1) detecting the characteristics of each frame in the target video; (2) simulating the detected features; (3) associating the features in different frames to generate the feature trajectory; (4) Using feature trajectories to predict feature locations in the "target" frame being encoded; (5) associating with macroblocks in the current frame near the predicted feature locations; (6) with these features Based on the position of the recently coded key frame to generate the key prediction value of the macro block in step 5; (7) by deciding the descriptor whose centroid descriptor is closest to the feature of the target video The cluster information is read into the feature information generated from the input video; (8) the secondary prediction value is generated from the feature information read in step 7.

Forming a feature model library

Simple model library: save pure feature information directly

As mentioned above, it can be generated from the input video and then retained a basic Feature information set. This feature information can then be reused in the Model Based Compression Architecture (MBCF) to improve the compression of another "target" video to be encoded. Saving the feature information directly in a file, database, or data store represents the simplest form of organizing and cataloging feature model libraries from one or more input video features.

In one embodiment, the features from the feature detection step and the feature tracking step The newsletter is stored in a file, database, or data store. This information may include, but is not limited to, the name of the input video to which the features are detected; a list of feature tracks, each of which has an associated feature ID; the track length of each feature track (equal to The number of feature instances contained in the trajectory) and its total bandwidth (defined as the number of total bits required by a conventional compression method (for example, H.264) to encode all feature instances in the trajectory); The type of detection of each feature instance in a track (for example, SURF, face), the frame in which the detection occurs, the x/y coordinates of the center of the feature, and the bandwidth of the feature; each feature track comes from An image point of a characteristic (representative) feature of the trajectory.

Note that importantly, the information from the feature detection steps and tracking steps of the input video is not directly used to compress the target video in the model-based compression architecture; however, if the feature model library needs to accumulate more than one input video. Feature information, feature detection and tracking information must still be saved, because when combining multiple video tracks, the composition of the feature clusters used for compression will also change.

In one embodiment, the information from the feature clustering step is stored in a file, database, or data store, separate from feature detection and tracking information. The feature cluster information may include, but is not limited to, a cluster list, each cluster has an associated index value; the number of members in each cluster and the feature model associated with the cluster centroid; from each cluster An image point located in the "super region" of the feature (which is a unique feature of a feature trajectory), and an associated feature model; Parameters associated with the way the cluster is implemented (for example, tolerances and iterations in the k-means clustering method)

There are several ways in which the feature model library needs to accumulate feature information for multiple input videos. In one embodiment, all of the feature tracks of the input video are directly aggregated, and the feature track summary set is re-clustered. However, this approach can be problematic when the total number of feature trajectories increases, as the size of the final feature cluster can become too large (making the clusters less informative) or the number of feature clusters will increase (and thus increase the number of features). The coding cost among the clusters).

In another embodiment in which the feature model library contains multiple input videos, the prioritization of the feature tracks is prior to prior to clustering. That is, the feature track summary set is deleted before the cluster is made, so that only the "most important" feature tracks are retained for clustering. In one embodiment, the trajectory bandwidth of the feature trajectory (which is defined as the number of total bits required for a conventional compression method (for example, H.264) to encode all feature instances in the trajectory) is arranged according to the trajectory bandwidth of the feature trajectory. Priority. Features that are difficult to encode by conventional compression methods are identified as high priority. In another embodiment, the prioritization of the feature trajectory is ranked according to redundancy (which is loosely defined as the number of repetitions of a feature in a trajectory (no variability)). Feature trajectory redundancy can be measured by computing various statistics (rank, condition number) associated with an overall matrix of different feature instances in the trajectory. High redundancy features are often repeated in the input video and are thus confirmed to be important for compression. In another embodiment, the prioritization of the feature trajectory is ranked according to similarities to particular types of important features (eg, faces). Features belonging to a particular feature type are identified as important. In further embodiments, the particular feature types may be specialized based on semantic content (eg, special mobile teams, special television programs, ..., etc.).

Figure 6A is a summary diagram for storing in the feature model library described above and And then the general steps of accessing the feature information, which are followed by the feature processing stream (FPS) as shown in FIG. After feature detection, simulation, association combining, and tracking, the FPS generates and stores unique features 610 of the feature trajectories. The FPS also generates and stores the spatially (SURF) descriptor 612 and the spectral (color strip map) descriptor 620 of the unique features. The FPS will cluster the spatial descriptors and the spectral descriptors 614 and calculate the cluster centroid. The FPS will generate a feature index value 616 from the clusters whose elements are descriptors of the cluster centroids. The repository then generates a classifier 618 based on the feature index values that can be used to access the features of the cluster. In FIG. 6B, when a new target video is encoded using the Model Based Compression Architecture (MBCF) described above, the FPS will access the index value 632 and retrieve the response in response to the detected feature in the target video. An associated result set 634 of multiple cluster members. The cluster members are then used in the MBCF, as described above, to help compress the corresponding feature regions in the target video. Those skilled in the art will appreciate that the methods described above can be performed in any order and need not be performed in the order described above.

Advanced Model Library: Hash-based indexing of video repositories

In addition to keeping the feature information directly in the file outlined above In addition to the simplest form of the feature model library in the case, database, or data store, the alternative is to use a hash-based indexing to form a more advanced way to access data from the video repository. Feature model library. In addition to the feature model 980 (FIG. 9A, FIG. 9B) generated by Feature Processing Streaming (FPS), a video repository also contains data image points from one or more input videos that have been processed by the FPS. . This is different from the simple feature model library that only contains feature image points and their associated models as described in the previous paragraph. All image points in the message. Hash-based indexing provides an efficient way to access feature information from a video repository. The feature-based processing is considered to be a sparse sample of the video data block 702 of FIG. 7, the dimensions of which are the number of frames (704A-704F) by the number of columns (708A-708F) by the number of rows (706A-706F). Feature instances often only appear in a small percentage of a given block of video data.

An example implementation of the method for generating a hash index based value is shown in FIG. 8A The flow chart of the example. The feature-based processing stream (FPS) in FIG. 8A begins 802 from the feature in the frame in the interrogation input video. The FPS then applies a hash tag to each of the detected features 804. The hash tag uses a one way hash function to convert the information used to identify the detected feature (its x/y position, frame, range, and associated model 980) into a hash value, such that later This feature can be easily accessed from the video repository. The FPS then adds each of the hashed features and its corresponding hash value to an index value 806, which is stored in the video repository along with the encoded video itself. The FPS then determines if all frames of the input video have been analyzed 808. If all frames have been analyzed, FPS will stop generating index value 810. If not all frames in the input video have been analyzed, the FPS will detect the feature 802 in the next frame.

Figure 8B shows the use of a method based on hash index values to access features. A flow chart of an example embodiment. The FPS analyzes the frame of the incoming video to detect feature 812. The FPS then applies the hash function to the detected feature 814 to generate its hash value. The FPS then uses the hash values of the detected features to search for index values for finding and extracting corresponding features (and its associated feature models 980) 816, 818 in the video repository. In another embodiment, the FPS extracts a plurality of feature models of a given feature from the video data block. The FPS will then compress 820 the checked based on the extracted feature model 980 or associated feature information. Measured features.

Those skilled in the art will appreciate that the compression method described above can be applied. To multiple frames, and compression does not have to be done frame by frame. However, the method illustrated in FIG. 8B merely illustrates the general principle behind accessing feature information from a sparsely populated video data block contained in a previously encoded video repository using index values; the accessed feature information is then Will be used to help compress new target video.

Note that the base feature-based processing stream (FPS) shown in Figures 8A through 8B Unlike the FPS outlined in the above paragraph (and shown in Figures 3 and 6A), in addition to feature detection, feature tracking and clustering are included. More notably, the FPS in Figures 8A-8B accesses feature information from the video repository (the video repository contains all of the image points of its input video), rather than containing only feature image points and The archive of information is accessed (as used by the FPS in Figures 3 through 6A).

Figure 8C is illustrated in Figures 8A through 8B in accordance with an exemplary embodiment of the present invention. Flowchart 822 of the general steps of the underlying FPS. The FPS processes an incoming video 824 and detects features from the video, as described in 832 of the present application. The FPS will then generate a hash-based index value 834 for the feature in the video. The FPS will utilize the extracted features 832 and the generated index values 834 to transcode the video 826 according to compression techniques known in the art and/or compression techniques described in this application. The FPS manages the spread 836 of the data based on the generated index value 834. For example, the encoded video will be stored in a special server cluster in the video repository. The video may be stored in a storage structure, a database, or other data structure organized for use in a large video collection. When requesting access to the video, the repository will load and access the requested video 830, which will stream the requested video 840. Management data After the cloth 836, the FPS will distribute the module 838 from the repository to help stream the video 840. In one embodiment, the FPS will distribute the module 838 from the repository to a client device to help stream video 840 on the device. Those skilled in the art will appreciate that the methods described above can be performed in any order and need not be performed in the order described above.

Use video repositories with general compression processing streams

Video repositories are not necessarily used in conjunction with feature-based processing streams. The collocation shown in Figure 8D does not necessarily involve the general use of a repository of model-based compression techniques. The Generalized Processing Flow (GPF) 850 of Figure 8D will first accept the target video 852 to be stored in the repository. The repository will transcode the video 854 according to compression techniques known in the art (not necessarily based on models) and/or compression techniques as described in this application. The GPF will store the video in a storage structure, database, or other data structure organized for use in a large video collection, as appropriate. When requesting access to the video, the GPF loads and accesses the requested video 858 from the repository. The GPF then streams the requested video 860. Those skilled in the art will appreciate that the methods described above can be performed in any order and need not be performed in the order described above. For example, in one embodiment, the GPF may transcode the video 854 after it has accessed the video 858 but before it streamed the video 860. In another embodiment, the GPF may transcode the video 854 after inputting the video 852, and then provide an additional transcoding to the video 854 after accessing the video 858 but before streaming the video 860. .

Application of model library

Basic operations: global model library and (personal) smart model library

Aspects of the invention may include features stored in the server/cloud Model library. By storing the model libraries in the cloud and accessing the feature information in the libraries when needed, the present invention can stream high-quality video with a lower bandwidth than the conventional codec, and the visual quality is degraded. Very little, not even any derogation. These models 980 can be reused not only in a single video (the "online" mode based on the model compression architecture [MBCF] described above) but also across multiple different, heterogeneous video (MBCF "Offline" mode). The system is capable of recognizing, validating, and reusing a model from one of the high quality video to process and present the video image in another high quality video. Reuse of model 980 reduces the file size of the library, allowing the device to reduce the necessary bandwidth when streaming video content.

The feature model libraries can exist in the cloud deployment (public or private), and the preferred system is downloaded to the user's mobile device only when needed. Technically and today's Amazon Kindle (trademark) and Apple iPad (trademark) device applications manage the content between the cloud and the user device, the present invention is able to store the model library offline and transfer the relevant model 980 to use when needed. The device is used to help with video compression/decompression.

A block diagram of an exemplary embodiment of a video repository 902 operatively coupled to client device 908 on network 170 is shown in FIG. 9A. Repository 902 includes an encoded video set 904. The video set 904 includes a first video set 906A, a second video set 906B, a third video set 906C, and an Nth video set 906D. Those skilled in the art will appreciate that video set 904 may contain any number of video or video sets. Video sets 906A through 906D within video set 904 may be associated with each other. For example, the first video set 906A may be a full season scenario for the first special television series. The second video set 906B may be a full season scenario for the second special television series. Similarly, the third video set 906C and the Nth video set 906D may contain other seasons or other television series. Those familiar with the technology will learn more about the video set 906A to Each of the 906Ds may contain the story of a television series, related movies (sequels or trilogy), mobile broadcasts, or any other related video.

Repository 902 will be operatively coupled to client device 908 on network 170. The guest The subscriber device includes a request generation module 914. The request generation module 914 sends a video request 916 to the repository 902 on the network 170. The repository 902 will receive the video request 916 at the request receiving module 918. The video request 916 is for the request for video contained in the video set 904. Upon issuing the video request 916, the client device 908 would expect to receive the requested video and, in response to the requested video, prepare an appropriate codec to decode the foreign bit string.

In order to send the requested video to the client device 908, the repository 902 will be requested The request receiving module 918 sends the requested video query 920 to the video set 904. The requested video query 920 may be for a request to initiate a query function for the video set 904 data structure. The requested video query 920 may also be a request for an index value that can be efficiently found in the video set 904 for the requested video. The video set 904 will be returned to the video query 920 to generate a requested video 922 to the stream generation module 924.

The stream generation module 924 generates a production associated with the requested video. The library 926 is added to the code 928 of the requested video. The generated library 926 (also referred to as a smart model library) contains the feature models needed to decode the requested encoded video 928. In one embodiment, the models in the generated smart model library 926 are inferred from the video repository and a hash index based reference to the feature model 980 of the video contained in the repository.

In another embodiment, the generated smart model library 926 The model is a global model library containing a set of reusable models (for example, feature models) 980 inferences. The models in the global model library can be reused not only for a single video but also for reuse in multiple different, heterogeneous video.

The video repository 902 stores a total of the encoded video 904, the global model library 980, or a hash-based index value that references the models of the video.

The generated library 926 and the encoded video 928 are transmitted over the network 170 to the client device 908. The library 926 can be transferred to any device containing a mobile device, such as an iPad, a smart phone, and a tablet. The client device 908 receives the generated library 926 and the encoded video 928 at the stream decoding module 910. The stream decoding module 910 decodes the encoded video 928 using the information in the generated library 926 and, optionally, other codecs known to the stream decoding module 910. The stream decoding module 910 outputs a decoded video 911. The decoded video 911 is transmitted to at least one of: memory 912A, display 912B, or storage module 912C.

Version design (personal) model library

9B is a block diagram of another exemplary embodiment of a video repository 902 for communicating with client device 908 over network 170. The operations of repository 902 and client device 908 are similar to those referenced in Figure 9A. However, additional modules, methods, and features are also illustrated in FIG. 9B. Those skilled in the art will appreciate that modules such as repository 902 and client device 908 are used interchangeably between the embodiments described herein in this application.

In one embodiment, repository 902 and client device 908 are configured to design a version of the library used to decode the video. Client device 908 includes a request generation module 914. As described above, the request generation module 914 sends a video request 916 to the request receiving module 918. The request receiving module 918 sends the requested video query 920 to the video set 904, such as Above. However, in one embodiment, the request receiving module 918 sends a client version query 952 of the requested video library to the version design module 954. The version design module will determine the client version 956 of the requested video library based on query 952. In many cases, a client device may request and download related video containing the associated codec or library. The example of related video is the follow-up story of the same TV program. Because the actors used in the TV program have a common relationship with the scenery, the subsequent stories contain similar frames. Another example is a mobile event where there is a commonality between venues, venues, mobiles, banners, or mobile devices in multiple frames. Therefore, if the client device 908 has previously downloaded an associated video and library, it may already have many or all of the necessary models needed to decode the encoded video 928. In this case, the client that wants to decode the encoded video 928 may have to update the library. Sending only the update, rather than the full library, preserves the bandwidth of the transmitted material to the client device 908, and because of the smaller download size, it increases the speed at which the user of the client device 908 begins viewing the requested video.

In one embodiment, the stream generation module 924 includes a differential library generation module 958 and a video encoding module 960. The stream generation module 924 receives the requested video 922 from the video set 904. The differential library generation module 958 receives the requested video and the client version 956 of the requested video library. In one embodiment, the differential library generation module 958 determines the client device 908 to use based on the requested video 922, the models 980, and the client version 956 of the requested video library. The update required to decode the video in the model-based compression architecture.

In another embodiment, the differential library generation module 958 will use the requested video 922, the hash index value, and the client version 956 of the requested video library. The basis is to determine the update required by the client device 908 to decode the video in the model-based compression architecture.

The differential library generation module 958 generates a differential library 962, which Only the updates (additional feature models) required by the library at the library storage module 964 already stored in the client device 908 are included. The video encoding module 960 generates the encoded video 928 based on the differential library 962 and the client version 956 of the library of requested video. Using a client-specific library version allows video distributors to provide different levels of viewing experience depending on the model received at the customer's premises. For example, one of the client library models can be used to help improve the quality of the video being viewed.

In another embodiment, the video encoding module 960 provides the best self-use. The compressed model produces the encoded video. The differential library generation module 958 generates the differential library 962 based on the knowledge of the model used to encode the video and the client version of the library resident in the client device. In this embodiment, only additional models, if any, are included in the differential library.

Client device 908 receives differential library 962 and encoded video 928. Client device 908 will receive differential library 962 at library configuration module 966. The library configuration module 966 loads the client version 956 of the requested video library from the library storage module 964. The library configuration module 966 combines the differential library 962 and the client version 956 of the requested video library into a combined library 970. The stream decoding module 910 then uses the combined library 970 to decode the encoded video 928 and generate a decoded video 911 that is spread to at least one of: memory 912A, display 912B, and The storage module 912C. The system recognizes, confirms, and reuses one of the high paintings A visual video model used to process and present video images in another high-definition video. Reuse of the model reduces the total file size of the libraries needed to decode multiple videos on client device 908 because the same model is reused to decode multiple videos.

Predictive model library

A block diagram of another exemplary embodiment of a video repository 902 coupled to client device 908 on network 170 is shown in FIG. For example, during the peak usage period of network 170, a predictive generation and dissemination library would benefit the user of client device 908. For example, assuming that the network has high traffic, if the repository does not have to transmit the library because the library has previously been generated and has been transferred to the client device 908, the network 170 can be used during periods of high usage. Small bandwidth.

The video repository 902 of FIG. 10 includes a stream generation module 924 that includes a predictive library generation module 1002. The predictive library generation module 1002 receives the user behavior description file 1006 generated by the user behavior description file module 1004. The user behavior description file module 1004 stores user information, such as demographic information, geographic information, social network connection information, or a mobile or mobile team group. The user behavior description file module 1004 may also include individual preference information of the type of video that the user may view. Some people may like baseball, NASCAR, and Family Guy; others may like Mad Men and Real TV. User preferences can be inferred from a Video-On-Demand (VOD) profile (eg, a list of videos previously downloaded from repository 902), subscribed from the user (eg, season pass) Inferred, inferred from the user's video queue, inferred from the user's prior purchase, or inferred from the collaborative filtering of any combination of such sources. User viewing preferences and behaviors can be used to refine the feature model library that is passed to individual devices; Further refinement can be achieved by combining user preferences and broadcast schedules.

The predictive library regeneration module 1002 will describe the file 1006 as a user behavior. A predictive encoded video request 1008 is generated based on the basis to generate a model library 102. For example, the predictive library generation module 1002 can generate a library of predictors for a particular television program, as shown in the user behavior description file 1006.

Predictive scatter and cache repositories improve video access, indexing, And archival. The expected demand context can help predict the distribution and cache of the repository and the libraries associated with the video.

In one embodiment, the demand context is predicted by dependency, VOD Pre-delivery, or scheduled broadcasts. The demand situation may include: long tail VOD (that is, requesting video that is not frequently selected), demographic behavior description file, broadcast schedule, mobile or mobile team group, social network, collaborative filtering, queue , all seasons, or pre-release purchases. Each scenario involves optimization of storage needs and video distribution.

In one embodiment, the demand context is a long tail VOD scenario. Long tail VOD The user is involved in selecting a video (possibly undesired video) from a video set to be streamed to the user. The video selection process is balanced to evenly access any video material in the episode. In the long tail VOD scenario, long tail VOD (without frequently selected video) is encoded using the high demand video feature model, thereby increasing the likelihood that the model material is available at the client device and making the residual video material easier to distribute (because Ideally, less residual video data will remain after disseminating higher demand data).

In another embodiment, the demand context is a recommendation system. Recommended system will be divided Analysis of historical video preferences of individual users, and driving users to choose downloads may be commensurate with the use The historical video information of the viewers. The feature model is organized based on the user's historical video preferences to support the dissemination of context. Feature models associated with anticipated user needs are passed in advance to avoid high network demand scenarios.

In another embodiment, the demand context is a regional preference (for example, from Demographic behavior description file information). Traditional preferences can be inferred from the demographic behavior description information, so the repository can deliver content to regional users. The content provider can assume resource costs in order to communicate this content to such users.

In another embodiment, the demand context is a broadcast schedule. Models are delivered by broadcast schedules (for example, planned network schedules) or by broadcast schedules. The model will be created based on records from one of the channels and reused to encode programs on another channel or another program on the same channel. Models can be inferred from video data that can be taken from DVDs, cables, .... In one embodiment, the transmission of the model may include enhanced information that may improve the quality and/or resolution of the video material. The repository will provide a deduced "quality" service that will supplement the existing broadcast model.

In another embodiment, the demand context is a mobile or mobile team group. Models based on the user's mobile/team group will have video data consistency (for example, the same player's face, team logo and uniform, team's venue, etc.) and will be geographically targeted. spread. These models may be based on multiple viewing views, replays, high time resolution, and immediate needs. The distribution of these models may be tiered.

In another embodiment, the demand context is a social network connection and/or collaborative filtering. The social network will anticipate the demand by determining the video needs of the user's peer/relative/friend. Collaborative filters indirectly predict user needs based on peers. Models will be social networks or Based on the collaborative filter, it is inferred from the video that the user expects to watch.

In another embodiment, the demand context is a queue of video.伫 column may be through The user who wants to be queued or time delayed/offset video data selects the expected priority order defined by the user. The model is spread based on the optimal model usage rate relative to the contents of the queue.

In another embodiment, the demand context is all season pass. When the content of the demand If the monetization or exclusivity is increased and more directly related to the content itself, the model may be based on an additional fee, where the surcharge is not a one-time use. In this demand scenario, a higher threshold is used to preserve the content being distributed and the necessary content delivery. In addition, the dissemination method has a high degree of data self-similarity as mobile video data (for example, the same actors, scenery, or a set of plots).

In another embodiment, the demand context is issued in advance. Pre-release purchases include pre-release video material, trailers, short films, or sample "webisode". The dissemination of video with a model library is based on the pre-release purchases that are delivered.

The use context of the organization that determines the repository data may be determined in advance or not determined in advance. The pre-determined use context will focus on the general performance of the video material. Non-determined usage scenarios focus on the specific performance of video material.

In one embodiment, the video set 904 of FIG. 10 generates a video 1010 for encoding in response to the predictively encoded video request 1008, and provides a hash-based index value for the associated video model.

In another embodiment, the predictive library generation module 1002 also retrieves the model 982 from a model library and uses the model to replace the hash-based index value. prediction The library generation module 1002 then generates a predictively generated library 1012. The predictively generated library 1012 is transmitted to the client device 908 on the network 170, and the client device 908 stores the predictively generated library 1012 in the library storage module 964. Those skilled in the art will appreciate that client device 908 stores the predictively generated library 1012 and uses the predictively generated library 1012 when it receives the appropriate encoded video to decode. Those skilled in the art will also appreciate that other embodiments of the repository 902 on the client device 908 can be combined with the predictive library to create an embodiment. For example, a library may be predictively generated and transmitted to client device 908 in a different manner, as described in connection with FIG.

The embodiments of the invention described above can be used with each other and not with each other. Used to form additional embodiments of the invention.

The present invention has been particularly shown and described with reference to the exemplary embodiments of the present invention, but those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the accompanying claims. The scope of the invention encompassed. For example, various system components, such as codecs, encoders, and decoders, have been cited herein; however, those skilled in the art will appreciate that any other suitable hardware or software digital processing can be used. To implement the video processing technology described in this article. For example, the invention can be implemented in a wide variety of computer architectures. The computer network of Figures 11A and 11B is for illustrative purposes only and is not intended to limit the invention.

The form of the invention may be a fully hardware embodiment, a fully software embodiment, or an embodiment containing both a hardware component and a software component. In one preferred embodiment, the invention is embodied in a software that includes, but is not limited to, firmware, resident software, microcode, and the like.

Furthermore, the form of the invention may be a computer program product accessible from a computer-usable or computer-readable medium for providing code for use by a computer or any instruction execution system, or in conjunction with a computer or any instruction. Execute the system to provide the code. For the purposes of this description, a computer-usable or computer-readable medium may contain, store, exchange, transmit, or transmit the program for use by any system, device, or device used by the instruction, or An instruction execution system, device, or device that contains, stores, exchanges, propagates, or transmits any device of the program.

The media may be electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems (or devices or devices) or media. A data processing system suitable for storing and/or executing code includes at least one processor coupled directly or indirectly to a memory component via a system bus. The memory elements may include regional memory, a large number of banks, and cache memory that are utilized during actual execution of the code, and the cache memory temporarily stores at least a particular code to reduce The time at which a large amount of storage is captured.

A network switch may also be coupled to the system to allow the data processing system to be coupled to other data processing systems or remote printers or storage devices via an intermediate private or public network. Data modems, cable modems, and Ethernet cards are only available in a variety of currently available types of network adapters.

In one embodiment, one of such environments is illustrated in Figure 11A. The (multiple) client computer/device 1110 and a cloud 1112 (or server computer or cluster thereof) provide processing, storage, and input/output devices for executing applications and the like. The (multiple) client computers/devices 1110 may also be linked to other computing devices via communication network 1116, which include other client devices/processors 1110 and (multiple) server computers 1112. communication Network 1116 may be a remote access network, a global network (for example, the Internet), a worldwide collection of computers, a regional network or a wide area network, and currently using individual protocols (TCP) /IP, Bluetooth, ..., etc.) Part of the gateway that communicates with each other. Other electronic devices/computer network architectures are suitable.

Figure 11B is a diagram showing the internal structure of a computer/computing node (e.g., client processor/device 1110 or server computer 1112) in the processing environment of Figure 11A. Each computer 1110, 1112 includes a system bus 1134, wherein the bus is a set of real or virtual hardware lines for data transfer between components of a computer or processing system. The busbar 1134 is basically a common conduit that connects different components of a computer system (for example, a processor, a disk storage, a memory, an input/output port, etc.) for use in the components. Transfer information. An I/O device interface 1118 attached to the system bus 134 for connecting various input devices and output devices (eg, keyboard, mouse, display, printer, speaker, etc.) to the computer 1110 , 1112. The network interface 1122 allows the computer to connect to various other devices that are attached to a network (for example, the network shown at 1116 of Figure 11A). Memory 1130 is computer software instructions 1124 and data 1128 used to implement embodiments of the present invention (for example, the codecs, video encoders/decoders, and feature models described in all of Figures 1 through 10) , model library and support code) provide volatile storage. The disk storage 1132 provides non-volatile storage for computer software commands 1124 (equivalent to "OS program 1126") and data 1128 used to implement embodiments of the present invention; it can also be used to store models for long-term storage. Or used to store video in compressed format. Central processor unit 1120 will also be attached to system bus 1134 and used to execute computer instructions. Please note that throughout the text, "computer software instructions" and "OS programs" are equivalent.

In one embodiment, the processor standard programs 1124 and 1128 are a computer program product (collectively 1124) that includes computer readable media that can be stored in the storage device 1128, which is provided for At least a portion of the software instructions of the system of the present invention. As is well known in the art, computer program product 1124 can be installed by any suitable software installation program. In another embodiment, at least a portion of the software instructions can also be downloaded over a cable connection, a communication connection, and/or a wireless connection. In other embodiments, the inventive program is a transmitted signal that is presently on the media (for example, radio waves, infrared waves, laser waves, sound waves, or on a global network (eg, the Internet) Computer program to transmit signal 1114 (in Figure 11A). The carrier medium or such carrier signals provide at least a portion of the software instructions for the standard programs/programs 1124, 1126 of the present invention.

In an alternative embodiment, the transmitted signal is an analog carrier or digital signal carried in the propagation medium. For example, the transmitted signal may be a digital signal that is transmitted over a global network (for example, the Internet), a telecommunications network, or other network. In one embodiment, the propagated signal is transmitted in the propagation medium for a period of time, for example, sent in a packet on a network in a period of milliseconds, seconds, minutes, or longer. The instructions of the software application. In another embodiment, the computer readable medium of the computer program product 1124 is a broadcast medium that the computer system 1110 can receive and read, for example, by receiving the broadcast medium and recognizing the being present in the broadcast medium. Propagation signals, as described above for computer program communication products.

It should be noted that the figures described herein illustrate example data/execution paths and components; however, those skilled in the art will appreciate the data operations, data arrangements, data flows to/from such individual components. Will depend on the party of the video material to be compressed The style and type change. Therefore, any arrangement of data modules/data paths can be used.

The present invention has been particularly shown and described with reference to the exemplary embodiments of the present invention, but those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the accompanying claims. The scope of the invention encompassed.

10-1~10-n‧‧‧Featured examples of detection

20-1~20-n‧‧‧ frame

30-1~30-n‧‧‧Characteristic area

40‧‧‧Overall matrix

Claims (23)

A method of providing video data, comprising: encoding a target video stream by using a feature-based compression method from a feature model of a global feature model library, the code implicitly utilizing the feature models to represent the a macroblock in the target video for encoding to cause encoded video data; and transmitting the encoded video data to a requesting device upon receipt of the command, the features from the global feature model library The model allows access by the requesting device and enables decoding of the encoded video material at the requesting device; wherein the global feature model library is formed by receiving one or more input video, each An input video is different from the target video stream; and feature information and a different feature model are generated for each of the input videos. The method of claim 1, wherein the feature-based compression method applies feature-based prediction to a plurality of different video sources based on the feature models. The method of claim 1, wherein the global feature model library is further formed by storing the feature models generated by the input video locations in a data storage body or a cloud storage body. The storage or cloud storage system provides a suitable feature model, the feature based compression method and the request device. The method of claim 1, wherein the global feature model library is further formed by: identifying and indexing features in the input video for each input video, such The indexed features form the individual feature model, the indexing arrangement including a location for the identified feature in the input video for each identified feature; and wherein the encoded video is decoded at the requesting device The data system uses the indexed features and corresponding feature locations in the input video to retrieve the feature models and decode the encoded video material. The method of claim 4, wherein the indexing of the features is based on a hash index. The method of claim 1, wherein the feature model from the global feature model library forms a working model subset library, which is specialized according to the requesting device or according to the target video stream. According to the method of claim 1, wherein the feature model from the global feature model library forms a working model subset library, which is a differential program based on the state of any library in the requesting device. Library. The method of claim 1, wherein the feature model from the global feature model library forms a working model subset library, which is a predictive model library, which is used at the end of the request device. The behavior description file is a function to store the feature model. The method of claim 8 wherein the predictive model library has modifiable (settable) parameters for application to a wide variety of demand scenarios. A video data system comprising: a repository for storing video data and serving as a streaming video source; and a codec operatively coupled to the repository, and the codec is responsive to a particular The video request is executed by a processor for (i) encoding corresponding to the requested particular view The stored video data in the repository, and (ii) streaming the encoded video data from the repository, wherein the codec is based on using a feature model from the global feature model library Feature prediction, wherein the global feature model library is formed by receiving one or more input videos, each input video being different from the stored video data in the repository corresponding to the requested specific video; And generating feature information and a different feature model for each of the input videos; causing the codec to be based on the stored video data in the repository corresponding to the requested specific video. The different video data is applied based on feature prediction, and the plurality of different video materials include the input video of the global feature model library. According to the video data system of claim 10, wherein the codec is: encoding the stored video data in the repository by feature-based compression based on feature prediction based on the features, and The encoded video data is transmitted to a requesting device, the encoded video data being from the streamed video material of the repository. The video data system of claim 11, wherein the feature models from the global feature model library are accessible to the requesting device and enable decoding of the encoded video material at the requesting device. According to the video data system of claim 11, wherein the global feature model library is further formed by: identifying and indexing features in the input video for each input video, the indexed features Forming the individual feature model, the indexing arrangement is included for each An identified feature to indicate a location of the identified feature in the input video; and wherein decoding the encoded video data at the requesting device uses the indexed features and corresponding features in the input video Positioning to obtain the feature models and decoding the encoded video material. According to the video data system of claim 13, wherein the indexing of features is based on a hash index. According to the video data system of claim 10, wherein the feature models are stored in the repository. According to the video data system of claim 10, wherein the feature model from the global feature model library forms a working model subset library, which is in accordance with the requesting device or according to the streamed encoded video. The information is specialized. According to the video data system of claim 10, wherein the feature model from the global feature model library forms a working model subset library, which is based on the difference of the state of any library in the requesting device. Library. According to the video data system of claim 10, wherein the feature model from the global feature model library forms a working model subset library, which is a predictive model library, which is end user The behavior description file is a function to store the feature model. According to the video data system of claim 18, the predictive model library has modifiable parameters for application to a wide variety of demand scenarios. A computer program product comprising a code component for controlling a computer to perform any of the systems specified in the scope of the preceding claims when the code component is loaded into a computer. A computer program product comprising a code component, when the code component is loaded into a battery In the brain, the computer is controlled to execute instructions to help implement the system of claim 1 of the patent. A computer program product comprising a code component that, when loaded into a computer, controls the computer to execute instructions to assist in implementing the system of claim 10 of the patent application. A video data system comprising: a repository component for storing video data and serving as a streaming video source; and a codec component operatively coupled to the repository component, and wherein the codec component is Responding to a request for a particular video is performed by a processor component for (i) encoding stored video material in the repository component corresponding to the requested particular video, and (ii) streaming from the repository component The encoded video material, wherein the codec component is based on feature prediction using a feature model from a global feature model library, wherein the global feature model library is formed by: receiving one or more Input video, each input video being different from the stored video data in the repository component corresponding to the requested particular video; and generating feature information and a different feature model for each of the input video;俾 causing the codec component to be based on the stored video material in the repository component corresponding to the requested particular video Feature-based prediction is applied to a plurality of different video data, the plurality of different video materials including the input video of the global feature model library.