WO2008079330A1

WO2008079330A1 - Video compression with complexity throttling

Info

Publication number: WO2008079330A1
Application number: PCT/US2007/026156
Authority: WO
Inventors: Philip G. Jacobsen; Scott C. Labrozzi
Original assignee: Inlet Technologies
Priority date: 2006-12-22
Filing date: 2007-12-21
Publication date: 2008-07-03

Abstract

Feedback control for a video compression unit is provided by receiving an actual compression time used in the compression of a frame of a video signal by the video compression unit at a compression complexity value, adjusting the compression complexity value in response to the actual compression time, and supplying the adjusted compression complexity value to the video compression unit for use in compressing at least one other frame of the video signal. The actual compression time may be compared to a target compression time, and the compression complexity value may be adjusted in response to the comparison of the actual compression time to the target compression time.

Description

VIDEO COMPRESSION WITH COMPLEXITY THROTTLING

CROSS REFERENCE TO RELATED APPLICATION [0001] The present application claims the benefit of and priority to U.S. Provisional Patent Application No. 60/876,643, filed December 22, 2006, entitled "Video Compression with Complexity Throttling," the disclosure of which is hereby incorporated herein by reference in its entirety.

BACKGROUND

[0002] The present application relates to video compression, and in particular to controlling video compression settings.

[0003] Digital video in its rawest form is represented by an enormous amount of data. Video itself can be characterized as being composed of frames or pictures, and the number of frames per second defines the frame rate. Each picture is a two-dimensional array of picture elements, or pixels, each digitally represented by some number of bits that define the pixel's color and/or brightness. The size of the pixel array specifies the frame's width and height.

[0004] For example, referring to FIG. 1 , a single frame 10 of a raw video signal includes a two-dimensional array of pixels 12. Each of the pixels 12 can be encoded as an m-bit value 14. A video signal is made up of a time- ordered sequence of frames 10A, 10B, etc. Standard video formats, such as MPEG-2, also provide that digital audio signal and/or a data signal can be encoded along with the digital video signal.

[0005] Accordingly, the aggregate video data rate is simply the product of the number of bits per pixel, the number of pixels per frame, and the frame rate. As an example of video's enormous size, raw digital high- definition digital video is often represented by 20 bits/pixel, has a frame size of 1920 x 1080 pixels, and a rate of 30 frames per second. Thus, the aggregate data rate of this form of high definition video is 20 x 1920 x 1080 x 30 = 1 ,244,160,000 bits/second (i.e., 1.2 gigabits/sec). To put this into perspective, a two-hour movie encoded as raw video would occupy 8640 gigabits (Gb) or 1080 gigabytes (GB). A conventional DVD stores 4.7 GB of data. It would therefore require 229 conventional DVDs to store a two-hour high-definition movie in this raw form.

[0006] As a result, nearly all video applications employ compression to minimize the space required to store and/or transmit video. Before compressed video can be viewed or displayed, it must be decompressed or reconstructed into a raw form. Some video compression techniques or formats are lossless, meaning that the original pixel data can be reconstructed, bit for bit, from the compressed form. However, such compression formats are not nearly as efficient at minimizing video's massive size as lossy compression techniques, in which not all information is retained. In lossy compression, there is no guarantee that the original pixel data can be reconstructed bit for bit from the compressed form. The art of lossy video compression, therefore, deals with minimizing the perceptual difference between the original and reconstructed video frames. The challenge in video compression is to reduce the aggregate data rate as much as possible while maintaining acceptable video quality following decompression.

[0007] When performing video compression, the original uncompressed video is obtained from a video source, e.g., a storage device, such as a hard disk, and/or a "live" source such as a video camera with no intermediate storage medium. A live source delivers video frames at a fixed time interval (the "frame interval"). As a result, the time to compress each frame, on average, should be less than or equal to the frame interval. This situation is often referred to as "real-time" compression. If the encoder cannot operate in real time, it may be forced to drop some frames in order to keep up with the frame delivery rate.

[0008] A typical practice for real-time compression is to buffer incoming video frames in a memory storage, such as a RAM, before supplying the frames to the video encoder. The video frames are placed in the memory before being delivered to the encoder, as illustrated in FIG. 2. As shown therein, a video source 20, which may include a real-time video source and/or a storage device, supplies raw video frames 24A to 24C to a buffer memory 22. The buffer memory 22 can store a finite number of video frames 24A to 24C. The video frames 24A to 24C are read from the buffer memory 22 by a video compression module 26; which compresses the frames according to video compression settings therein. The resulting compressed frames are output by the video compression module 26. The compressed frames may, for example, be stored at a local storage device and/or transmitted, for example, to a broadcast subscriber.

[0009] The configuration illustrated in FIG. 2 allows the video compression module 26 to spend more time encoding some frames and less on others. For example, the configuration illustrated in FIG. 2 may allow the video compression module 26 to spend more time than the frame interval on a given frame. However, as the buffer memory 22 has a finite capacity, it can only hold a limited number of video frames. As video data is extremely large, the number of video frames that can practicably be held in storage is typically relatively small compared to the overall size of the digital video signal. Thus, while the buffer memory 22 can temporarily store video frames, there is a fixed limit on how many frames can be buffered at once. Therefore, if the average compression time per frame exceeds the video capture rate, the memory may quickly run out of space, which may result in the dropping of frames from the buffer memory 22. Buffering, therefore, cannot overcome the need for the average compression time per frame to be less than the video capture rate.

[0010] When the video source 20 is a memory storage device, there is no predefined constraint on the video compression time per frame, and there is no possibility of losing frames. Nevertheless, there are often practical limits on how much time can be devoted to video compression. Limited computing resources and/or fixed publishing deadlines for distribution of the compressed video may, for example, restrict the time allowed for compression.

[0011] Video compression requires significant computational power. In addition, there is a direct correlation between the computational requirements for video compression (in terms of processing time and/or processing power) and the resulting quality for a desired target data rate. Generally, the more time that is spent on deciding how to best compress a frame, the higher the quality of the resulting compressed frame will be. Conventional video-compression algorithms are designed for implementation on platforms with differing computational power. As a result, the compression algorithms are flexible in their demands for computational power. The algorithms contain compression parameters, for example, that may be adjusted to control the tradeoff between complexity and quality. In addition, optional compression tools may be turned on or off depending on the available computational resources and/or the desired quality of the compressed frames.

[0012] The compression complexity and the resulting compression time per frame is dependent on the contents of the video frame as well as the algorithm employed. An all-black video frame is significantly easier to compress than a frame containing a complex scene, for example. The compression time is also highly dependent on the amount of motion in the encoded video signal and/or the pictorial differences between temporally successive frames. As a result, the compression time per frame and the resulting compression time required to encode an entire video clip is highly dependent on the content. Given the degree and unpredictability with which frames can vary within a video clip, it is extremely difficult to predict with any certainty the amount of time and computational resources required to encode it.

[0013] Conventional video-compression algorithms generate various types of compressed frames. For example, the MPEG-2 standard defines I (Intra)-Frames, P(Predictive)-Frames and B(Bidirectional predictive)-Frames. I-Frames are produced independently of other frames (i.e. only from data in the current frame) and require few encoding tools. For example, I-Frames do not encode motion relative to other frames. Motion encoding is one of the most computationally intensive operations in video compression. I-Frames are therefore relatively simple and quick to compress. P-Frames are produced using previous frames and require more processing, tools and time than I-Frames. B-Frames are produced using previous and future frames and require the most processing, tools and time.

[0014] Generally, during compression an encoder determines in advance what frame types will be produced next. Furthermore, the order of frame encoding may depend on the type of frame that is being produced. For example, as illustrated in FIG. 3, a video encoder may encode a first frame 1 as an l-frame. The encoder may then skip ahead and encode the next P- frame (frame 5) using the encoded l-frame (frame 1 ) as a reference frame. The intermediate frames (frames 2-4) are then encoded as bi-directional predictive B-frames using both the l-frame (frame 1) and the P-frame (frame 5) as reference frames. Next, another P-frame (frame 9) is encoded using the preceding P-frame (frame 5) as a reference frame. Then, the intermediate frames 6-8 are encoded as B-frames using both the first and second P-frames (frame 5 and 9) as reference frames. Another l-frame is then encoded as frame 10. Typically, a new l-frame is inserted for every 12 to 15 B- and P- frames. An l-frame may also be produced when differences in frame data make the generation of effective P- or B-frames difficult.

[0015] In conventional video compression, a fixed set of compression parameters and tools is defined based on the capabilities of the compression hardware and time limitations. The parameters and tools are defined prior to beginning compression, and typically represent a worst-case guess based on an estimate of the most complex and resource-consuming content possible. A worst-case guess is typically used, because an incorrect choice can result in dropped frames, which can significantly degrade the quality of the resulting compressed signal, and re-encoding the video signal can involve a significant amount of time and expense. The result of this guess is generally a less-than- optimum utilization of the computational resources within the allowed compression time. For example, the compression process either completes sooner than necessary or does not utilize all of the computational power available to it.

[0016] When compressing live content (which, as previously discussed, requires real-time processing), choosing stringent parameters and complex tools may cause the average per-frame processing interval to exceed the frame interval (i.e. the time between of arrival of frames from the live content source). Even if this occurs only for a short period, frames may be dropped if the processing buffer fills up. As a result, achieving a desired level of buffer and/or processor resource utilization while respecting time constraints and the need to preserve all frames is often an uncertain, trial- and-error process. SUMMARY

[0017] Some embodiments provide methods of providing feedback control for a video compression unit, including receiving an actual compression time used in the compression of a frame of a video signal by the video compression unit at a compression complexity value, adjusting the compression complexity value in response to the actual compression time, and supplying the adjusted compression complexity value to the video compression unit for use in compressing at least one other frame of the video signal.

[0018] The methods may further include comparing the actual compression time to a target compression time, and the compression complexity value may be adjusted in response to the comparison of the actual compression time to the target compression time.

[0019] The methods may further include increasing the compression complexity value when the actual compression time is less than the target compression time, and decreasing the compression complexity value when the actual compression time is greater than the target compression time.

[0020] The methods may further include comparing a buffer utilization level of a buffer that stores video frames for processing by the video compression unit to a threshold buffer utilization level, increasing the compression complexity value when the buffer utilization level is lower than the threshold buffer utilization level, and decreasing the compression complexity value when the buffer utilization level is greater than the threshold buffer utilization level.

[0021] The methods may further include determining a buffer utilization level of a buffer that stores video frames for processing by the video compression unit and, when the actual compression time is greater than the target compression time, increasing the compression complexity value only when the buffer utilization level is greater than a threshold buffer utilization level.

[0022] The resource may include a buffer utilization level of a buffer that stores video frames for processing by the video compression unit and the compression parameter may include a compression complexity, and the methods may further include increasing the compression complexity value when the buffer utilization is less than a first threshold level and decreasing the compression complexity value when the buffer utilization level is greater than a second threshold level.

[0023] Methods of compressing a video signal including a plurality of video frames according to some embodiments include compressing at least one video frame in accordance with video compression settings, following compression, modifying the video compression settings based on an actual compression time of the at least one video frame and at least one target setting, and compressing at least one other video frame in accordance with the modified video compression settings.

[0024] The methods may further include repeating steps of compressing frames according to the video compression settings modifying the video compression settings in response to actual compression times throughout compression of the video signal.

[0025] The target setting may include a target compression time per frame. In some embodiments, the target settings include a target memory level of a buffer memory that is configured to store the video frames prior to compression.

[0026] Furthermore, the settings may be altered based also on a frame type and/or a desired frame complexity. In some embodiments, the settings may be altered based also on processor utilization of a processor of the video compression unit.

[0027] A video compression system according to some embodiments includes a throttling control unit configured to receive a signal indicative of an amount of a resource used in the compression of a frame of a video signal using a compression parameter, to adjust the compression parameter in response to the signal, and to supply the adjusted compression parameter as an output signal.

[0028] The resource may include an actual compression time of the frame, a buffer utilization level and/or a processor utilization level, and the compression parameter may include a compression complexity. In some embodiments, the compression parameter may include a target processor utilization and/or a target compression time per frame. [0029] The system may further include a video compression unit configured to receive the adjusted compression parameter from the throttling control unit and to compress the video signal in accordance with the adjusted compression parameter. The video compression unit is further configured to provide a feedback signal to the throttling control unit.

[0030] The system may further include a buffer memory configured to store frames of the video signal for processing by the video compression unit, and the buffer memory may be configured to provide a buffer utilization level to the throttling control unit.

[0031] Methods of providing feedback control for a video compression unit according to further embodiments include receiving a signal indicative of an amount of a resource used in the compression of a frame of a video signal by the video compression unit using a compression parameter, adjusting the compression parameter in response to the signal, and supplying the adjusted compression parameter to the video compression unit.

[0032] The resource may include an actual compression time of a frame by the video compression unit, a buffer utilization level of a buffer that stores video frames for processing by the video compression unit and/or a processor utilization of a processor in the video compression unit.

[0033] The compression parameter may include a compression complexity, a target processor utilization and/or a target compression time per frame.

[0034] The methods may further include determining a frame type for a subsequent frame, and the compression parameter may be adjusted in response to the frame type.

[0035] The resource may include a buffer utilization level of a buffer that stores video frames for processing by the video compression unit and the compression parameter may include a compression complexity. The methods may further include increasing the compression complexity when the buffer utilization is less than a first threshold level and decreasing the compression complexity when the buffer utilization level is greater than a second threshold level. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate certain embodiment(s) of the invention. In the drawings:

[0037] FIG. 1 schematically illustrates a video signal as a sequence of frames made up of pixels.

[0038] FIG. 2 illustrates conventional video compression systems and/or methods using a buffered input signal.

[0039] FIG. 3 illustrates encoding of various types of video frames.

[0040] FIG. 4 illustrates video compression systems and/or methods according to some embodiments.

[0041] FIGS. 4, 5, 6, 7 and 8 are flowcharts illustrating systems and/or methods according to some embodiments.

DETAILED DESCRIPTION OF EMBODIMENTS [0042] Embodiments of the present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout.

[0043] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present invention. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. [0044] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes" and/or "including" when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

[0045] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0046] Some embodiments may facilitate adjustment of video compression settings during video compression to increase utilization of available computational resources. According to some embodiments, the complexity of a video encoding/compression algorithm can be dynamically adjusted. Using a "complexity throttling" technique according to some embodiments may reduce the need to try to predict the optimal compression settings before beginning the compression process. Rather, according to some embodiments, compression settings can be adjusted during the compression process to increase use of computational resources while enabling the compression process to complete within a target time.

[0047] In some embodiments, complexity throttling is achieved by utilizing a feedback loop in which compression settings, compression complexity and/or other parameters are adjusted in response to feedback inputs. The feedback inputs can include, for example, the actual compression time required for one or more previous frames and the target compression time per frame. Other possible feedback inputs are described below.

[0048] Systems and/or methods for performing video compression using complexity throttling according to some embodiments are illustrated in FIG. 4. As shown therein, a video source 120, which may include a real-time video source and/or a storage device that stores a digital video signal, supplies raw video frames 124A to 124C to an optional buffer memory 122. If present, the buffer memory 122 can store a finite number of video frames 124A to 124C. Video frames 124A to 124C are read from the buffer memory 122 (or received directly from the video source 120) by a dynamically configurable video compression unit 126, which compresses the frames according to dynamically configurable video compression settings. The resulting compressed frames 130 are output by the video compression unit 126. The compressed frames may be, for example, stored at a local storage device and/or transmitted, for example, to a broadcast subscriber.

[0049] A throttling control unit 140 monitors actual and desired video compression parameters and provides a feedback control signal 142 to the video compression unit 126 in response to the video compression parameters. The feedback control signal 142 can be generated in response to a number of system and/or compression parameters supplied by the buffer memory 122 and/or by the video compression unit 126.

[0050] The feedback control signal 142 can take many forms. In some embodiments, the feedback control signal 142 can provide a metric that can be used by the video compression unit 126 to control the video compression settings. For example, in some embodiments, the feedback control signal 142 can comprise a number between 0 and 100 that corresponds to a desired compression complexity level, where "0" represents the least complex algorithm that can be implemented by the video compression unit 126 and "100" represents the most complex algorithm that can be implemented by the video compression unit 126. The complexity level supplied in the feedback control signal 142 can be interpreted by the video compression unit 126 in many different ways. That is, the video compression unit 126 can decide how best to achieve the desired complexity level supplied by the throttling control unit 140 via the feedback control signal 142.

[0051] In some embodiments, the feedback signal could represent a requested percentage increase or decrease in complexity. For example, a feedback signal of 20 could represent a requested 20% increase in complexity, while a feedback signal of -10 could represent a requested 10% decrease in complexity.

[0052] In some embodiments, the feedback control signal 142 can comprise a number between 1 and 100 that represents a desired level of processor utilization of a processor of the video compression unit 126. Thus, in response to the feedback control signal 142, the video compression unit 126 can increase or decrease the level of complexity of the compression algorithm in an attempt to achieve a desired level of processor utilization.

[0053] In further embodiments, the feedback control signal 142 can include a target compression time. The video compression unit 126 can adjust the level of complexity of the compression algorithm in an attempt to achieve the target compression time. As will be appreciated by those skilled in the art, a more complex compression algorithm may generate a more efficiently compressed video signal for a given quality level and/or a better quality level for a given compression ratio.

[0054] In some embodiments, the feedback control signal 142 can provide explicit commands for altering specific compression settings of the video compression unit 126. For example, the feedback control signal 142 could specify a particular compression parameter for the video compression unit 126, such as motion vector search area size, as discussed in more detail below.

[0055] The video compression parameters monitored by the throttling control unit 140 may include, for example, information supplied by the video compression unit 126 via signal 144, including the actual compression time used in one or more previous frames and/or the processor utilization of the video compression unit 126. The video compression parameters can also include the target compression time per frame, the target memory level of the buffer memory 122, the encoding frame type of the next frame, and/or the desired encoding frame complexity. The actual memory level of the buffer memory 122 (supplied via a feedback signal 146) can also be used as an input to the throttling control unit 140.

[0056] After each compressed frame is generated by the video compression unit 126, the compression parameters are passed as feedback inputs via signal 144 to the throttling control unit 140, which then generates a new complexity level, video compression settings, etc., for a subsequent frame in response to the feedback inputs.

[0057] In some embodiments, the new compression settings may be used for the next subsequent frame. In other embodiments, there may be some lag or delay before the video compression settings are provided to the video compression unit 126. Alternatively or additionally, the new video compression settings may not be updated in some embodiments unless the buffer memory exceeds a threshold level. For example, if the processing time of a frame exceeds the frame interval, the throttling control unit 140 may not update the video compression settings in the video compression unit 126 if the buffer memory 122 is nearly empty, but rather may wait to see if increase in frame processing time is transient, or if instead the buffer 122 starts to fill up, before applying the new compression settings.

[0058] In some embodiments, the compression settings may not be updated after every frame. For example, in some embodiments, the compression settings may be updated only before or after an l-frame, or before or after a P-frame has been processed. In some embodiments, the compression settings may not be updated immediately before and/or after a B-frame has been processed. Similarly, in some embodiments, the compression settings may be updated only after a given number of consecutive frames have been processed and/or only after a predetermined time interval has passed.

[0059] Many video compression settings can significantly affect the per-frame processing time of a video compression unit. In some embodiments, the selection of a video compression setting to change may depend on the amount of change desired in order to obtain a desired frame compression time and/or may depend on the overall effect that the compression setting has on the quality of the compressed video. For example, if only a small change in processing time is desired, the throttling control unit 140 may change a video compression setting that causes only a small change in processor utilization while also having a small impact on video quality. Conversely, if a large change in processing time is desired (for example, if the complexity level needs to be significantly decreased because the buffer 122 is about to run out of memory, etc.), the video compression unit 126 and/or the throttling control unit 140 may change a video compression setting that has a strong effect on processor utilization, even if the selected video compression setting also has a strong effect on the quality of the compressed video.

[0060] In some embodiments, more than one video compression setting may be adjusted in response to a new complexity level specified by the throttling control unit 140.

[0061] An input that specifies the intended frame type (e.g., I-, B-, or P-frame) or even intended future frame types can optionally be provided to the throttling control unit 140. The frame type(s) may indicate, on a relative basis, how much processing time a frame or future set of frames will take. An input that defines the relative complexity of the frame is also optional. The value of this input may depend on the compression format. A higher complexity value indicates that, for a given compression format, the subject frame will require more processing time to compress.

[0062] The utilization level of one or more processors performing compression in the video compression unit 126 can also enter the feedback computation. Thus, the throttling control unit 140 can make adjustments based on many elements affecting the performance of the overall encoding system.

[0063] The compression settings that are adjusted by the video compression unit 126 and/or the throttling control module 140 can include, for example, variable-length coding settings, and/or motion estimation settings, such as motion vector encoding settings and other motion-related settings.

[0064] In general, when a video frame is processed, the frame is typically divided into a plurality of 8 pixel x 8 pixel blocks, although other size blocks are possible. Each of the blocks is transformed via a transform, such as a discrete cosine transform (DCT), into 64 frequency domain coefficients which represent, in order, low frequency to high frequency components of the block. As the human eye is more sensitive to low-frequency components than high-frequency components of a spatial image signal, the high-frequency components can be encoded using fewer bits than the low-frequency components. Thus, the more perceptually significant low frequency components can be encoded with more precision. However, some blocks contain important high frequency information.

[0065] Once determined, the transform coefficients are compressed using run-length encoding (RLE), in which consecutive numbers of same- valued DCT coefficients are grouped into "runs" and replaced with a few symbols indicating the number and type of coefficients in the run. The RLE encoded coefficients can then be encoded using a variable length coding (VLC) technique, such as Huffman coding, in which commonly encountered groups of symbols are replaced with corresponding entries from a lookup table, resulting in some savings in the number of bits used to encode the symbol group. Another technique called arithmetic coding can provide a more efficient way to encode a sequence of symbols. However, arithmetic coding is more computationally demanding than Huffman coding, and is therefore used less often. However in some embodiments, an arithmetic coding technique may be used instead of Huffman coding when it is desired to increase the complexity of the compression process according to some embodiments.

[0066] As noted above, motion estimation can consume a significant amount processing power in a video compression unit. Thus, adjusting the motion estimation settings may have a proportionately larger effect on the per- frame processing time of the video compression unit 126 compared to adjusting other compression settings. Likewise, adjusting the motion estimation settings may also have a proportionately larger effect on the quality of compressed frames compared to adjusting other compression settings.

[0067] In motion estimation, the video compression unit 126 attempts to find a region in a previously or future encoded frame (called a "reference frame") that closely matches each block in the frame currently being encoded. For purposes of motion estimation, a frame may be logically divided up into groups of "macro blocks," which may be divided into smaller sized blocks, potentially of different sizes. For each block in the current frame, the motion estimation algorithm attempts to determine one or more "motion vectors" that define horizontal and vertical offsets from the location of the block in the current frame to the location in the reference frame. The corresponding region in the reference frame is used to predict the pixels in the block of the current frame. The difference between the block and the selected region in the reference frame is determined and encoded.

[0068] The size of the horizontal and/or the vertical components of a motion vector may be restricted, so that the area in the reference frame that is analyzed is reduced, thereby reducing the complexity of the motion vector analysis. The area in the reference frame that is analyzed to determine the motion vector, if any, is referred to as the "search area." Restricting the search area may reduce the number of bits used to encode the motion vector and, more importantly, may reduce the amount of processing power that is required to find the best matching region in the reference frame. However, restricting the search area also reduces the chance that a suitable matching region can be found at all.

[0069] One way to perform motion estimation is to evaluate every possible 16-by-16 region in the search area, and select the best match from among all candidate regions. The best match may be determined by performing a sum of absolute differences (SAD) or sum of squared differences (SSD) computation between each 16-pixel by 16-pixel region and the macro block. However, analyzing every 16-pixel by 16-pixel region in a search area can require a relatively large amount of processing power. Instead of evaluating every 16-pixel by 16-pixel region in a search area, a motion estimation algorithm may select a limited number of candidate motion vectors and evaluate only the 16-pixel by 16-pixel regions corresponding to these candidate vectors.

[0070] One approach is to select the candidate motion vectors in several stages. For example, a number of initial candidate vectors may be selected and evaluated. The results may be used to eliminate some portions of the search area and/or to identify the more promising portions of the search area. A number of new candidate vectors may be selected and the process may be repeated until a final motion vector is selected. Using a larger number of candidate motion vectors allows the encoder to find a 16-pixel by 16-pixel region in the reference frame that better matches each macro block, thus reducing the prediction error.

[0071] Another approach is to calculate a motion vector for luminance (brightness) values of a block and then use the same motion vector for chrominance values of the block. However, for some types of images, such an approach may not yield the best motion vectors for the chrominance values.

[0072] In another approach, the motion vectors selected for other macro blocks in the current and/or previous frames in the video sequence can be analyzed and used to predict the motion in the current macro block. A subset of candidate motion vectors can be selected and evaluated based on this analysis to determine the best motion vector. According to some embodiments, compression complexity can be changed by setting the video compression unit 126 to use, or not to use, other motion vectors in current and/or previous frames to identify candidate motion vectors for a current frame.

[0073] Thus, there is a clear tradeoff in the selection of candidate motion vectors between compression complexity and image quality and/or compression efficiency. According to some embodiments, compression complexity can be modulated by increasing or decreasing the number of candidate motion vectors, and/or by dynamically choosing from among different techniques for identifying candidate motion vectors.

[0074] In addition, some video compression standards allow motion vectors to have non-integer values. That is, the encoder may estimate that the motion between the reference frame and current frame for a given macro block is not an integer number of pixels. Thus, to predict the pixels in a current macro block, the corresponding region in the reference frame may be interpolated to estimate the pixel values at non-integer pixel locations. The difference between this prediction and the actual pixel values is computed and encoded as described above. Accordingly, in some embodiments, video compression complexity may be reduced by restricting motion vectors to integer values and/or to particular non-integer values, such as ^ΛA pixel values.

[0075] Accordingly, a feedback loop including a video compression unit 126 and a throttling control unit 140 as shown in FIG. 4 can facilitate adjustment of video compression settings based on one or more compression parameters of previous frames. For content that is relatively easy to encode/compress, the compression complexity can be increased to utilize more computational resources. Likewise, for content that is difficult to encode/compress, the compression complexity can be lowered to reduce the per-frame compression time.

[0076] The utilization of the optional buffer memory 122 can be used as a primary or secondary input to adjust compression settings. If the available memory in the buffer 122 is low (i.e., the buffer 122 is full or almost full), the compression complexity can be lowered to speed up the compression and empty the memory more quickly. The more memory that is available in the buffer 122, the more the compression settings can be increased (i.e., the complexity of the compression algorithm can be increased) until a target buffer utilization level is reached.

[0077] Moreover, as discussed above, the level of memory usage in the buffer 122 can be used as a factor in determining if and when compression complexity should be reduced in response to an increase in the frame processing time. For example, even if the time required to compress a frame increases, the throttling control unit 140 may not adjust the compression complexity of the video compression unit 126 unless more than a threshold amount of the buffer memory 122 is used.

[0078] A feedback loop according to some embodiments can be expressed mathematically as follows:

C[n] = f(C[n-1], targetT, compressionT[n-1]) (1)

[0079] where the variable C represents, in the aggregate, the complexity of the compression algorithm employed by the video compression unit 126. C[n] is the complexity setting for the next frame to be compressed while C[n-1] is the complexity setting used for the previous frame. The variable targetT is the target compression time per frame. For real-time compression, targetT may be equal to the frame interval (i.e. the reciprocal of the video input capture rate in frames per second). In some embodiments, targetT may be equal to a value that is less than the frame interval, to provide for some overshoot error in selecting compression settings. The variable compressionT is the actual compression time for the previous (n-1th) frame. Accordingly, Equation (1) indicates that the compression settings for the current frame (frame n) are selected as a function of the complexity settings of a previous frame (frame n-1), the target compression time (targetT), and the actual compression time of the previous frame (compressionT).

[0080] The actual meaning of C, i.e., the settings it represents and the parameters affecting the complexity of those settings, may depend on the compression algorithm and what settings and tools are available. Increasing values of C represent higher complexity settings (e.g., the use of a more complex transform coefficient coding algorithm, the use of arithmetic coding instead of Huffman coding, the use of a larger search area for motion compensation, the selection of algorithms for determining motion vectors, etc.), and as a result, more burdensome and/or time-consuming computational requirements.

[0081] A more specific feedback loop can be expressed as follows:

C[n] = x * C[n-1] + (1-x) C[n-1] * (targetT /compressionT) (2)

[0082] The parameter X is a constant between 0 and 1 , and may be considered a damping factor for the feedback loop, so that compression settings may react more slowly than otherwise to changes in the frame compression time.

[0083] The feedback loop of Equation (2) illustrates how the complexity settings may be adjusted over time based on the compression results for previous frames according to some embodiments. When the compression time for the previous frame is lower than the target compression time, the compression complexity is increased for the next frame.

[0084] The same approach may be used to modulate compression complexity based on buffer memory level, compression frame type and/or frame complexity. Accordingly, another exemplary feedback loop can be expressed as follows:

C[n] = x ^* C[n-1] + (1-x) C[n-1] * (targetT / compressionT) * (targetM / actualM) (3)

where the variable targetM is the target memory level for the memory buffer while actualM represents the actual memory level for the buffer. If no memory is used, these values are both assumed to equal 1. Accordingly, when the buffer memory used after processing the previous frame is lower than the target buffer memory level, the compression complexity can be increased for the next frame. Conversely, when the buffer memory used after processing the previous frame is higher than the target buffer memory level, the compression complexity can be decreased.

[0085] FIG. 5 is a flowchart illustrating throttling control systems/methods according to some embodiments. As shown therein, a throttling control unit 140 receives compression parameters relative to the compression of a video frame by a video compression unit 126 (Block 192). The compression parameters can include, for example, actual and target frame compression times, buffer utilization, processor utilization, frame type and frame complexity, among others.

[0086] In response to the compression parameters, the throttling control unit updates one or more target settings for compression of future frames (Block 194). For example, the target settings can include a complexity level, a processor utilization level, a target compression time, and/or an explicit compression setting, such as a motion estimation setting.

[0087] The updated target setting is then supplied to the video compression unit 126 (Block 196), which uses the target setting to compress a subsequent video frame.

[0088] FIG. 6 is a flowchart illustrating systems/methods according to some embodiments. Referring to FIG. 6, operations 200 according to some embodiments include receiving a frame of a video signal (Block 202). In some embodiments the video signal may comprise a raw (i.e. unencoded) video signal, such as a video signal output by a video source. However, in some embodiments, the video signal may include an encoded video signal. Accordingly, some embodiments may relate to transcoding of video signals. "Transcoding" refers to the process of converting an encoded video signal from one format to another format and/or otherwise re-coding a coded video signal using different encoding settings that were used to encode the original encoded signal.

[0089] The received frame is processed (i.e., compressed) according to a set of video compression settings stored in the video compression unit 126 (Block 204). As explained above, the video compression settings may define a video compression algorithm that is more or less complex (i.e., that requires more or less computing power to perform). The processed frame is output as part of an encoded video signal.

[0090] After processing the frame, a determination is made whether the processed frame is the last frame of the video signal to be processed (Block 206). If so, the operations terminate. Otherwise, the time required to process the previous frame (the "compression time") is determined (Block 208). The compression settings are then adjusted in response to the compression time of one or more of the previous frames (Block 210). As explained above, if the compression time of the previous frame was longer than desired, the compression settings may be adjusted to reduce the complexity of the compression algorithm and thereby potentially reduce the amount of time required to compress a subsequent frame. If the compression time of the previous frame was shorter than desired, the compression settings may be adjusted to increase the complexity of the compression algorithm and thereby potentially improve the quality of a subsequent compressed frame.

[0091] In some embodiments, compression settings may be changed at any time during a compression process. In other embodiments, compression settings can be changed only when a new l-frame is generated. Furthermore, in some embodiments, a new l-frame may be generated whenever compression settings are changed.

[0092] Furthermore, in some embodiments, compression settings may not be changed if a subsequent frame is expected to require fewer computational resources and/or if the processor utilization of the video compression unit 140 is expected to be low.

[0093] The next frame in the input signal is then obtained (Block 212), and the new frame is processed as described above.

[0094] Operations according to further embodiments are illustrated in FIG. 7. In particular, the operations illustrated in FIG. 7 may be performed in a video compression system including an input buffer memory.

[0095] Referring to FIG. 7, operations 300 according to some embodiments include receiving a frame of a video signal (Block 302). The received frame is processed (i.e. compressed) according to a set of video compression settings (Block 304). After processing the frame, a determination is made whether the processed frame is the last frame of the video signal (Block 306). If so, the operations terminate. Otherwise, the memory utilization of the buffer memory is determined (Block 308) and the compression time is determined (Block 310). The compression settings are then adjusted in response to the compression time of the previous frame and/or in response to the level of utilization of the buffer memory (Block 312). The next frame in the input signal is then obtained (Block 314), and the new frame is processed as described above.

[0096] In some embodiments, the compression settings can be adjusted in response only to the amount of memory available in the buffer.

[0097] Referring to FIG. 8, operations 400 according to some embodiments include receiving a frame of a video signal (Block 402). The received frame is processed according to a set of video compression settings (Block 404). After processing the frame, a determination is made whether the processed frame is the last frame of the video signal (Block 406). If so, the operations terminate. Otherwise, the compression time for the frame is compared to a threshold, such as a target compression time/range (Block 408). That is, the threshold could be defined as a range between a first threshold and a second threshold.

[0098] If the actual compression time is determined to be greater than the target compression time/range, the actual buffer utilization after processing the frame is then compared to a buffer utilization threshold (Block 412). If the buffer usage is greater than the threshold utilization level, the compression settings may be adjusted (Block 414) to decrease the complexity of the compression algorithm. Moreover, if the compression time was determined in Block 408 to be less than the threshold compression time/range, the compression settings may be adjusted in Block 414, for example, to increase the complexity of the compression algorithm. If the actual compression time is within a target compression range, no changes may be made to the compression complexity. The compression settings may be adjusted in response to the compression time of the previous frame and in response to the level of utilization of the buffer memory. The next frame in the input signal is then obtained (Block 416), and the new frame is processed as described above.

[0099] As will be appreciated by one of skill in the art, the present invention may be embodied as methods, data processing systems, and/or computer program products. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects all generally referred to herein as a "circuit" or "module." Furthermore, the present invention may take the form of a computer program product on a tangible computer usable storage medium having computer program code embodied in the medium that can be executed by a computer. Any suitable tangible computer readable medium may be utilized including hard disks, CD ROMs, optical storage devices, or magnetic storage devices.

[00100] Some embodiments of the present invention are described above with reference to flowchart illustrations and/or block diagrams of methods, systems and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[00101] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.

[00102] The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[00103] It is to be understood that the functions/acts noted in the blocks may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[00104] In the drawings and specification, there have been disclosed typical embodiments of the invention and, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation, the scope of the invention being set forth in the following claims.

Claims

THAT WHICH IS CLAIMED IS:

1. A method of providing feedback control for a video compression unit, comprising: receiving an actual compression time used in the compression of at least one frame of a video signal by the video compression unit at a compression complexity value; adjusting the compression complexity value in response to the actual compression time; and supplying the adjusted compression complexity value to the video compression unit for use in compressing at least one other frame of the video signal.

2. The method of Claim 1 , further comprising: comparing the actual compression time to a target compression time, wherein the compression complexity value is adjusted in response to the comparison of the actual compression time to the target compression time.

3. The method of Claim 2, further comprising increasing the compression complexity value when the actual compression time is less than the target compression time, and decreasing the compression complexity value when the actual compression time is greater than the target compression time.

4. The method of Claim 3, further comprising comparing a buffer utilization level of a buffer that stores video frames for processing by the video compression unit to a threshold buffer utilization level; increasing the compression complexity value when the buffer utilization level is lower than the threshold buffer utilization level; and decreasing the compression complexity value when the buffer utilization level is greater than the threshold buffer utilization level.

5. The method of Claim 3, further comprising determining a buffer utilization level of a buffer that stores video frames for processing by the video compression unit and, when the actual compression time is greater than the target compression time, increasing the compression complexity value only when the buffer utilization level is greater than a threshold buffer utilization level.

6. The method of Claim 1, wherein the resource comprises a buffer utilization level of a buffer that stores video frames for processing by the video compression unit and wherein the compression parameter comprises a compression complexity, the method further comprising increasing the compression complexity value when the buffer utilization is less than a first threshold level and decreasing the compression complexity value when the buffer utilization level is greater than a second threshold level.

7. A method of compressing a video signal comprising a plurality of video frames, comprising: compressing at least one video frame in accordance with video compression settings; following compression, modifying the video compression settings based on an actual compression time of the at least one video frame and at least one target setting; and compressing at least one other video frame in accordance with the modified video compression settings.

8. The method of Claim 7, further comprising repeating steps of compressing frames according to the video compression settings and modifying the video compression settings in response to actual compression times throughout compression of a portion of the video signal.

9. The method of Claim 7, wherein the at least one target setting comprises a target compression time per frame.

10. The method of Claim 7, wherein the settings are altered based also on a frame type and/or a desired frame complexity.

11. The method of Claim 7, wherein the target settings include a target memory level of a buffer memory that is configured to store the video frames prior to compression.

12. The method of Claim 7, further comprising determining a processor utilization of a processor of the video compression unit, and altering the video compression settings in response to the determined processor utilization.

13. A video compression system comprising: a throttling control unit configured to receive a feedback signal indicative of an amount of a resource used in the compression of at least one frame of a video signal carried out using a compression parameter, to adjust the compression parameter in response to the feedback signal, and to supply the adjusted compression parameter as an output signal.

14. The system of Claim 13, wherein the throttling control unit response to the feedback signal indicating an actual compression time of the at least one frame, a buffer utilization level and/or a processor utilization level.

15. The system of Claim 13, wherein the throttling control unit uses the output signal to adjust a compression complexity value.

16. The system of Claim 13, wherein the throttling control unit uses the output signal to adjust a target processor utilization and/or a target compression time per frame.

17. The system of Claim 13, further comprising: a video compression unit configured to receive the adjusted compression parameter from the throttling control unit and to compress the video signal in accordance with the adjusted compression parameter, wherein the video compression unit is configured to provide the feedback signal to the throttling control unit.

18. The system of Claim 17, further comprising: a buffer memory configured to store frames of the video signal for processing by the video compression unit, wherein the buffer memory is configured to provide a buffer utilization level to the throttling control unit; wherein the throttling control unit is configured to provide the output signal in response to the buffer utilization level.

19. A method of providing feedback control for a video compression unit, comprising: receiving a feedback signal indicative of an amount of a resource used in the compression of at least one frame of a video signal by the video compression unit using a compression parameter; adjusting the compression parameter in response to the feedback signal; and supplying the adjusted compression parameter to the video compression unit.

20. The method of Claim 19, further comprises adjusting the compression parameter in response to the feedback signal indicating an actual compression time of a frame by the video compression unit, a buffer utilization level of a buffer that stores video frames for processing by the video compression unit and/or a processor utilization of a processor in the video compression unit.

21. The method of Claim 19, wherein the compression parameter comprises a compression complexity.

22. The method of Claim 19, wherein the compression parameter comprises a target processor utilization and/or a target compression time per frame.

23. The method of Claim 19, further comprising determining a frame type for a subsequent frame, wherein the compression parameter is adjusted in response to the frame type.

24. The method of Claim 19, wherein the resource comprises a buffer utilization level of a buffer that stores video frames for processing by the video compression unit and wherein the compression parameter comprises a compression complexity, the method further comprising increasing the compression complexity when the buffer utilization is less than a first threshold level and decreasing the compression complexity when the buffer utilization level is greater than a second threshold level.