WO2002071639A1 - Systems and methods for error resilient encoding - Google Patents

Systems and methods for error resilient encoding Download PDF

Info

Publication number
WO2002071639A1
WO2002071639A1 PCT/US2002/006726 US0206726W WO02071639A1 WO 2002071639 A1 WO2002071639 A1 WO 2002071639A1 US 0206726 W US0206726 W US 0206726W WO 02071639 A1 WO02071639 A1 WO 02071639A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame
value
frames
macroblock
scene
Prior art date
Application number
PCT/US2002/006726
Other languages
French (fr)
Other versions
WO2002071639A8 (en
Inventor
Ioannis Katsavounidis
Chung-Chieh Kuo
Original Assignee
Intervideo, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Intervideo, Inc. filed Critical Intervideo, Inc.
Priority to JP2002570429A priority Critical patent/JP2004532540A/en
Priority to EP02721263A priority patent/EP1374430A4/en
Publication of WO2002071639A1 publication Critical patent/WO2002071639A1/en
Publication of WO2002071639A8 publication Critical patent/WO2002071639A8/en

Links

Classifications

    • H—ELECTRICITY
    • H03—ELECTRONIC CIRCUITRY
    • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
    • H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
    • H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
    • H—ELECTRICITY
    • H03—ELECTRONIC CIRCUITRY
    • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
    • H03M13/00—Coding, decoding or code conversion, for error detection or error correction; Coding theory basic assumptions; Coding bounds; Error probability evaluation methods; Channel models; Simulation or testing of codes
    • H—ELECTRICITY
    • H03—ELECTRONIC CIRCUITRY
    • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
    • H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
    • H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
    • H03M7/40—Conversion to or from variable length codes, e.g. Shannon-Fano code, Huffman code, Morse code
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/573—Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/65—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using error resilience
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234318—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into objects, e.g. MPEG-4 objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/147—Scene change detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/20—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
    • H04N19/29—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding involving scalability at the object level, e.g. video object layer [VOL]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W84/00—Network topologies
    • H04W84/02—Hierarchically pre-organised networks, e.g. paging networks, cellular networks, WLAN [Wireless Local Area Network] or WLL [Wireless Local Loop]
    • H04W84/10—Small scale networks; Flat hierarchical networks
    • H04W84/14—WLL [Wireless Local Loop]; RLL [Radio Local Loop]

Definitions

  • the present invention is related to video and image coding and in particular to systems and methods for coding video image information in a compressed and error resilient manner. Description of the Related Art
  • MPEG is an ISO/IEC standard developed by MPEG (Moving Picture Experts Group). There are several versions of the MPEG standard, such as MPEG-1, MPEG-2, MPEG-4, and MPEG-7, and they are intended to standardize certain aspects of image and audio compression. As with other forms of video compression, such as H.261, H.262, H.263, H.263+, H.263++, H.26L, MPEG compression attempts to eliminate redundant or irrelevant data. For example, an MPEG encoder uses information from selected frames to reduce the overall video data that needs to be transmitted for certain other frames.
  • a video frame can be encoded in one of three ways, as an intraframe, as a predicted frame, and as a bi-directional frame.
  • a video frame can also be skipped in order to reduce the resulting file size or bit-rate.
  • An intraframe typically contains the complete image data for that frame and so does not rely on image data from other frames.
  • Intraframe encoding provides the least compression.
  • a predicted frame generally contains just enough information to allow a decoder to display the frame based on a recent preceding intraframe or predicted frame. This means that the predicted frame contains the data that relates to how the image has changed from the previous frame and residual error correction data.
  • a bi-directional frame is generated from information from the surrounding intraframe(s) and/or predicted frames, including residual error correction data. Using data from the surrounding frames, the decoder uses interpolation to calculate the position and color of each pixel.
  • MPEG-4 The MPEG-4 standard was developed for use with both low and high bit rate applications. For example, MPEG-4 has been enhanced for use in interactive video games, videoconferencing, videophones, interactive storage media, multimedia mailing, wireless multimedia and broadcasting applications. MPEG-4 provides for object scalability, improved error robustness and enhanced compression.
  • MPEG-4 has enhanced error resiliency as compared to previous versions of MPEG so that video data can be more successfully transmitted over such error prone networks.
  • one error resiliency technique provided for by the MPEG-4 standard is the use of resync markers in the video bit-stream.
  • MPEG-4 has adopted fixed interval synchronization and specifies that video object plane (VOP) start codes and resynchronization markers (i.e., the start of a video packet) appear only at legal fixed interval locations in the bitstream. This helps to avoid the problems associated with start codes emulations.
  • VOP video object plane
  • resynchronization markers i.e., the start of a video packet
  • Another error resiliency technique provided for by the MPEG-4 standard is the use of a reversible variable-length code. This code can be decoded even when read backwards, enabling a decoder to use uncorrupted information from a newly found resync marker back to the point in the data where the error occurred.
  • Still another error resiliency technique adopted by MPEG-4 is data partitioning, used to separate motion information from texture information using a second resynchronization marker inserted between motion and texture information.
  • the decoder can utilize the motion information to conceal the error by using the motion information to compensate the previous decoded frame or VOP.
  • the present invention is related to video encoding and in particular to systems and methods for encoding video information for transmission in a compressed manner and/or an error resilient manner.
  • Embodiments of the present invention advantageously enable the transmission of video information even in low-bit rate, high noise environments.
  • embodiments of the present invention enable video transmission to be successfully performed over cellular networks and the like.
  • enhanced error resilience is provided by automatically inserting two consecutive intracoded I-frames upon a scene change. By including two intracoded frames upon a scene change, error propagation is reduced on the decoder-side if one of the two sequential intracoded frames is lost or undecodable.
  • a method of enhancing error resiliency in an encoded video transmission comprising: receiving an indication that a first frame corresponds to a scene change; intracoding the first frame at least partly in response to receiving the indication that the first frame corresponds to a scene change; automatically intracoding the next consecutive frame; and transmitting the intracoded first frame and the intracoded next consecutive frame.
  • a video coding apparatus comprising: a first instruction stored in a processor readable memory, the first instruction configured to receive an indication that a first frame is to be intracoded; a second instruction stored in processor readable memory, the second instruction configured to intracode the first frame at least partly in response to the indication; and a third instruction stored in processor readable memory, the third instruction configured to automatically intracode the frame immediately after the first frame as a result of the indication that the first frame is to be intracoded.
  • a method of encoding video frames comprising: receiving a first instruction to intracode a first frame; intracoding the first frame in response to the first instruction; and intracoding a second frame as a result of the first instruction to intracode the first frame.
  • an encoding apparatus comprising: a means for receiving a first indication to intracode a first frame; a means for intracoding the first frame in response to the first instruction; and a means for intracoding a second frame as a result of the first instruction to intracode the first frame.
  • An enhanced Adaptive Intra Refresh (AIR) process provides efficient error resiliency by selectively and adaptively encoding macroblocks in a predicted frame.
  • an adaptive motion area process is performed to determine which macroblocks are to be intracoded.
  • Adaptive motion change detection can efficiently reduce the large propagation error, including errors occurring in the motion area.
  • An intercode distortion value and an intracode distortion value are calculated, as are an intercode bit quantity and an intracode quantity. Based on a comparison of the calculated intercode distortion value and the intracode distortion value, and on a comparison of the intercode bit quantity and the intracode bit quantity for each macroblock, a decision is made as to which predicted frame macroblocks are to be intracoded.
  • One embodiment of the present invention is a method of performing adaptive intra refresh, the method comprising: receiving a packet loss probability value; receiving a motion vector for a first macroblock in a first frame; mapping the motion vector onto portions of a plurality of other macroblocks in a previous frame; calculating at least a first transition factor value based at least in part on the mapping; calculating an estimated inter distortion value for the first macroblock based on at least the packet loss probability value and the first transition factor value; calculating an estimated intra distortion value for the first macroblock based on at least the packet loss probability value; receiving a first bit quantity value corresponding to a bit quantity used to intracode the first macroblock; receiving a second bit quantity value corresponding to a bit quantity used to intercode the first macroblock; and based at least in part on the estimated inter distortion value, the estimated intra distortion value, the first bit quantity value, and the second bit quantity value, transmitting the first macroblock as one of an intra macroblock and an inter macroblock.
  • Another embodiment of the present invention is a method of selectively intracoding macroblocks, the method comprising: receiving a packet loss probability value; receiving a motion vector for a first macroblock in a first frame; based at least in part on the motion vector, determining which portions of macroblocks in a previous frame would be used in predicting the first macroblock; calculating at least a first propagation strength value based at least in part on determining which portions of macroblocks in the previous frame would be used in predicting the first macroblock; calculating an estimated inter distortion value for the first macroblock based on at least the packet loss probability value and the first propagation strength value; calculating an estimated intra distortion value for the first macroblock based on at least the packet loss probability value; calculating a quantization distortion value for the first macroblock; receiving a first bit quantity value corresponding to a bit quantity used to intracode the first macroblock; receiving a second bit quantity value corresponding to a bit quantity used to intercode the first macroblock; and based at least in part on the estimated inter distortion value
  • Still another embodiment of the present invention is a method of selectively intracoding macroblocks in a plurality of macroblocks in a first frame, the method comprising: receiving a packet loss probability value; receiving a corresponding motion vector for each macroblock in the plurality of macroblocks; based at least in part on the corresponding motion vector, determining which portions of macroblocks in a previous frame would be used in predicting said each macroblock; calculating for each of said macroblocks at least a first corresponding propagation strength value based at least in part on determining which portions of macroblocks in the previous frame would be used in predicting said each macroblock; calculating for each of said macroblocks an estimated inter distortion value based upon at least the packet loss probability value and the corresponding at least first propagation strength value; calculating for each of said macroblocks an estimated intra distortion value based upon at least the packet loss probability value; calculating for each of said macroblocks a quantization distortion value; and based at least in part on the estimated inter distortion, the estimated intra distortion, and the
  • One embodiment of the present invention is a circuit configured to selectively intracode macroblocks, the circuit comprising: a first instruction configured to receive a packet loss probability value; a second instruction configured to receive a motion vector for a first macroblock in a first frame; a third instruction configured to determine, based at least in part on the motion vector, which portions of macroblocks in a previous frame would be used in predicting the first macroblock; a fourth instruction configured to calculate at least a first propagation strength value based at least in part on the determination of which portions of macroblocks in the previous frame would be used in predicting the first macroblock; a fifth instruction configured to calculate an estimated inter distortion value for the first macroblock based on at least the packet loss probability value and the first propagation strength value; a sixth instruction configured to calculate an estimated intra distortion value for the first macroblock based on at least the packet loss probability value; and a seventh instruction configured to selectively provide for transmission the first macroblock as one of an intra macroblock and an inter macroblock based at least in part on the estimated inter
  • Embodiments of the present invention provide for adaptive skipping of frames in an encoding process based on a defined cost function.
  • the visual quality particularly in scene change areas, is improved while scenes are efficiently encoded.
  • One embodiment of the skipping process is iterative in that a frame is selectively dropped from the sequence of frames by computing a mean of absolute differences between the frames adjacent to the frame of interest, and by weighting the computation with a temporal parameter, wherein the frame having the least impact on the scene is dropped. This procedure is repeatedly iterated until a target frame rate, which is related to the desired bit rate and frame size, is achieved.
  • Other measurement techniques such as a sum of the mean absolute differences (SMAD) or sums of root mean squared (SRMS) values, can be used in place of or in addition to the mean of absolute differences technique.
  • SAD mean absolute differences
  • SRMS root mean squared
  • One embodiment of the present invention is a method of selecting which video sequence frames to skip during an encoding process, the method comprising: receiving a sequence of frames; for each frame in at least a portion of the sequence of frames, determining a corresponding mean of absolute differences value weighted by a temporal parameter between frames bracketing said each frame; and at least partly based on the corresponding mean of absolute differences value weighted by a temporal parameter calculated for each frame, skipping a first frame in the sequence of frames.
  • Another embodiment of the present invention is a method of determining which frames to skip in a video sequence, the method comprising: receiving a first frame rate goal; selecting a first potential frame to be skipped, the first potential frame between a first frame and a last frame in a video sequence scene; calculating a first cost value for the first potential frame to be skipped, the first cost value related to the visual impact that would result from skipping the first potential frame; storing the first cost value in computer readable memory; selecting a second potential frame to be skipped, the second potential frame between the first frame and the last frame; calculating a second cost value for the second potential frame to be skipped; storing the second cost value in computer readable memory; selecting a third potential frame to be skipped, the third potential frame between the first frame and the last frame; calculating a third cost value for the third potential frame to be skipped; storing the third cost value in computer readable memory; determining which one of the first, second, and third cost values has the least cost; skipping the one of the one of the one of the
  • Still another embodiment of the present invention is a method of selecting image frames to be skipped during an encoding process, the method comprising: receiving a sequence of frames including a first frame; calculating a first cost function value for the first frame, wherein the cost function includes a first temporal parameter related to a time difference and luminosity differences between frames bounding the first frame; calculating a second cost function value for a second frame, wherein the cost function includes a second temporal parameter related to a time difference between frames and luminosity differences bounding the second frame; and selecting for skipping one of the first frame and the second frame based at least in part on the first cost function value and the second cost function value.
  • Yet another embodiment of the present invention is a processor circuit configured to select video frames to skip in a sequence of frames, the processor circuit comprising: a first instruction configured to calculate for each frame in at least a portion of the sequence of frames a corresponding mean of absolute differences value weighted by a temporal parameter between frames bracketing said each frame in the portion of the sequence of frames; and a second instruction configured to designate a first frame to be skipped based at least in part on the corresponding mean of absolute differences value weighted by the temporal parameter.
  • One embodiment of the present invention is a frame rate control apparatus, comprising: a first instruction configured to receive a frame rate target; a second instruction configured to calculate a first cost function value for a first frame in a sequence of frames, wherein the first cost function value is based at least in part on a first temporal parameter related to a time difference and luminosity differences between frames in the sequence of frames bounding the first frame; a third instruction configured to calculate a second cost function value for a second frame in the sequence of frames, wherein the second cost function based at least in part on a second temporal parameter related to a time difference and luminosity differences between frames in the sequence of frames bounding the second frame; a fourth instruction configured to select for skipping one of the first frame and the second frame based at least in part on the first cost function value and the second cost function value; and a fifth instruction configured to determine if the target frame rate is met with the selected one of the first frame and the second frame skipped.
  • FEC forward error correction
  • FEC coding is efficiently and selectively applied in real-time to important data, such as motion vectors, DC coefficients and header information, rather then generating FEC bits for unimportant or less important data.
  • This selected important data may be located in a packet between a packet resync field and a motion marker.
  • the selected packet bits targeted for FEC coding are concatenated together and the FEC code bits are generated for the concatenated bits.
  • the resulting FEC bits are placed in an additional packet after the regular frame or VOP packets to ensure MPEG compatibility.
  • One embodiment of the present invention is a method of providing forward error correction (FEC) on a plurality of frame packets, the method comprising: concatenating selected portions of packet data corresponding to a plurality of frame packets for a first frame; generating forward error correction bits for the concatenated selected portions of packet data; and transmitting the forward error correction bits in a separate packet identified with a user data identifier code or the like, including other unique identifier codes to be assigned in the future by MPEG-standards committee and the like.
  • FEC forward error correction
  • Another embodiment of the present invention is an error correction generation circuit, comprising: a first instruction stored in processor readable memory configured to generate forward error correction data for selected portions of packet data that are to be transmitted in a corresponding plurality of frame packets; a second instruction stored in processor readable memory configured to store the forward error correction data in a first packet separate from the plurality of frame packets; and a third instruction stored in processor readable memory configured to identify the first packet with a first data identifier code.
  • Still another embodiment of the present invention is an encoder circuit, comprising: a means for generating forward error correction data for selected portions of packet data from a plurality of frame packets; a means for storing the forward error correction data in a first packet separate from the plurality of frame packets; and a means for identifying the first packet with a first data identifier code.
  • embodiments of the present invention provide for using a Header Extension Code (HEC) in a sequence of video packets or in every video packet, and not just on the first video packet following the VOP header as with conventional encoders. This better ensures that even if a packet is lost or corrupted, subsequent packets can still be decoded and used. Further, even many conventional decoders will be able to handle the inclusion of the enhanced use of HECs.
  • HEC Header Extension Code
  • a Video-Object-Layer (VOL) header has a flag set indicating that a fixed Video Object Plane (VOP) increment is to be used, followed by the fixed time increment value. This will facilitate the decoder's detection of missing frames, that is, frames either skipped by the encoder in order to achieve higher compression or lost during transmission.
  • VOP Video Object Plane
  • Embodiments of the present invention provide bit allocation on a scene level, on a frame level, and/or on a macroblock level.
  • a bit allocation module determines an appropriate distribution of a fixed bit budget.
  • a weight is assigned to a scene based on the number of intracoded frames and the number of intercoded frames. Intracoded frames are weighted more heavily than intercoded frames to account for the greater number of bits needed to encode an intraframe.
  • the bit allocation module distributes the fixed bit budget within a scene by comparing the current bit usage and the target bit usage and based on the comparison, adjusts a quantization parameter or step size for the current frame.
  • an encoding quantization step or quantization parameter is dynamically adjusted. This provides for fast convergence to a desired bit budget, even for short scenes or Groups of Video Object Planes (GOVs).
  • the QP for I-frames is dynamically adjusted based on the actual bit usage of the previous scene or GOV. This QP adaptation of I-frame can be very useful for achieving or coming close to achieving the bit budget when there are high motion segments during a scene sequence.
  • One embodiment of the present invention is a bit allocation method for use with video sequences, the method comprising : receiving a clip bit budget for at least a first clip; determining a scene quantity for the first clip; determining a quantity of predicted frames in a first scene in the first clip; and calculating a bit budget for the first scene based at least in part on the scene quantity for the first clip, the first clip bit budget, a quantity of intracoded frames in the first scene, and the quantity of predicted frames in the first scene.
  • Another embodiment of the present invention is a bit allocation method for use with video clip scenes, the method comprising : calculating a bit budget for a first scene; calculating bit budgets for corresponding frames, including at least a first frame, within the first scene; and calculating bits budgets corresponding to macroblocks within the first frame.
  • Still another embodiment of the present invention is an apparatus for allocating bits, comprising: a first instruction configured to assign a first bit budget for a first scene based at least in part on how many intracoded frames are in the first scene, a complexity determination for the first scene, and how many intercoded frames are in the first scene; and a second instruction configured to allocate bit budgets to frames within the first scene based at least in part on a current bit usage and a target bit usage.
  • Yet another embodiment of the present invention is a bit budgeting apparatus, comprising : a first instruction configured to receive a bit budget for at least a first video sequence; a second instruction configured to determine a scene quantity for the first video sequence; a third instruction configured to determine a quantity of predicted frames in a first scene in the first video sequence; and a fourth instruction configured to determine a bit budget for the first scene based at least in part on the scene quantity for the first clip, the first clip bit budget, a quantity of intracoded frames in the first scene, and the quantity of predicted frames in the first scene.
  • Embodiments of the present invention provide accurate systems and methods for locating scene changes. Because scene change frames are intracoded, which generally utilizes more bits then intercoding while providing greater error resiliency, the accurate determination of scene change frames is important in ensuring that error resiliency is efficiently performed.
  • a first root mean squared (RMS) value is calculated for a first frame relative to a second frame.
  • a second temporal derivative RMS value is calculated for the first frame relative to the second frame, and the second frame relative to a third frame. Based at least in part on the second temporal derivative RMS value, the second frame is designated as a scene change frame.
  • Frames are adaptively grouped based on the scene change detection. As discussed below, other criteria, based on the mean of absolute differences (MAD), and/or locating RMS local maxima, and/or locating MAD local maxima, can be used to detect a scene change.
  • MAD mean of absolute differences
  • One embodiment of the present invention is a method of detecting a scene change in a digital video sequence having a plurality of frames, the method comprising: calculating a first root mean squared (RMS) value for a first frame relative to a second frame and the second frame relative to a third frame; calculating a first mean absolute difference (MAD) value for the first frame relative to the second frame; determining if the first RMS value meets a first criterion; determining if the first MAD value meets a second criterion; and designating the second frame as a scene change frame at least partly in response to determining that both the first RMS value meets the first criterion and the first MAD value meets the second criterion.
  • RMS root mean squared
  • MAD mean absolute difference
  • Another embodiment of the present invention is a method of detecting a scene change in a digital video sequence, the method comprising: calculating a second temporal derivative RMS value for a first frame relative to a second frame and the second frame relative to a third frame; and based at least in part on the second derivative value, determining that the second frame is a scene change frame.
  • Still embodiment of the present invention is an apparatus for identifying a scene change in a video sequence, the apparatus comprising: a first instruction stored in processor readable memory, the first instruction configured to calculate a first root mean squared (RMS) value for a first portion of a video sequence relative to a second portion of the video sequence; a second instruction stored in processor readable memory, the second instruction configured to calculate a second temporal derivative RMS value; and a third instruction configured to cause the second portion of the video sequence to be intracoded based at least in part on the second derivative RMS value.
  • RMS root mean squared
  • One embodiment of the present invention is method of determining which portions of a video sequence are to be intracoded, the method comprising: calculating a first root mean squared (RMS) value for a first portion of the video sequence; calculating a first mean absolute difference (MAD) value for the first portion of the video sequence; determining if the first RMS value meets a first criterion; determining if the first MAD value meets a second criterion; determining if the first MAD value meets a third criterion; and causing an intracoding operation to be performed at least partly in response to at least two of the first, second and third criteria being met.
  • RMS root mean squared
  • MAD mean absolute difference
  • FIG. 1 Another embodiment of the present invention is a scene change detection apparatus, comprising: an RMS circuit having a first frame information input and an RMS output, the RMS circuit configured to provide at the RMS output a value corresponding to root mean squared differences information between at least two frames based on frame information received on the first frame information input; a MAD circuit having a second frame information input and a MAD output, the MAD circuit configured to provide at the MAD output a value corresponding to mean absolute differences information between at least two frames based on frame information received on the second frame information input; and an evaluator circuit coupled to the RMS output and the MAD output, the evaluator circuit configured to detect a scene change frame based at least on the value corresponding to root mean squared differences information and the value corresponding to mean absolute differences information, and to provide a scene change designation.
  • Figure 1A illustrates an example networked system for implementing a video distribution system.
  • Figures 1B-C illustrate an example encoder architecture in accordance with an embodiment of the present invention.
  • Figures 2A-B illustrate an example refresh map and an example scan order that can be used with an embodiment of the present invention.
  • Figure 3 illustrates an example analysis of a video sequence used to locate a scene change.
  • Figures 4A-4B illustrate an example of adaptive frame skipping in accordance with an embodiment of the present invention.
  • Figure 5 illustrates an example use of second order motion compensation.
  • Figure 6 illustrates an example packetized bitstream.
  • Figure 7 illustrates an example use of consecutive I-frames in accordance with an embodiment of the present invention
  • Figures 8A-H illustrate example processes for adaptive intra refresh.
  • Figure 9 illustrates an example rate control process in accordance with an embodiment of the present invention.
  • Figure 10 illustrates an example scene level recursive bit allocation process.
  • Figure 11 illustrates an example graph of Forward Error Correction overhead vs. average BER correction capability.
  • the present invention is related to video encoding and in particular to systems and methods for encoding video information for transmission in a compressed and/or an error resilient manner.
  • embodiments of the present invention advantageously enable the transmission of video information even in low-bit rate, noise, error-prone environments.
  • Embodiments of the present invention can be used with a variety of video compression standards, such as, by way of example, the MPEG-4 standard, as well as MPEG-1, MPEG-2, H.261, H.262, H.263, H.263+, H.263++, and H.26L, and video standards yet to be developed.
  • FIG. 1A illustrates a networked system for implementing a video distribution system in accordance with one embodiment of the invention.
  • An encoding computer 102 receives a video signal, which is to be encoded to a relatively compact and robust format.
  • the encoding computer 102 can correspond to a variety of machine types, including general purpose computers that execute software and to specialized hardware.
  • the encoding computer 102 can receive a video sequence from a wide variety of sources, such as via a satellite receiver 104, a video camera 106, and a video conferencing terminal 108.
  • the video camera 106 can correspond to a variety of camera types, such as video camera recorders, Web cams, cameras built into wireless devices, and the like.
  • Video sequences can also be stored in a data store 110.
  • the data store 110 can be intemal to or external to the encoding computer 102.
  • the data store 110 can include devices such as tapes, hard disks, optical disks, and the like. It will be understood by one of ordinary skill in the art that a data store, such as the data store 110 illustrated in Figure 1A, can store unencoded video, encoded video, or both.
  • the encoding computer 102 retrieves unencoded video from a data store, such as the data store 110, encodes the unencoded video, and stores the encoded video to a data store, which can be the same data store or another data store.
  • a source for the video can include a source that was originally taken in a film format.
  • the encoding computer 102 distributes the encoded video to a receiving device, which decodes the encoded video.
  • the receiving device can correspond to a wide variety of devices that can display video.
  • the receiving devices shown in the illustrated networked system include a cell phone 112, a personal digital assistant (PDA) 114, a laptop computer 116, and a desktop computer 118.
  • the receiving devices can communicate with the encoding computer 102 through a communication network 120, which can correspond to a variety of communication networks including a wireless communication network. It will be understood by one of ordinary skill in the art that a receiving device, such as the cell phone 112, can also be used to transmit a video signal to the encoding computer 102.
  • the encoding computer 102 can correspond to a wide variety of computers.
  • the encoding computer 102 can be a microprocessor or processor (hereinafter referred to as processor) controlled device, including, but not limited to a terminal device, such as a personal computer, a workstation, a server, a client, a mini computer, a main-frame computer, a laptop computer, a network of individual computers, a mobile computer, a palm top computer, a hand held computer, a set top box for a TV, an interactive television, an interactive kiosk, a personal digital assistant, an interactive wireless communications device, a mobile browser, a Web enabled cell phone, a personal digital assistant (PDA) or a combination thereof.
  • PDA personal digital assistant
  • an encoder computer may also be included in the camera 106, the cell phone 112, the PDA 114, the laptop computer 116, and/or the desktop computer 118.
  • the computer 102 may further possess input devices such as a keyboard, a mouse, a trackball, a touch pad, or a touch screen and output devices such as a computer screen, printer, speaker, or other input devices now in existence or later developed.
  • the encoding computer 102 can correspond to a uniprocessor or multiprocessor machine.
  • the encoder and decoder computers can include an addressable storage medium or computer accessible medium, such as random access memory (RAM), an electronically erasable programmable read-only memory (EEPROM), masked read-only memory, one-time programmable memory, hard disks, floppy disks, laser disk players, digital video devices, Compact Disc ROMs, DVD-ROMs, other optical media, video tapes, audio tapes, magnetic recording tracks, electronic networks, and other techniques to transmit or store electronic content such as, by way of example, programs and data.
  • RAM random access memory
  • EEPROM electronically erasable programmable read-only memory
  • masked read-only memory such as hard disks, floppy disks, laser disk players, digital video devices, Compact Disc ROMs, DVD-ROMs, other optical media, video tapes, audio tapes, magnetic recording tracks, electronic networks, and other techniques to transmit or store electronic content such as, by way of example
  • the encoding and decoding computers are equipped with a network communication device such as a network interface card, a modem, Infra-Red (IR) port, a wireless network interface, or other network connection device suitable for connecting to a network.
  • the computers execute an appropriate operating system, such as Linux, Unix, Microsoft® Windows® 3.1, Microsoft® Windows® 95, Microsoft® Windows® 98, Microsoft® Windows® NT, Microsoft® Windows® 2000, Microsoft® Windows® Me, Microsoft® Windows® XP, Apple® MacOS®, IBM® OS/2®, Microsoft® Windows® CE, or Palm OS®.
  • the appropriate operating system may advantageously include a communications protocol implementation, which handles all incoming and outgoing message traffic passed over the network, which can include a wireless network.
  • the operating system may differ depending on the type of computer, the operating system may continue to provide the appropriate communications protocols necessary to establish communication links with the network.
  • Figure IB illustrates an example encoding system 100B in accordance with an embodiment of the present invention.
  • the term encoding system includes one or more encoders.
  • the encoding system 100B comprises, by way of example, one or more of processors, program logic, or other substrate configurations representing data and instructions, which operate as described herein.
  • the encoding system 100B can comprise controller circuitry, integrated circuits, gate arrays, application specific circuits, processor circuitry, processors, general purpose single-chip or multi-chip microprocessors, digital signal processors, embedded microprocessors, microcontrollers and the like, executing software code, including instructions and data stored in computer readable memory.
  • the encoding system 100B can be housed in one or more leaded, leadless, or ball grid array semiconductor packages, on one or more circuit boards, and/or using one or more hybrid packages. All or portions of the encoding system 100B may be included in a fixed terminal, such as a desktop computer, or in a portable terminal, such as a cellular phone, portable computer, personal digital assistant, video camera, or the like.
  • the encoding system 100B can, in an example embodiment, correspond to the encoding computer 102.
  • an encoding system in accordance with the present invention can be used to conduct video conferencing, to aid in the storage and transmission of movies or other images, and the like.
  • the encoding system 100B encodes and compresses video information for transmission to a decoder.
  • the encoding system 100B includes a preprocessing module or circuit 102B, a bit allocation module or circuit 104B, and an encoder module or circuit 106B.
  • the preprocessing module or circuit 102B including a video sequence analyzer, is used to detect when a scene change has taken place and to determine how a given frame, VOP or picture, is to be encoded.
  • a video object layer contains a sequence of 2D representations of arbitrary shape at different time intervals that is referred to in MPEG-4 as a video object plane (VOP).
  • VOP video object plane
  • Each of the VOP regions can be non-rectangular and may correspond to particular image or video content of interest, such as physical objects within a scene.
  • Video object planes (VOPs) are divided into macroblocks of size 16x16.
  • a macroblock is encoded in six blocks, four for luminosity and two for chromaticity, of size 8x8. To obtain a macroblock structure from an arbitrary shaped VOP, the bounding box of the VOP is calculated and extended to multiples of the macroblock size.
  • VOP For most current applications, and in particular for wireless applications using the so-called "simple profile,” there is generally only 1 VOP per frame, which is a rectangular VOP.
  • frame can also include a VOP, such as an MPEG-4 VOP, or a picture.
  • VOP can also refer to a frame.
  • the VOPs can be structured in groups of video object planes (GOV).
  • GOP video object planes
  • frames or pictures can be arranged in groups of pictures (GOPs).
  • scene as used herein, may also refer to a GOV or a GOP and visa versa.
  • a frame or video object may be encoded as an intracoded frame (an "I- frame” or “I- VOP”), as a predicted frame (a "P-frame” or “PNOP”), or as a bi-directional frame (a "B-frame” or “BNOP”).
  • MPEG-1 also provides for a D-frame.
  • a D-frame is a frame that has no motion vectors, so that a zero vector is assumed, and has texture DCT data.
  • DCT Discrete Cosine Transformation
  • the MPEG-4 simple profile does not support B-frames or B-VOPs. However, the simple profile does support frame skipping. A video frame can be skipped in order to reduce the resulting file size or bit-rate. Because the MPEG-4 simple profile does not support B-frames or D-frames, the following discussions will not focus on such frames. Nonetheless, embodiments of the present invention can be used with B-frames and D- frames in accordance with other profiles and other standards.
  • the term frame can correspond to either an interlaced frame or to a noninterlaced frame, i.e., a progressive frame.
  • a noninterlaced frame i.e., a progressive frame.
  • each frame is made of two separate fields, which are interlaced together to create the frame.
  • Such interlacing is not performed in a non-interlaced or progressive frame. While illustrated in the context of non-interlaced or progressive video, one or ordinary skill in the field will appreciate that the principles and advantages described herein are applicable to both interlaced video and noninterlaced video.
  • An intracoded I-frame typically only includes information from the image itself and thus an I-frame can be decoded independently of other frames.
  • P and B frames are also referred to as intercoded frames because they are encoded based on data from other frames.
  • the preprocessing module 102 generates a file, referred to as an input frame-type file, containing the frame-type designations corresponding to the frames.
  • the frame-type information is passed to other portions of the encoding system 100B using variables and the like. While the preprocessing module 102B is illustrated in Figure IB as being included in the encoding system 100B, the preprocessing module 102B can be physically separate from the other portions of the encoding system 100B. In such an embodiment, the preprocessing module 102B can produce a text file that includes frame-type designation that is then input by the remainder of the encoding system 100B.
  • YUV-4:2:0 files are header-less files with concatenated frames, where, for each frame, the (luminosity) Y-pixel values are provided first, followed by the (Chromaticity-blue) Cb-values, and then the (Chromaticity-red) Cr-values.
  • the term "4:2:0" indicates that chromaticity values are subsampled by a factor 4 with respect to luminosity.
  • the preprocessing module 102B performs frame evaluation and encoding designation. As will be described below, each frame is designated by the preprocessing module 102B as an I-frame, a P-frame, or as a skipped frame. In other embodiments, the preprocessing module 102B may also designate frames as B-frames or D-frames. B-frame encoding may be performed if there is sufficient computational power, available bandwidth (B-frames take much more bandwidth than skipped frames), and if allowed by the corresponding standard. For example, the MPEG-4 simple-profile syntax, used in wireless networks, does not allow for B-frames.
  • the example file format generated by the preprocessing module 102B includes a line per input frame, with a frame-type designation character on each line: 0, 1 or 2.
  • a "0" indicates an I-frame
  • a "1” indicates a P-frame
  • a "2" indicates a skipped frame.
  • designations can be provided for a bidirectional frame and a D-frame.
  • scene change frames are generally intracoded.
  • the preprocessing module's scene change analysis performs a color-weighted Root Mean Squared (RMS) calculation and a Mean Absolute Differences (MAD) calculation between the i th frame E, and the k th frame F k .
  • RMS color-weighted Root Mean Squared
  • MAD Mean Absolute Differences
  • F(x, y) denotes the (x, y) . th pixel in frame F, and w and h are the width and height of the frame, respectively.
  • Y(x, y) indicates the luminance value, while U(x, y) and V(x, y) are the two chromaticity components.
  • the Mean Absolute Difference (MAD) measure can be defined as follows:
  • the MAD does not need to include the two chromaticity components.
  • MAD(F l ,F k ) and/or RMS(F l ,F k ) are large or greater than a selected criteria, this indicates that the content of E, is substantially different from F k .
  • MAD(F l _ l , E, ) is larger than a pre- specified threshold, E, is designated a scene change frame.
  • An example threshold value for designating a scene change frame is approximately 25.
  • a second temporal derivative of the RMS can be used to determine if a frame is scene change frame, as follows:
  • the second temporal derivative of the RMS is based on the RMS value for the previous frame F,- ⁇ relative to the current frame F diet the RMS value of the current frame F, relative to the next frame F, +1 , and the RMS value of the next frame F, + ⁇ to the subsequent frame F 1+2 .
  • the second temporal derivative of the RMS value will be negative with relatively high amplitude when F t is a scene-change frame, as illustrated in Figure 3.
  • a pre-specified threshold E
  • both the RMS values the values of the second derivative of RMS generally provide a correct indication of a scene change.
  • An example second derivative of RMS threshold value for determining a scene change is -6.5.
  • the second derivative of the RMS is a good peak detector, it is somewhat sensitive to noise.
  • a frame is designated a scene change frame.
  • scene changes frames will be intracoded as I-frames or I-VOPs.
  • a frame is designated as a scene change, and thus will be coded in INTRA mode, when its MAD is greater than 20 and the second derivative of RMS is negative and has an absolute value of greater than 4.
  • a frame is designated as a scene change, and thus will be coded in TNTRA mode, when its RMS is greater than 40 and/or when the second derivative of RMS is negative and has an absolute value of greater than 8.
  • other thresholds can be used.
  • a second derivative of MAD can be used, as similarly described above with respect to the second derivative of RMS, as a further indication of whether a frame corresponds to a scene change or not.
  • An additional criterion can be used to determine when a scene change has occurred. For example, in one embodiment, a determination is made as to whether the MAD value is a local maximum, that is, has increased from a previous frame to the frame at issue, and then decreased from the frame at issue to the next frame. If so, this indicates that it is likely the frame at issue is a scene change frame and should be intracoded. In addition, a similar determination may be made for the RMS value. For example, a determination is made as to whether the RMS value is a local maximum, that is, has increased from a previous frame to the frame at issue, and then decreased from the frame at issue to the next frame. If so, this too indicates that it is likely the frame at issue is a scene change frame and should be intracoded.
  • a voting process can be used, wherein if at least two of the RMS, the second derivative of the RMS, and the MAD, meet corresponding criteria, then a frame is designated as a scene change that is to be intracoded.
  • the RMS and second derivative of the RMS meet the corresponding criteria, and if the MAD is a local maximum, then the frame is designated as a scene change frame.
  • the frame is designated as a scene change frame.
  • the number of frames that needs to be encoded per second is preferably reduced as much as acceptable.
  • One technique used to reduce the number of frames encoded per second is to skip-frames in the encoding process.
  • Two example frame-skipping techniques are fixed frame skipping and adaptive frame skipping.
  • Conventional encoders drop frames to reduce the frame rate according to a simple skipping algorithm. For example, a conventional encoder will drop every 4 of 5 frames in a video clip to convert the video clip from a 30 frames per second rate to a 6 frames per second rate.
  • fixed frame skipping tends to provide better error resiliency in noisy environments, while adaptive frame skipping tends to provide better visual results in low noise environments.
  • the preprocessing module 102B calculates, based on the bit-rate/frame-rate formula defined in Equation 7 below, the target encoding frame-rate and then switches between adaptive and fixed skipping in order to meet a target encoding frame rate.
  • the input video frame sequence is subsampled along the time axis in order to achieve a desired or predetermined average frame rate.
  • the rate of frame skipping can be irregular and can vary along the sequence length.
  • low activity frames are identified and skipped, and scene-change frames are kept and intracoded.
  • Non-scene changes having some activity frame are intercoded. Because the skipped frames are intelligently selected based on changes on visual activity, the visual result when reproduced by the decoder will be better than with fixed frame skipping, assuming no or relatively few errors occur.
  • the preprocessing module 102B codes skipped frames using a "not_coded" bit-flag or indicator set in the video object plane (VOP) header in an MPEG bit stream.
  • VOP video object plane
  • An MPEG-4 video packet starts with the VOP header or the video packet header, followed by motion_shape_texture(), and ends with next_resync_marker() or next_start_code).
  • a VOP specifies particular image sequence content and is coded into a separate video object layer by coding contour, motion and texture information.
  • a skipped frame is skipped altogether, without inserting VOP-header information in the bitstream.
  • a skipped frame may be recreated by a decoder using interpolation or by repeating a previous frame.
  • the decoder may perform interpolation by pixel averaging between a preceding frame and a subsequent decoded frame, weighted by their time difference.
  • the information regarding the video sequence frame rate is normally carried in the Video-Object-Layer (VOL) header.
  • vop_time_increment_resolution determines the number of time units for each encoding cycle.
  • the vop_time_increment value in the Video-Object-Plane (VOP) header carries the time stamp for each frame.
  • the decoder may incorrectly determine the frame rate of the entire sequence.
  • the "fixed_vop_rate" flag is set by the encoder module 106B in the VOL header, which then provides (via the value of fixed_vop_time_increment) the default frame rate. This technique better ensures the successful decoding or determination of the frame rate upon the successful decoding of the VOL header.
  • the fixed_vop_time_increment value can later be stored in a global variable of the decoder, which will use the value to determine whether certain frames need to be interpolated or not.
  • the frames to be interpolated can either be frames skipped by the encoder, or lost during transmission.
  • the error-resilience performance of the MPEG-4 decoder will be enhanced because it will decode the correct number of frames, thereby avoiding loss-of- sync problems with the audio stream.
  • the process is iterative in that a frame is selectively dropped from the sequence of frames by computing a mean of absolute differences between the frames adjacent to the frame of interest, and by weighting the computation with a temporal parameter, wherein the frame having the least impact on the scene is dropped. This procedure is repeatedly iterated until a target frame rate, which is related to the desired bit rate and frame size, is achieved.
  • the process 400 proceeds to state 404.
  • the desired frame rate is set or specified.
  • the desired frame rate may be user specified or may be dynamically determined.
  • the cost function, or adverse impact, that would result from dropping a particular frame is calculated for each frame between the first and last frame in a scene.
  • the cost function can be based at least in part on the mean absolute differences (MAD) between frames closely or most closely bracketing or bounding the particular frame of interest, or on the sum of the mean absolute differences (SMAD).
  • the cost function can be based on sums of RMS (SRMS) for frames bracketing the particular frame of interest.
  • the frame associated with the lowest cost that is, having the least adverse impact on visual quality, is skipped or dropped.
  • a determination is made as to whether the remaining frames will allow the target frame rate to be met. If the target frame rate can now be met, the adaptive frame rate skipping process 400 proceeds to the end state 414. Otherwise, the process 400 proceeds to state 412, and a remaining frame having the lowest cost will be dropped. The cost of all the frames remaining between the first and last frames may be recalculated at state 412 based on the frames that are currently remaining, and the frame with the lowest cost will be dropped.
  • the process 400 repeats states 410 and 412 until the target frame rate has been met, or the number of already consecutively skipped frames is at a specified maximum. While the process 400 is described with respect to selecting which frames to skip, the process 400 can be similarly used to decide what frames should be bi-directionally encoded.
  • the video sequence analyzer skips frames in a "greedy" way, that is, until r des is met.
  • the weighted sum of MAD and the time difference between two frames are used in specifying a cost function for identifying the next frame to be skipped.
  • the frame whose cost function meets predetermined criteria or is minimum among remaining candidates for skipping is skipped.
  • E 4 is now considered as a candidate for the next skipped frame.
  • a cost function is calculated assuming that E 4 has been skipped.
  • E 2 and E 7 will be the left and right frames bounding the skipped segment of E 3 - E 6 .
  • the cost function is then defined as:
  • 29.97 r ong is used to normalize the original frame rate r 0ng with respect to the NTSC frame rate of
  • the weighting coefficient value ⁇ 5.0, determined experimentally, provides a suitable result.
  • the weighting coefficient value ⁇ may be dynamically determined.
  • the frame whose cost function is minimum among the candidates is skipped. This process is iteratively performed until the desired frame rate is met.
  • the time difference measure TD can be modified such that no more than n consecutive frames are skipped. For example, if a predetermined maximum number of consecutive frames that can be skipped simultaneously is set to 4, TD can be modified as follows
  • the complexity for the MAD calculation can be reduced, though the accuracy will be reduced, if only even (or only odd) coordinate pixels are used in the calculation. For example:
  • Another example process to adaptively determine which frame to skip estimates the total induced distortion, both spatial and temporal, for each candidate frame for skipping, and then skips the frame whose absence would result in the least distortion.
  • the process utilizes sums of MAD (SMAD) or sums of RMS (SRMS).
  • SAD sums of MAD
  • SRMS sums of RMS
  • all the MADs do not have to be recalculated. Instead, the already calculated appropriate MADs are summed differently, depending on which frame is being considered for skipping.
  • the process is performed as follows. Assume that F t is the frame currently being considered for skipping. For example, assume F t is frame
  • Equation 6b the cost function subtracts out the contribution from previously skipped frames.
  • the distortion is calculated as follows:
  • ⁇ (n) is a coefficient that depends on the number (h) of consecutive skipped frames and takes into account how much, on the average or based on a statistical sampling, interpolated frames at the decoder are different from the original frames.
  • ⁇ (n) is a coefficient that depends on the number (h) of consecutive skipped frames and takes into account how much, on the average or based on a statistical sampling, interpolated frames at the decoder are different from the original frames.
  • ⁇ (n) increases as the number of skipped frames increases.
  • Equation 6b To reduce the computational complexity and resources needed, optionally only the MAD between consecutive frames (j-1 and j) is used to estimate the distortion of a complete segment of skipped frames, as in Equation 6b above. The additional distortion when skipping a certain frame is calculated and the temporal component is added.
  • a user specifies the desired encoding frame rate.
  • the desired frame rate can be based on the video sequence statistics, such as temporal and spatial complexity, frame size, frame rate and target bit rate or compression ratio.
  • a heuristic can be used to select the desired frame rate. The following is an example heuristic equation for calculating the frame rate, normalized with respect to the Quarter Common Intermediate Format (QCIF) frame size, containing 144 lines and 176 pixels per line:
  • QCIF Quarter Common Intermediate Format
  • the encoding frame rate should preferably be in the range of :
  • the weighting coefficient ⁇ is optionally set equal to the average RMS or MAD of the designated entire sequence.
  • the error resilience flag is set by the user to switch between adaptive frame skipping, which has less error resiliency but a better visual result when there are no or few errors, and fixed frame skipping, which provides better error resiliency with a lesser visual result.
  • an RMS circuit 102C is used to calculate RMS values as described above
  • a Second Derivative of RMS circuit 104C is used to calculate the second derivative of RMS as described above
  • a MAD circuit 108C is used to calculate the MAD values as described above
  • a SUM OF MAD circuit HOC is used to calculate the SUM of MAD values as described above
  • a Second Derivative of MAD circuit 114C is used to calculated the Second Derivative of MAD as described above.
  • An Evaluator circuit 112C coupled to the outputs of the RMS circuit 102C, the Second Derivative of RMS circuit 104C, the MAD circuit 108C, and the SUM OF MAD circuit HOC, and the Second Derivative of MAD circuit 114C, is used to determine when a scene change has occurred and what frames to skip, based on one or more of the outputs, as discussed above.
  • different embodiments need not include all or any portion of the circuits illustrated in Figure lC.
  • the bit allocation module or circuit 104B provides for bit allocation on a scene, frame, and/or macroblock level.
  • the bit allocation module 104B reads the file or otherwise receives the information generated by the preprocessing module 102B, including the frame-type designations, and calculates a bit budget for each scene, GOV or GOP based on the coded frames.
  • the bit allocation module determines an appropriate distribution of a fixed bit budget.
  • a first intracoded frame defines a beginning of a scene.
  • a weight is assigned to the scene based on the number of intracoded frames and the number of intercoded frames, where intracoded frames are weighted more heavily then intercoded frames to account for the greater number of bits needed to encode an intraframe.
  • the bit allocation module distributes the fixed bit budget within a scene by comparing the current bit usage and target bit usage, and based on the comparison, adjusts a quantization parameter or step size for the current frame.
  • bit-allocation module 104B first parses the input frame- type file from the preprocessing module 102B. The number or quantity of GOVs is then calculated. Based on the calculated bit budget, the encoder module 106B then encodes each GOV using the novel rate control process in accordance with an encoder parameter file, discussed below.
  • N c i number of coded P-VOPs (predicted, intercoded VOPs) in scene, or GOVi.
  • N c number of coded frames for the clip
  • VOPs (intracoded VOPs) T c equivalent total number of VOPs in clip
  • An example method of determining the bit-budget for a scene or GOV is as follows.
  • B bits are allocated for each GOV (i).
  • the method is not limited to using the 1:10 ratio.
  • B, and T c are then determined as follows:
  • the bit allocation for a given scene is based on the total number of frames in the scene, wherein an intracoded frame is normalized to be the equivalent of several predicted frames.
  • this example bit allocation formula does not take into account the spatial and temporal complexity of each GOV or GOP. In other embodiments, if sufficient computational and time resources are available, the bit allocation formula takes the temporal and spatial complexity into consideration to provide a still more intelligent bit allocation for each GOV.
  • a two-pass encoding process takes into account spatial and temporal complexity.
  • the first pass detects scene changes and collects frame complexity.
  • the second pass performs the actual encoding using complexity guided bit allocation.
  • the first pass process will now be described in greater detail.
  • a new GOV or GOP is started from a scene change instance.
  • the two-pass rate control process provides substantially uniform quality for each temporally segmented GOV so that quality variation is better limited to GOV or GOP boundaries. This approach is taken because minimizing quality variation, as measured by the Peak Signal to Noise Ratio (PSNR), the Root Mean Square Error or other image fidelity metric, among different scenes provides less benefit with respect to human visual perception.
  • PSNR Peak Signal to Noise Ratio
  • the complexity measure in accordance with one embodiment of the present invention is relatively invariant with the quantization parameter (QP) used.
  • QP quantization parameter
  • the bit count for non-texture information such as frame headers/syntax and motion vectors, denoted by H Ti is constant or almost constant in terms of quantization parameter (QP) change.
  • QP quantization parameter
  • the bit count of the texture information which does vary with a change in QP.
  • M is the MAD computed with a motion-compensated residual that is substantially invariant with respect to the QP (i.e., Q,), and ai and a 2 are Taylor expansion coefficients of texture bits T, over QP.
  • the coefficients ⁇ and a 2 are normally of the same order, that is, have similar values. As can be seen, the lower the QP, the greater then number of texture bits needed to encode a given frame.
  • C g ! addresses both the motion and texture bit count, and is substantially QP invariant.
  • the frame complexity can be generated in one pass, that is, with one QP.
  • the calculated frame complexity will be utilized in the second encoding pass as will now be described.
  • the rate control process 900 in the second pass consists of a three level hierarchy, scene (GOV or GOP)-level bit allocation 902, frame- level bit allocation 904, and macroblock-level QP adjustment 906 which utilizes the obtained frame complexity values C g) ,.
  • the GOV or GOP level recursive bit allocation process 1000 is applied, as illustrated in Figure 10.
  • bits are assigned to the scene (GOV or GOP) of index g according to the following formula:
  • the case with 0 ⁇ 1.0 represents a bit-allocation tradeoff between the buffer and the quality constraints.
  • the GOV or GOP scene-level bit allocation advantageously suitably allocates the bit budget to each GOV or GOP while meeting both the buffer and the quality constraints.
  • the frame level bit allocation process 904 illustrated in Figure 9 performs such a frame bit allocation process, and is similar to the process 1000 for the GOV or GOP level bit allocation.
  • the variables corresponding to frames are utilized.
  • the macroblock level QP adjustment 906 can be alternatively performed in accordance with the following description.
  • a safe margin 0.8 of the maximal buffer by way of example
  • all the macroblocks are quantized with the same quantization parameter (QP) using a one-pass rate control.
  • QP quantization parameter
  • N MB is the number of macroblocks in one frame
  • MAD k is the mean absolute difference of MB k
  • QP k -i is the QP for a previous macroblock.
  • the QP for the current MB k can be in the range of [QP k - ⁇ -2, QP k - t +2], as determined in accordance with the following rules:
  • the macroblock bit allocation process can be disabled or not used so that the decoder can assume that the QP is the same for each macroblock. This helps prevent the decoder from using the wrong QP when portions of a frame have been corrupted or lost.
  • a novel rate control process is used to meet or substantially meet the calculated bit budget.
  • IMP4 a novel rate control process
  • Conventional MPEG-4 rate control does provide adequate performance for many applications. For example, many conventional rate control processes do not explicitly support multiple scenes. Instead, these conventional rate control processes assume that an entire sequence comprises a single scene, and therefore fail to provide for satisfactory rate control. By contrast, an embodiment of the present invention takes into account when scene changes occur, and so provides enhanced rate control.
  • a self-converging rate control processed is used to meet the bit budget of each GOV by adjusting the quantization parameter QP of each frame, where QP is equal to half the quantization step size.
  • the quantizer parameter QP can have 31 values [1-31].
  • the rate control process determines the QP based on past bit usage, the number of un-coded frame and the rest bits for a given GON. Thus, if the current bit usage exceeds the assigned bit budget by more than a certain amount or percentage, the quantization parameter, and therefore the quantization step size, are increased. If, instead, the current bit usage falls beneath the assigned bit budget by more than a certain amount or percentage, the quantization parameter, and therefore the quantization step size, are decreased.
  • the process can be performed in accordance with the following example pseudo-code:
  • QPnext max (1, INT(QPcur-StepDwnPrcnt* QPcur));
  • Margin 1 is a constant that allows the current bit usage to exceed the assigned bit budget, so that the system has an opportunity to stabilize.
  • Marginl can be set equal to 1.15, allowing the current bit usage to exceed the assigned bit budget by 15%.
  • Margin2 is a constant that allows the current bit usage to under-run the assigned bit budget, so that the system has an opportunity to stabilize. For example, Margin2 can be set equal to 0.85, allowing the current bit usage to under-run the assigned bit budget by 15%.
  • StepUpPrcnt is constant related to how much the quantization parameter is to be increased.
  • StepUpPrcnt may be set equal to 0.1.
  • StepDwnPrcnt is constant related to how much the quantization parameter is to be increased.
  • StepDwnPrcnt may be set equal to 0.1.
  • the actual bit usage B act is compared with the pre-assigned bit budget B ass , and if the actual bit usage varies by more then a certain amount or percentage (Margin3, Margin 4) from the budgeted bit usage, the quantization parameter QP for an I-frame (QPI) is adjusted up or down by a certain amount (StepUp, StepDwn) or percentage as needed.
  • the quantization parameter adjustment process can be performed in accordance with the following example pseudo-code: If (B act > Margin3 * Bass)
  • Margin3 1.2
  • the initial value of QPI may be set to 10, while the initial value of QP for a P-VOP may be set to 12. Note that when the quantization parameter QPI for I- VOP is changed, the QP assignment of the following P-frames may also change.
  • the encoder module 106B then performs adaptive motion change detection to efficiently reduce large propagation errors.
  • adaptive intra refresh is used to reduce error propagation in an MPEG data stream by the selective intra-coding of macroblocks in p-frames.
  • AIR is used to help determine how many macroblocks should be intra-encoded in the detected motion region of a frame. While the performance of macroblock intra-refresh increases error resilience of the compressed bitstream, increasing the number of intra-coded macroblocks correspondingly increase the number of bits used to encode these macroblocks. Further, if there is a fixed bit rate, the quantization error has to increase for the other, non-intracoded, macroblocks. Thus, preferably, bandwidth and the bit error probability (BER) are taken into account to determine the percentage or number of macroblocks that are to be intracoded.
  • BER bit error probability
  • the encoder module 106B optionally used Cyclic Intra Refresh (CIR) to encode a predetermined number of macroblocks (MBs) in each frame.
  • CIR Cyclic Intra Refresh
  • Cyclic Intra Refresh (CIR) and Adaptive Intra Refresh (AIR) are performed as follows.
  • the number of Intra macroblocks in a VOP is specified by the user in the encoder parameter file.
  • the number of Intra macroblocks in a VOP depends on target bit rate, frame rate, bit buffer usage, channel noise feedback, and other transmission related parameters.
  • the encoder module 106B estimates the amount of motion for each macroblock and selects heavy motion area to be encoded in INTRA mode to enhance error resiliency. The results of the estimation are recorded a refresh map at the macroblock level.
  • FIG. 2 A An example conventional refresh map 202 is illustrated in Figure 2 A.
  • the encoder module 106B refers to the refresh map and selectively determines whether to encode a given macroblock of the current VOP in TNTRA mode or not.
  • the estimation of motion is performed by comparing SAD (Sum of the Absolute Difference) and SAD t h- SAD refers to the Sum of the Absolute Differences value between the current macroblock and the macroblock in the same location of the previous VOP.
  • the SAD is already calculated when performing motion estimation. Therefore, the SAD calculation does not have to be repeated as part of the AIR process.
  • SAD th is used as a threshold value in determining whether a given macroblock is a motion area. If the SAD of the current macroblock is larger then SAD th , this macroblock is regarded as motion area.
  • a macroblock is regarded as a motion area, it remains as a candidate motion area until it is encoded in Intra mode a predetermined number of times.
  • the value for this "predetermined number of times" is set "1," in other embodiments, the predetermined number of times can be set equal to 2, or a higher value.
  • Horizontal scanning is used to resolve among macroblocks that are candidates to be encoded in Intra mode within the moving area as illustrated in map 204 in Figure 2B.
  • the AIR refresh rate that is, the fixed number of Intra macroblocks in a VOP is preferably determined in advance.
  • the number of Intra macroblocks in a VOP is set to "2" .
  • the first VOP is a scene change frame containing elements 802, 804. Therefore, all macroblocks in the 1st VOP are encoded in Intra mode, as illustrated in Figure 8 A [a].
  • the refresh map is set to "0", where a 0 indicates that an Intra refresh is not to be performed and a 1 indicates that an Intra refresh is to be performed, because the 1st VOP is encoded without reference to a previous VOP.
  • Elements 802, 804 have moved down by an additional macroblock and to the right by an additional macroblock.
  • the encoder module 106B refers to the Refresh Map illustrated in Figure 8 A [g]. If the refresh map indicates that a macroblock is be Intra refreshed, the macroblock is encoded in Intra mode, as illustrated by the macroblocks containing an "X" in Figure 8 A [h]. The corresponding refresh map value for an intracoded macroblock is decreased by 1 as illustrated in Figure 8A [i].
  • the processing is substantially the same as that for the 2nd VOP as illustrated in Figure 8A [j]-[k], where if the SAD for a given macroblock is larger than SAD th , the given macroblock is regarded as motion area.
  • the refresh map is updated as illustrated in Figure 8 A [k], where the refresh map entry corresponding to a motion macroblock is set to 1.
  • the processing is substantially the same as for the 3rd VOP. If a current macroblock has a 1 associated with it in the refresh map, it is encoded in Intra mode as illustrated by the macroblocks containing an "X" in Figure 8 A [m]. The corresponding refresh map value for an intracoded macroblock is decreased by 1 as illustrated in Figure 8A[n]. [0165] If the decreased value is 0, the corresponding macroblock is not regarded as a motion area. If the SAD for a given macroblock is larger than SAD th , the given macroblock is regarded as motion area. The refresh map is updated as illustrated in Figure 8A [p].
  • a novel enhanced AIR process is performed as follows to select which macroblocks are to be intracoded in a predicted frame.
  • An intercode distortion value and an intracode distortion value are calculated, as are an intercode bit rate and an intracode bit rate. Based on a comparison of the calculated intercode distortion value and the intracode distortion value, and on a comparison of the intercode bit rate and the intracode bit rate for each macroblock, a decision is made as to which macroblocks are to be intracoded.
  • the enhanced AIR process will now be described in greater detail.
  • the expected distortion that would result if the macroblock were lost or corrupted is estimated.
  • the distortion can be reduced if the reference macroblock in the prediction is intracoded.
  • recursive tracking in conjunction with the prediction path can be used to determine the expected distortion of the macroblock.
  • the dashed lines 804B to 818B, 806B to 820B, 820B to 826B, 812B to 822B, 814B to 824B, 822B to 828B, and 828B to 830B indicate motion vectors (MV) that are part of the encoded bitstream from a macroblock in the previous frame to a macroblock in the current frame.
  • the angled solid lines such as those from 802B to 818B, 806B to 820B, 810B to 822B, 816B to 824B, 818B to 826B, 824B to 828B, and 826B to 830B, indicate a zero- motion vector, where a lost motion vector is set to zero.
  • the encoder module 106B While encoding a given current macroblock, the encoder module 106B performs a motion search on the previous frame and locates a macroblock that most closely matches the current macroblock or is otherwise determined to be a good prediction frame. This located macroblock from the previous frame, depicted by a non-hatched circle, such as macroblocks 802B, 806B, 81 OB, 816B, 818B, 820B, 826B, is called a prediction macroblock.
  • a residual error is calculated and further encoded using the Discrete Cosine Transform (DCT), then quantized using a selected quantization step or quantization parameter (QP), and entropy coded using variable length coding (VLC).
  • DCT Discrete Cosine Transform
  • QP quantization step or quantization parameter
  • VLC variable length coding
  • the encoded bitstream consists of motion vector information, entropy coded quantized DCT coefficients for the residual error, and corresponding header information.
  • the decoder When the decoder receives the encoded bitstream, the decoder processes the coded information and reconstructs the macroblocks. When information for a macroblock is missing, which may be due to packet loss or other error conditions, the decoder preferably conceals the corresponding macroblock using one or more error- concealment strategies, such as the basic concealment discussed above. As discussed above, when a macroblock is missing, basic concealment copies a macroblock at the same spatial location from the previous frame. This is equivalent to receiving a zero-motion vector and zero DCT coefficients.
  • the encoder system 100 includes corresponding decoder circuitry so that it can mimic the decoder process and reconstruct what the decoder will reconstruct both in the absence of errors, and in the presence of one or more errors, such as a single error affecting just the current macroblock ("MBC").
  • MBC macroblock
  • EC concealment error
  • Error attenuation occurs when half-pixel accuracy is used for prediction either in the vertical or horizontal direction or in both the vertical and the horizontal directions. Error attenuation, comparable to a low pass filter, occurs as a result of the low- pass frequency characteristic of the pixel averaging operation applied when half-pixel motion is used.
  • the propagated error via half-pixel motion in the horizontal direction ECh/2, the propagated error via half-pixel motion in the vertical direction ECv/2, and the propagated error via half-pixel motion in the horizontal and vertical direction EChv/2 can be determined.
  • Half pixel interpolation is illustrated in Figure 8D, showing integer pixel locations, half-pixel locations in the horizontal direction, half-pixel locations in the vertical direction, and half-pixel locations in the horizontal and vertical dimension.
  • the half-pixel averaging filter that is normally applied to pixel values can be applied to the concealment error, EC, to define four types of propagated error arrays:
  • ECh/2 error through horizontal half-pixel motion (value calculated on crosses "x" in Figure 8D)
  • EChv/2 error through horizontal and vertical half-pixel motion (value calculated on squares in Figure 8D)
  • a motion vector MV can map macroblocks in the current frame Framen, aligned with a grid of 16-pixel rows and columns, into 16x16 pixels in the predicted frame Frame n . ⁇ that are not necessarily aligned on the same grid. Indeed, as illustrated in figure 8E, a macroblock in Frame n can map to portions of up to four macroblocks in the predicted frame Frame justify- ⁇ .
  • An error present on one or more of the four possible macroblocks from the previous frame used for prediction for a macroblock in the current frame will be reflected in the macroblock in the current frame.
  • the error relationship can be proportional to the overlap area. For example the error relationship can be proportional or based on the number of pixels that they overlap.
  • the up to four prediction macroblocks are identified that would be used when encoding the macroblock in Inter mode.
  • ⁇ . . is one of the transition factors ⁇ EC , ⁇ ⁇ /2 , ⁇ v/2 and ⁇ ⁇ v/2 defined previously, depending on the type of motion (half or integer pixel motion vector)
  • ⁇ I ( ⁇ ) is the concealment error ⁇ E 2 C for macroblock i.
  • FIG. 8B With reference to Figure 8B, beginning with the macroblock 830B in current Frame n , there are two macroblocks in Frame n -i that may be used by a decoder to recreate macroblock 830B, a macroblock 826B used for normal decoding, and a macroblock 828B used for concealment. Each of the macroblocks 826B, 828B in Frame n -i may correspond to up to 4 aligned macroblocks, as discussed above.
  • the same "decode or conceal" strategy can be recursively applied for the two macroblocks 826B, 828B in Frame n . ⁇ to locate 4 macroblocks 818B, 824,B, 822B, 829B in Frame n-2, and then reach Frames with 8 macroblocks 802B, 804B, 806B, 808B, 810B, 812B, 814B, 816B, and so on.
  • Each of the 8 macroblocks in Frame n _ 3 has a probability of appearing in Frame n at the current macroblock, if a certain series of errors/packet loss occurs during transmission.
  • the probability of a particular path to the macroblock 830B occurring can be determined by multiplying the p and q values along the path.
  • there exist paths that have probability p 2 such as those where two packet losses in a row occur, and a path defined by 812B-830B with probability
  • the expected distortion for the current macroblock in Frame n can be estimated using the propagation model described above.
  • the expected distortion is defined using the following equation:
  • D'(n -1) is the expected distortion for the reference macroblocks in Frame n - ⁇ , as modified by the transition factors to take into account the possible half-pixel motion from Frame n -i to Frame n . Expanding this formula for the reference macroblock in Frame n -i, expected distortion is defined as follows:
  • D(n) p ⁇ E 2 C (n) + - 2)) Equation 21
  • ⁇ ((1 _, n) is one of the 4 transition factors ( ⁇ EC , ⁇ ⁇ /2 , ⁇ v/2 and ⁇ ⁇ v/2 ) for the reference macroblock in Frame n - ⁇ , depending on the motion vector from Frame n -i to Framen.
  • D"(n - 2) is the expected distortion for the reference macroblocks in Frame n _ 2 as modified by the transition factors to take into account the possible half-pixel motion from Frame n - 2 to Framen-i and from Frame n -i to Frame n . Expanding this term further, the expected distortion is defined as:
  • Frame n - 3 is an I-frame or if the frame buffer is limited or restricted to 3 frame, then D m (n -3) is equal to zero. Otherwise, the same procedure is recursively applied to previous frame macroblocks. Similarly, if an Intra macroblock is encountered during the recursive processes, it is assumed that the distortion is equal to p ⁇ c , because there is no motion vector, and thus no error-propagation term.
  • the information stored for each macroblock of previous frames can be utilized to calculate the expected distortion for each macroblock for the current Frame n .
  • this expected distortion is due to errors in transmission and is not correlated to the distortion due to quantization for each macroblock. Therefore, the expected distortion term needs to be added to the quantization error to determine the total distortion for each macroblock.
  • This total distortion is referred to as "total Inter-mode distortion,” or D TINTER , as it relates to Inter mode encoding of macroblocks.
  • a full Rate- Distortion optimization can be performed that involves determination of an optimal weighting factor ⁇ to be used for evaluating a cost function for each macroblock, given by:
  • macroblocks with ⁇ R ⁇ 0 are grouped together and ordered in decreasing order with
  • a less resource intensive process is to calculate the expected distortion for each macroblock due to transmission error, while ignoring or excluding quantization error. Then, the differential between the expected distortion for Intra and Inter mode can be used as the criterion for selecting macroblocks to be Intra coded by ordering them according to this criterion.
  • Adaptive Intra Refresh can be used to help determine how many macroblocks should be intra-encoded in the detected motion region of a frame.
  • AIR can be enabled and disabled in the encoder parameter file using an AIR bit set by a user and read by the encoder module 106B.
  • the user also specifies another parameter, the AIR refresh rate.
  • the AIR refresh rate determines how many macroblocks should be intra-coded in the detected motion region of one frame.
  • Adaptive motion change detection can efficiently reduce the large propagation error, even when the error occurs in the motion region.
  • Figure 8H illustrates an embodiment of the E-AIR process.
  • the motion vector or vectors for the current macroblock of interest in Frame (n) are received.
  • the motion vector is used to locate which macroblocks from a previous Frame (n-1) are to be used in predicating the current macroblock.
  • a determination is made as to how much, in terms of area or pixels, of each of the located macroblocks in Frame (n-1) will be used in generating the current macroblock.
  • the error variances ( ⁇ EC> ⁇ Ech/2, ⁇ EC V/2, ⁇ Echv/2) are calculated, including the overlap
  • the Intra Error Distortion D INTRA for Frame (n) is calculated.
  • the Inter Error Distortion D ⁇ NTER for Frame (n) is recursively calculated.
  • the recursive calculation can include the error distortion and quantization distortion from previous frames, such as Frame (n-1), Frame (n- 2), and so on, whose errors may propagate to Frame (n).
  • the recursion may be limited to a predetermined number of frame generations, until all or a predetermined amount of the frame buffer is being used, or the recursion may stop when an Intra frame is reached.
  • the value DeltaD is calculated by taking the difference between DINTRA and DI NT ER, or by otherwise comparing D 1NT RA and D ⁇ NTER-
  • the bit quantity or bit rate RI NTRA and R INTER for intracoding Frame (n) and for intercoding Frame (n) respectively are determined.
  • a comparison of R INTRA and R INTER is made by calculating the difference DeltaR.
  • the decision to intracode or to intercode is made based on DeltaR, DeltaD and Lambda using the illustrated criteria. Alternatively, those macroblocks having a DeltaD may be chosen for intracoding.
  • FIG. 8C illustrates experimental results comparing the use of Cyclic Intra Refresh, trace 802C, with the use of the enhanced ATR method described immediately above, trace 804C.
  • the overall gain is approximately 1 dB in the PSNR. In one embodiment, the additional computational load is approximately 10%.
  • CIR Cyclic Intra Refresh
  • AIR Cyclic Intra Refresh
  • ATR_refresh_rate macroblocks are encoded in AIR mode
  • CTR_refresh_rate macroblock are encoded in the conventional CIR mode.
  • the encoder parameter file specifies many different encoding parameters, including those discussed above.
  • the encoder parameter file can be used in conjunction with the preprocessing module output by reading the frame- type file, which specifies the encoding type are determined by preprocessing process described above.
  • the encoder parameter file includes fields to enable/disable AIR, CIR, and SMC, to specify the ATR and CIR refresh rates, and a flag used to enable or disable the inclusion of two I-frames at the beginning of each scene, GOV or GOP.
  • HEC Header Extension Code
  • a Header Extension Code is included by the encoder module 106B in every packet in a sequence of video packets or in every video packet, and not just on the first video packet following the VOP header as with conventional encoders. This better ensures that even if a packet is lost or corrupted, subsequent packets can still be decoded and used. Further, even typical conventional decoders will be able to handle the inclusion of the enhanced use of HECs as the use of additional HECs is compatible with the MPEG-4 bitstream syntax. Adding a header, including sequence information, to all packets increases overhead by only about 40 bits per packet, or about 0.2%, but results in a noticeable improvement in decoding.
  • Second-order Motion Compensation is optionally provided to enhance error resiliency.
  • the SMC process is performed by the encoder module 106B and generates supplemental motion vectors so that each predicted frame can be predicted separately from two preceding frames.
  • Sequence 502 of Figure 5 illustrates the SMC process, in which k th frame has motion vectors from both from the (k-l) t h frame and the (k-2) tn frame. Therefore, even if the motion vectors from the (k-l) th frame are corrupted, or the (k-l) th frame is itself corrupted, the k t h frame can still be predicted from the (k-2) th frame using the corresponding motion vectors.
  • the scene image quality at the decoder-side will be better protected from transmission errors.
  • the use of SMC can effectively suppress error propagation by excluding k th frame from being used in any later prediction as illustrated in Figure 5 by sequence 504.
  • a frame buffer is included in the encoder module 106B to store the previously decoded frame at time (t-2). This previously decoded frame is used to calculate the second order motion vectors. In one embodiment, these redundant motion vectors are not used in the encoder to produce residuals.
  • the decoder uses the second order motion vectors when the bitstream is corrupted during the transmission and the first order motion vectors or corresponding frame is corrupted. Otherwise, the second order motion vectors need not be used in the decoder.
  • full, unrestricted motion search can be performed in order to determine these second-order motion vectors.
  • the information regarding motion between frames (t-2) and (t-1) and between frames (t-1) and (t) can be combined in order to estimate these second-order motion vectors.
  • the SMC data can optionally be included, via the "user data mechanism", as explained below, for the first P-VOP following a scene change or for each P-VOP.
  • the advantage of having SMC on only the first P-VOP is that bandwidth is not wasted when there is no error, thereby providing better coding efficiency.
  • providing SMC for every P-VOP or for many P-VOPs enhances the robustness and decoding ability of the decoder, especially for cases of severe error conditions.
  • An additional video packet referred to as "User data" or an "SMC" video packet, for each P-VOP is used to transmit these second-order motion vectors.
  • This packet contains, in the same predictive fashion and using the same variable-length codes as in the standard motion vectors, a motion vector for each macroblock or selected macroblocks of the current P-VOP.
  • An HEC is included in this special SMC video packet, which allows the SMC video packet to be decoded even if other packets for this P-VOP are lost.
  • this packet is positioned in the bitstream at the end of each P- NOP. A user can enable or disable the use of SMC by setting to 1 or 0 the corresponding option in the encoder parameter file.
  • Figure 6 illustrates an example packetized bitstream showing the relative position of packets in the bitstream, including the SMC packet 602.
  • a so-called "User data start code” (hex code B2) or the like, including other unique identifier codes to be assigned in the future by MPEG-standards committee and the like, precedes the HEC and motion vector information.
  • the user data start code signals standard decoders not capable of using the second order motion vectors to ignore all bits following it until the next start code in the bitstream, which in this will be a VOP start code.
  • the encoder includes a unique 16-bit identifier in order not to confuse the SMC user data extensions with data that other people may decide to include in the bitstream following the same convention.
  • the use of two consecutive I-frames advantageously prevents the prediction of a frame in the current scene using scene content from other scenes, without degrading the performance of the SMC. Because the first two consecutive frames in a scene are intracoded, neither first nor second-order motion vectors are inserted into the I- frames.
  • the inclusion of the consecutive I-frames can be under the control of the preprocessing module 102B which can designate both a scene change frame and the next frame as intracoded frames.
  • the encoder module 106B can automatically intracode a frame following a frame designated as an intracoded frame by the preprocessing module 102B.
  • a "consecutive I-frame" flag is provided in the encoder parameter file that can be independent of the SMC flag.
  • the presence of two consecutive I-frames at the beginning of each scene can be used for the decoder to conceal transmission errors more efficiently, even if the SMC mode is not turned on, or when it is turned on for just the first P-VOP following the (double) I-frame after a scene change.
  • Adaptive Intra Refresh is optionally supported by the encoder module 106B as a by-product of SMC.
  • This mode enabled when selecting SMC for just the first P-VOP or for every P-VOP, encodes in INTRA mode those macroblocks that have as prediction macroblocks from frames (t-1) and (t-2) two significantly different macroblocks, as measured by the MAD distance measure.
  • An example threshold is 20. Thus, if the MAD between the two prediction macroblocks for a given macroblock in the current frame is greater than 20, this macroblock is intracoded.
  • the encoder module 106B also performs general encoder functions, such as motion estimation, residual calculation, and the like.
  • the encoder output can be stored for later transmission or can be transmitted in substantially real-time to a receiving terminal, such as a cellular phone, containing an appropriate decoder.
  • Intra_dc_vlc_thr is set to "0", so that all DC coefficients are coded using DC VLC in a frame or VOP.
  • the ac_pred_flag may be disabled for all Intra macroblocks. Both these options are permitted by the syntax and so are supported by standard decoders, and can result in higher quality for the case of error in transmission. This improvement can be on the order of 0.1- 0.2dB in PSNR.
  • the DC coefficient of each 8x8 DCT block of an TNTRA macroblock can either be coded together with the 63 AC DCT coefficients, using what is known as an "INTER VLC” table, or separately, using what is known as an "TNTRA VLC” table.
  • the DC data bits are located before the DC marker (DCM), together with the header bits, while the data bits for AC coefficients are placed after the DC marker.
  • the ac_pred_flag is another option that indicates whether for a specific block of an TNTRA macroblock, the top row and first column DCT coefficients, are coded independently or differentially with respect to the neighboring blocks. To enhance error resilience it is preferable to set ac_pred_flag to 0.
  • error-correction is supported at the source level by using Forward Error Correction (FEC).
  • FEC Forward Error Correction
  • BCH Bose-Chaudhuri-Hocquenghem
  • Reed-Solomon are supported.
  • BCH is an error detection and correction technique based on Cyclic Redundancy Code.
  • n the number of information bits.
  • the BCH code has a minimum distance of at least 2t + 1.
  • Each binary BCH code (n, k, t) can correct up to t bit errors, and thus it is also referred to as a t-error-correcting code.
  • Different block sizes may be used.
  • a block size of 511 is used.
  • FEC is performed at a packetizer level of the elementary video bitstream, which can be considered as source-level error correction.
  • channel level error- correction introduces redundancy at the bit-level after multiplexing.
  • FEC can provide significant error resilience, at the cost of some bit budget.
  • Figure 11 illustrates an example graph of Forward Error Correction overhead vs. average BER correction capability. As illustrated, there is a close relation between FEC redundancy and error correcting capability, which is a strong indicator of error resilience. Preferably, at least double the expected BER is provided for.
  • FEC process rather then apply FEC to all packet data, a more efficient process is performed that reduces the number of error correction bits generated as compared to conventional approaches, while still providing significant error correcting capability.
  • One embodiment of the FEC process optionally generates FEC bits only for selected portions of the packets, and in particular, for those portions that are considered more essential or important for purposes of reproducing a frame sequence by the decoder.
  • the FEC process provides a systematic code, that is, the FEC correction or parity bits are separate from the original uncoded data bits. Thus, even if all the FEC bits are lost, the original selected portions of the packet are still potentially decodable.
  • the FEC data is encoded and transmitted in an MPEG-4 compliant manner as explained below.
  • the decoder will still be able to process the frame motion and texture data.
  • FEC is efficiently applied to important data, such as motion vectors, DC coefficients and header information, and FEC bits are not generated for unimportant or less important data.
  • This more important data may be located in a packet between a packet resync field and a motion marker.
  • the selected bits targeted for FEC coding are concatenated together with those from other frame packets and the FEC code bits are generated for the concatenated bits.
  • the resulting FEC bits are placed in an additional packet after the regular frame or VOP packets to ensure MPEG compatibility.
  • a packet identifier is stored in association with a corresponding value indicating how many bits and/or which bits where used to generate FEC bits.
  • this additional FEC packet further includes a user data identifier code, "user_data_start_code,” used to identify user defined data, and as such will be ignored by conventional decoders not equipped to process the FEC packet.
  • user_data_start_code used to identify user defined data, and as such will be ignored by conventional decoders not equipped to process the FEC packet.
  • the FEC packet will not be used by decoders equipped to handle the FEC bits. But when errors do occur, FEC decoding will help recover data that will allow for decoding even under severe error conditions.
  • embodiments of the present invention advantageously enable the transmission of video information even in low-bit rate, high noise environments.
  • embodiments of the present invention enable video transmission to be successfully performed over cellular networks and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Detection And Prevention Of Errors In Transmission (AREA)
  • Error Detection And Correction (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

The present invention is related to error resilient video encoding. Embodiments of the present invention provide systems (Figure 1B, 100B) and methods for reducing error propagation in a video data stream, for refreshing macroblocks, for reducing frame rates in a video data stream, for generating error correction information for a media stream, for performing bit rate allocation for a video data stream, and for detecting scene changes in a video data stream.

Description

SYSTEMS AND METHODS FOR ERROR RESILIENT ENCODING
Copyright Rights A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by any one of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
Background of the Invention Field of the Invention
[0001] The present invention is related to video and image coding and in particular to systems and methods for coding video image information in a compressed and error resilient manner. Description of the Related Art
[0002] MPEG is an ISO/IEC standard developed by MPEG (Moving Picture Experts Group). There are several versions of the MPEG standard, such as MPEG-1, MPEG-2, MPEG-4, and MPEG-7, and they are intended to standardize certain aspects of image and audio compression. As with other forms of video compression, such as H.261, H.262, H.263, H.263+, H.263++, H.26L, MPEG compression attempts to eliminate redundant or irrelevant data. For example, an MPEG encoder uses information from selected frames to reduce the overall video data that needs to be transmitted for certain other frames.
[0003] Typically, a video frame can be encoded in one of three ways, as an intraframe, as a predicted frame, and as a bi-directional frame. In addition, a video frame can also be skipped in order to reduce the resulting file size or bit-rate. An intraframe typically contains the complete image data for that frame and so does not rely on image data from other frames. Intraframe encoding provides the least compression. A predicted frame generally contains just enough information to allow a decoder to display the frame based on a recent preceding intraframe or predicted frame. This means that the predicted frame contains the data that relates to how the image has changed from the previous frame and residual error correction data. A bi-directional frame is generated from information from the surrounding intraframe(s) and/or predicted frames, including residual error correction data. Using data from the surrounding frames, the decoder uses interpolation to calculate the position and color of each pixel.
[0004] The MPEG-4 standard was developed for use with both low and high bit rate applications. For example, MPEG-4 has been enhanced for use in interactive video games, videoconferencing, videophones, interactive storage media, multimedia mailing, wireless multimedia and broadcasting applications. MPEG-4 provides for object scalability, improved error robustness and enhanced compression.
[0005] The ever-increasing demand for multimedia communications via the wired/wireless Internet faces the challenge of packet loss as well as bandwidth fluctuation. The dependency between image frames makes the compressed video stream vulnerable even to a small number of lost packets. MPEG-4 has therefore been particularly enhanced for use in low bit rate (< 64 kbs), error prone applications, such as mobile, wireless applications, and error-prone ATM (asynchronous transfer mode) network applications. Mobile operation tends to be more susceptible to transmission errors as there is often less data redundancy, in order to reduce bit rates, and greater sources of "noise." For example, wireless channels can be corrupted by environmental noise, and in the case of mobile applications, by burst noise resulting from multipath fading and shadowing caused by buildings and other structures. With respect to ATM network applications, cells can be lost due to network congestion and buffer overflow.
[0006] MPEG-4 has enhanced error resiliency as compared to previous versions of MPEG so that video data can be more successfully transmitted over such error prone networks. For example, one error resiliency technique provided for by the MPEG-4 standard is the use of resync markers in the video bit-stream. In particular, MPEG-4 has adopted fixed interval synchronization and specifies that video object plane (VOP) start codes and resynchronization markers (i.e., the start of a video packet) appear only at legal fixed interval locations in the bitstream. This helps to avoid the problems associated with start codes emulations. Through the use of resync markers included by an encoder in the video data, synchronization lost after an error can be regained by a decoder. [0007] Another error resiliency technique provided for by the MPEG-4 standard is the use of a reversible variable-length code. This code can be decoded even when read backwards, enabling a decoder to use uncorrupted information from a newly found resync marker back to the point in the data where the error occurred. Still another error resiliency technique adopted by MPEG-4 is data partitioning, used to separate motion information from texture information using a second resynchronization marker inserted between motion and texture information. Thus, if there is an error and the texture information is undecodable or lost, the decoder can utilize the motion information to conceal the error by using the motion information to compensate the previous decoded frame or VOP.
[0008] However, despite the use of the enhancements described above, many MPEG-4 encoders and decoders fail to provide sufficient error resiliency as is often desired in error-prone applications, such as in cellular phone applications. Thus, the transmission of MPEG-4 compliant video streams over cellular networks often results in unrecoverable corrupted data and the significant degradation in the quality of the video data seen by a recipient. Such video degradation can make video communication over error-prone networks undesirable for a user's perspective, and disadvantageously impedes the adoption and use of video transmissions over error-prone networks.
[0009] Further, to meet target bit rates, conventional encoders drop frames to reduce the frame rate according to a simple skipping algorithm. For example, a conventional encoder will drop every 4 of 5 frames in a video clip to convert the video clip from a 30 frames per second rate to a 6 frames per second rate. However, this simple form of skipping often has a significant adverse impact on the visual quality when decoded.
Summary of the Invention
[0010] The present invention is related to video encoding and in particular to systems and methods for encoding video information for transmission in a compressed manner and/or an error resilient manner. Embodiments of the present invention advantageously enable the transmission of video information even in low-bit rate, high noise environments. For example, embodiments of the present invention enable video transmission to be successfully performed over cellular networks and the like.
[0011] In one embodiment of the present invention, enhanced error resilience is provided by automatically inserting two consecutive intracoded I-frames upon a scene change. By including two intracoded frames upon a scene change, error propagation is reduced on the decoder-side if one of the two sequential intracoded frames is lost or undecodable.
[0012] By way of example, in one embodiment a method of enhancing error resiliency in an encoded video transmission is provided, the method comprising: receiving an indication that a first frame corresponds to a scene change; intracoding the first frame at least partly in response to receiving the indication that the first frame corresponds to a scene change; automatically intracoding the next consecutive frame; and transmitting the intracoded first frame and the intracoded next consecutive frame.
[0013] In another embodiment, a video coding apparatus is provided, comprising: a first instruction stored in a processor readable memory, the first instruction configured to receive an indication that a first frame is to be intracoded; a second instruction stored in processor readable memory, the second instruction configured to intracode the first frame at least partly in response to the indication; and a third instruction stored in processor readable memory, the third instruction configured to automatically intracode the frame immediately after the first frame as a result of the indication that the first frame is to be intracoded.
[0014] In still another embodiment, a method of encoding video frames is provided, the method comprising: receiving a first instruction to intracode a first frame; intracoding the first frame in response to the first instruction; and intracoding a second frame as a result of the first instruction to intracode the first frame.
[0015] In yet another embodiment, an encoding apparatus is provided, comprising: a means for receiving a first indication to intracode a first frame; a means for intracoding the first frame in response to the first instruction; and a means for intracoding a second frame as a result of the first instruction to intracode the first frame.
[0016] An enhanced Adaptive Intra Refresh (AIR) process provides efficient error resiliency by selectively and adaptively encoding macroblocks in a predicted frame. In one embodiment, an adaptive motion area process is performed to determine which macroblocks are to be intracoded. Adaptive motion change detection can efficiently reduce the large propagation error, including errors occurring in the motion area. An intercode distortion value and an intracode distortion value are calculated, as are an intercode bit quantity and an intracode quantity. Based on a comparison of the calculated intercode distortion value and the intracode distortion value, and on a comparison of the intercode bit quantity and the intracode bit quantity for each macroblock, a decision is made as to which predicted frame macroblocks are to be intracoded.
[0017] One embodiment of the present invention is a method of performing adaptive intra refresh, the method comprising: receiving a packet loss probability value; receiving a motion vector for a first macroblock in a first frame; mapping the motion vector onto portions of a plurality of other macroblocks in a previous frame; calculating at least a first transition factor value based at least in part on the mapping; calculating an estimated inter distortion value for the first macroblock based on at least the packet loss probability value and the first transition factor value; calculating an estimated intra distortion value for the first macroblock based on at least the packet loss probability value; receiving a first bit quantity value corresponding to a bit quantity used to intracode the first macroblock; receiving a second bit quantity value corresponding to a bit quantity used to intercode the first macroblock; and based at least in part on the estimated inter distortion value, the estimated intra distortion value, the first bit quantity value, and the second bit quantity value, transmitting the first macroblock as one of an intra macroblock and an inter macroblock.
[0018] Another embodiment of the present invention is a method of selectively intracoding macroblocks, the method comprising: receiving a packet loss probability value; receiving a motion vector for a first macroblock in a first frame; based at least in part on the motion vector, determining which portions of macroblocks in a previous frame would be used in predicting the first macroblock; calculating at least a first propagation strength value based at least in part on determining which portions of macroblocks in the previous frame would be used in predicting the first macroblock; calculating an estimated inter distortion value for the first macroblock based on at least the packet loss probability value and the first propagation strength value; calculating an estimated intra distortion value for the first macroblock based on at least the packet loss probability value; calculating a quantization distortion value for the first macroblock; receiving a first bit quantity value corresponding to a bit quantity used to intracode the first macroblock; receiving a second bit quantity value corresponding to a bit quantity used to intercode the first macroblock; and based at least in part on the estimated inter distortion value, the estimated intra distortion value, the quantization distortion value, the first bit quantity value, and the second bit quantity value, providing for transmission the first macroblock as one of an intra macroblock and an inter macroblock.
[0019] Still another embodiment of the present invention is a method of selectively intracoding macroblocks in a plurality of macroblocks in a first frame, the method comprising: receiving a packet loss probability value; receiving a corresponding motion vector for each macroblock in the plurality of macroblocks; based at least in part on the corresponding motion vector, determining which portions of macroblocks in a previous frame would be used in predicting said each macroblock; calculating for each of said macroblocks at least a first corresponding propagation strength value based at least in part on determining which portions of macroblocks in the previous frame would be used in predicting said each macroblock; calculating for each of said macroblocks an estimated inter distortion value based upon at least the packet loss probability value and the corresponding at least first propagation strength value; calculating for each of said macroblocks an estimated intra distortion value based upon at least the packet loss probability value; calculating for each of said macroblocks a quantization distortion value; and based at least in part on the estimated inter distortion, the estimated intra distortion, and the quantization distortion values for each macroblock in the plurality of macroblocks, designating a subset of the plurality of macroblocks to be intracoded.
[0020] One embodiment of the present invention is a circuit configured to selectively intracode macroblocks, the circuit comprising: a first instruction configured to receive a packet loss probability value; a second instruction configured to receive a motion vector for a first macroblock in a first frame; a third instruction configured to determine, based at least in part on the motion vector, which portions of macroblocks in a previous frame would be used in predicting the first macroblock; a fourth instruction configured to calculate at least a first propagation strength value based at least in part on the determination of which portions of macroblocks in the previous frame would be used in predicting the first macroblock; a fifth instruction configured to calculate an estimated inter distortion value for the first macroblock based on at least the packet loss probability value and the first propagation strength value; a sixth instruction configured to calculate an estimated intra distortion value for the first macroblock based on at least the packet loss probability value; and a seventh instruction configured to selectively provide for transmission the first macroblock as one of an intra macroblock and an inter macroblock based at least in part on the estimated inter distortion value and the estimated intra distortion value.
[0021] Embodiments of the present invention provide for adaptive skipping of frames in an encoding process based on a defined cost function. The visual quality, particularly in scene change areas, is improved while scenes are efficiently encoded. One embodiment of the skipping process is iterative in that a frame is selectively dropped from the sequence of frames by computing a mean of absolute differences between the frames adjacent to the frame of interest, and by weighting the computation with a temporal parameter, wherein the frame having the least impact on the scene is dropped. This procedure is repeatedly iterated until a target frame rate, which is related to the desired bit rate and frame size, is achieved. Other measurement techniques, such as a sum of the mean absolute differences (SMAD) or sums of root mean squared (SRMS) values, can be used in place of or in addition to the mean of absolute differences technique.
[0022] One embodiment of the present invention is a method of selecting which video sequence frames to skip during an encoding process, the method comprising: receiving a sequence of frames; for each frame in at least a portion of the sequence of frames, determining a corresponding mean of absolute differences value weighted by a temporal parameter between frames bracketing said each frame; and at least partly based on the corresponding mean of absolute differences value weighted by a temporal parameter calculated for each frame, skipping a first frame in the sequence of frames.
[0023] Another embodiment of the present invention is a method of determining which frames to skip in a video sequence, the method comprising: receiving a first frame rate goal; selecting a first potential frame to be skipped, the first potential frame between a first frame and a last frame in a video sequence scene; calculating a first cost value for the first potential frame to be skipped, the first cost value related to the visual impact that would result from skipping the first potential frame; storing the first cost value in computer readable memory; selecting a second potential frame to be skipped, the second potential frame between the first frame and the last frame; calculating a second cost value for the second potential frame to be skipped; storing the second cost value in computer readable memory; selecting a third potential frame to be skipped, the third potential frame between the first frame and the last frame; calculating a third cost value for the third potential frame to be skipped; storing the third cost value in computer readable memory; determining which one of the first, second, and third cost values has the least cost; skipping the one of the one of the first, second, and third potential frames corresponding to the least cost value; determining if the first frame rate goal is met; and at least partly in response to determining the first frame rate goal is not met, recalculating cost values associated with frames remaining from the first, second and third potential frames, and skipping the one of the remaining frames having the least recalculated cost value.
[0024] Still another embodiment of the present invention is a method of selecting image frames to be skipped during an encoding process, the method comprising: receiving a sequence of frames including a first frame; calculating a first cost function value for the first frame, wherein the cost function includes a first temporal parameter related to a time difference and luminosity differences between frames bounding the first frame; calculating a second cost function value for a second frame, wherein the cost function includes a second temporal parameter related to a time difference between frames and luminosity differences bounding the second frame; and selecting for skipping one of the first frame and the second frame based at least in part on the first cost function value and the second cost function value.
[0025] Yet another embodiment of the present invention is a processor circuit configured to select video frames to skip in a sequence of frames, the processor circuit comprising: a first instruction configured to calculate for each frame in at least a portion of the sequence of frames a corresponding mean of absolute differences value weighted by a temporal parameter between frames bracketing said each frame in the portion of the sequence of frames; and a second instruction configured to designate a first frame to be skipped based at least in part on the corresponding mean of absolute differences value weighted by the temporal parameter.
[0026] One embodiment of the present invention is a frame rate control apparatus, comprising: a first instruction configured to receive a frame rate target; a second instruction configured to calculate a first cost function value for a first frame in a sequence of frames, wherein the first cost function value is based at least in part on a first temporal parameter related to a time difference and luminosity differences between frames in the sequence of frames bounding the first frame; a third instruction configured to calculate a second cost function value for a second frame in the sequence of frames, wherein the second cost function based at least in part on a second temporal parameter related to a time difference and luminosity differences between frames in the sequence of frames bounding the second frame; a fourth instruction configured to select for skipping one of the first frame and the second frame based at least in part on the first cost function value and the second cost function value; and a fifth instruction configured to determine if the target frame rate is met with the selected one of the first frame and the second frame skipped.
[0027] In one embodiment, error resiliency is enhanced using forward error correction (FEC) information. FEC coding is efficiently and selectively applied in real-time to important data, such as motion vectors, DC coefficients and header information, rather then generating FEC bits for unimportant or less important data. This selected important data may be located in a packet between a packet resync field and a motion marker. In particular, for a given frame or VOP, the selected packet bits targeted for FEC coding are concatenated together and the FEC code bits are generated for the concatenated bits. Optionally, the resulting FEC bits are placed in an additional packet after the regular frame or VOP packets to ensure MPEG compatibility.
[0028] One embodiment of the present invention is a method of providing forward error correction (FEC) on a plurality of frame packets, the method comprising: concatenating selected portions of packet data corresponding to a plurality of frame packets for a first frame; generating forward error correction bits for the concatenated selected portions of packet data; and transmitting the forward error correction bits in a separate packet identified with a user data identifier code or the like, including other unique identifier codes to be assigned in the future by MPEG-standards committee and the like.
[0029] Another embodiment of the present invention is an error correction generation circuit, comprising: a first instruction stored in processor readable memory configured to generate forward error correction data for selected portions of packet data that are to be transmitted in a corresponding plurality of frame packets; a second instruction stored in processor readable memory configured to store the forward error correction data in a first packet separate from the plurality of frame packets; and a third instruction stored in processor readable memory configured to identify the first packet with a first data identifier code.
[0030] Still another embodiment of the present invention is an encoder circuit, comprising: a means for generating forward error correction data for selected portions of packet data from a plurality of frame packets; a means for storing the forward error correction data in a first packet separate from the plurality of frame packets; and a means for identifying the first packet with a first data identifier code.
[0031] Further, embodiments of the present invention provide for using a Header Extension Code (HEC) in a sequence of video packets or in every video packet, and not just on the first video packet following the VOP header as with conventional encoders. This better ensures that even if a packet is lost or corrupted, subsequent packets can still be decoded and used. Further, even many conventional decoders will be able to handle the inclusion of the enhanced use of HECs.
[0032] Optionally, a Video-Object-Layer (VOL) header has a flag set indicating that a fixed Video Object Plane (VOP) increment is to be used, followed by the fixed time increment value. This will facilitate the decoder's detection of missing frames, that is, frames either skipped by the encoder in order to achieve higher compression or lost during transmission.
[0033] Embodiments of the present invention provide bit allocation on a scene level, on a frame level, and/or on a macroblock level. A bit allocation module determines an appropriate distribution of a fixed bit budget. In one embodiment, a weight is assigned to a scene based on the number of intracoded frames and the number of intercoded frames. Intracoded frames are weighted more heavily than intercoded frames to account for the greater number of bits needed to encode an intraframe. The bit allocation module distributes the fixed bit budget within a scene by comparing the current bit usage and the target bit usage and based on the comparison, adjusts a quantization parameter or step size for the current frame.
[0034] To further increase coding efficiency, an encoding quantization step or quantization parameter (QP) is dynamically adjusted. This provides for fast convergence to a desired bit budget, even for short scenes or Groups of Video Object Planes (GOVs). In addition, after encoding each scene or GOV, the QP for I-frames is dynamically adjusted based on the actual bit usage of the previous scene or GOV. This QP adaptation of I-frame can be very useful for achieving or coming close to achieving the bit budget when there are high motion segments during a scene sequence.
[0035] One embodiment of the present invention is a bit allocation method for use with video sequences, the method comprising : receiving a clip bit budget for at least a first clip; determining a scene quantity for the first clip; determining a quantity of predicted frames in a first scene in the first clip; and calculating a bit budget for the first scene based at least in part on the scene quantity for the first clip, the first clip bit budget, a quantity of intracoded frames in the first scene, and the quantity of predicted frames in the first scene.
[0036] Another embodiment of the present invention is a bit allocation method for use with video clip scenes, the method comprising : calculating a bit budget for a first scene; calculating bit budgets for corresponding frames, including at least a first frame, within the first scene; and calculating bits budgets corresponding to macroblocks within the first frame.
[0037] Still another embodiment of the present invention is an apparatus for allocating bits, comprising: a first instruction configured to assign a first bit budget for a first scene based at least in part on how many intracoded frames are in the first scene, a complexity determination for the first scene, and how many intercoded frames are in the first scene; and a second instruction configured to allocate bit budgets to frames within the first scene based at least in part on a current bit usage and a target bit usage.
[0038] Yet another embodiment of the present invention is a bit budgeting apparatus, comprising : a first instruction configured to receive a bit budget for at least a first video sequence; a second instruction configured to determine a scene quantity for the first video sequence; a third instruction configured to determine a quantity of predicted frames in a first scene in the first video sequence; and a fourth instruction configured to determine a bit budget for the first scene based at least in part on the scene quantity for the first clip, the first clip bit budget, a quantity of intracoded frames in the first scene, and the quantity of predicted frames in the first scene.
[0039] Embodiments of the present invention provide accurate systems and methods for locating scene changes. Because scene change frames are intracoded, which generally utilizes more bits then intercoding while providing greater error resiliency, the accurate determination of scene change frames is important in ensuring that error resiliency is efficiently performed. In one embodiment, a first root mean squared (RMS) value is calculated for a first frame relative to a second frame. A second temporal derivative RMS value is calculated for the first frame relative to the second frame, and the second frame relative to a third frame. Based at least in part on the second temporal derivative RMS value, the second frame is designated as a scene change frame. Frames are adaptively grouped based on the scene change detection. As discussed below, other criteria, based on the mean of absolute differences (MAD), and/or locating RMS local maxima, and/or locating MAD local maxima, can be used to detect a scene change.
[0040] One embodiment of the present invention is a method of detecting a scene change in a digital video sequence having a plurality of frames, the method comprising: calculating a first root mean squared (RMS) value for a first frame relative to a second frame and the second frame relative to a third frame; calculating a first mean absolute difference (MAD) value for the first frame relative to the second frame; determining if the first RMS value meets a first criterion; determining if the first MAD value meets a second criterion; and designating the second frame as a scene change frame at least partly in response to determining that both the first RMS value meets the first criterion and the first MAD value meets the second criterion.
[0041] Another embodiment of the present invention is a method of detecting a scene change in a digital video sequence, the method comprising: calculating a second temporal derivative RMS value for a first frame relative to a second frame and the second frame relative to a third frame; and based at least in part on the second derivative value, determining that the second frame is a scene change frame.
[0042] Still embodiment of the present invention is an apparatus for identifying a scene change in a video sequence, the apparatus comprising: a first instruction stored in processor readable memory, the first instruction configured to calculate a first root mean squared (RMS) value for a first portion of a video sequence relative to a second portion of the video sequence; a second instruction stored in processor readable memory, the second instruction configured to calculate a second temporal derivative RMS value; and a third instruction configured to cause the second portion of the video sequence to be intracoded based at least in part on the second derivative RMS value.
[0043] One embodiment of the present invention is method of determining which portions of a video sequence are to be intracoded, the method comprising: calculating a first root mean squared (RMS) value for a first portion of the video sequence; calculating a first mean absolute difference (MAD) value for the first portion of the video sequence; determining if the first RMS value meets a first criterion; determining if the first MAD value meets a second criterion; determining if the first MAD value meets a third criterion; and causing an intracoding operation to be performed at least partly in response to at least two of the first, second and third criteria being met. [0044] Another embodiment of the present invention is a scene change detection apparatus, comprising: an RMS circuit having a first frame information input and an RMS output, the RMS circuit configured to provide at the RMS output a value corresponding to root mean squared differences information between at least two frames based on frame information received on the first frame information input; a MAD circuit having a second frame information input and a MAD output, the MAD circuit configured to provide at the MAD output a value corresponding to mean absolute differences information between at least two frames based on frame information received on the second frame information input; and an evaluator circuit coupled to the RMS output and the MAD output, the evaluator circuit configured to detect a scene change frame based at least on the value corresponding to root mean squared differences information and the value corresponding to mean absolute differences information, and to provide a scene change designation.
Brief Description of the Drawings [0045] Embodiments of the present invention will now be described with reference to the drawings summarized below. These drawings and the associated description are provided to illustrate example embodiments of the invention, and not to limit the scope of the invention.
[0046] Figure 1A illustrates an example networked system for implementing a video distribution system.
[0047] Figures 1B-C illustrate an example encoder architecture in accordance with an embodiment of the present invention.
[0048] Figures 2A-B illustrate an example refresh map and an example scan order that can be used with an embodiment of the present invention.
[0049] Figure 3 illustrates an example analysis of a video sequence used to locate a scene change.
[0050] Figures 4A-4B illustrate an example of adaptive frame skipping in accordance with an embodiment of the present invention.
[0051] Figure 5 illustrates an example use of second order motion compensation.
[0052] Figure 6 illustrates an example packetized bitstream. [0053] Figure 7 illustrates an example use of consecutive I-frames in accordance with an embodiment of the present invention
[0054] Figures 8A-H illustrate example processes for adaptive intra refresh.
[0055] Figure 9 illustrates an example rate control process in accordance with an embodiment of the present invention.
[0056] Figure 10 illustrates an example scene level recursive bit allocation process.
[0057] Figure 11 illustrates an example graph of Forward Error Correction overhead vs. average BER correction capability.
Detailed Description of Preferred Embodiments
[0058] The present invention is related to video encoding and in particular to systems and methods for encoding video information for transmission in a compressed and/or an error resilient manner. Thus, embodiments of the present invention advantageously enable the transmission of video information even in low-bit rate, noise, error-prone environments. Embodiments of the present invention can be used with a variety of video compression standards, such as, by way of example, the MPEG-4 standard, as well as MPEG-1, MPEG-2, H.261, H.262, H.263, H.263+, H.263++, and H.26L, and video standards yet to be developed. Aspects of the MPEG-4 standard are defined in "Coding of Audio-Visual Objects: Systems," 14496-1, ISO/TEC JTC1/SC29/WG11 N2501, November 1998, and "Coding of Audio-Visual Objects: Visual," 14496-2, ISO/IEC JTC1/SC29/WG11 N2502, November 1998, and the MPEG-4 Video Verification Model is defined in "MPEG-4 Video Verification Model 17.0", ISO/TEC JTC1/SC29/WG11 N3515, Beijing, China, July 2000.
[0059] Figure 1A illustrates a networked system for implementing a video distribution system in accordance with one embodiment of the invention. An encoding computer 102 receives a video signal, which is to be encoded to a relatively compact and robust format. The encoding computer 102 can correspond to a variety of machine types, including general purpose computers that execute software and to specialized hardware. The encoding computer 102 can receive a video sequence from a wide variety of sources, such as via a satellite receiver 104, a video camera 106, and a video conferencing terminal 108. The video camera 106 can correspond to a variety of camera types, such as video camera recorders, Web cams, cameras built into wireless devices, and the like. Video sequences can also be stored in a data store 110. The data store 110 can be intemal to or external to the encoding computer 102. The data store 110 can include devices such as tapes, hard disks, optical disks, and the like. It will be understood by one of ordinary skill in the art that a data store, such as the data store 110 illustrated in Figure 1A, can store unencoded video, encoded video, or both. In one embodiment, the encoding computer 102 retrieves unencoded video from a data store, such as the data store 110, encodes the unencoded video, and stores the encoded video to a data store, which can be the same data store or another data store. It will be understood that a source for the video can include a source that was originally taken in a film format.
[0060] The encoding computer 102 distributes the encoded video to a receiving device, which decodes the encoded video. The receiving device can correspond to a wide variety of devices that can display video. For example, the receiving devices shown in the illustrated networked system include a cell phone 112, a personal digital assistant (PDA) 114, a laptop computer 116, and a desktop computer 118. The receiving devices can communicate with the encoding computer 102 through a communication network 120, which can correspond to a variety of communication networks including a wireless communication network. It will be understood by one of ordinary skill in the art that a receiving device, such as the cell phone 112, can also be used to transmit a video signal to the encoding computer 102.
[0061] The encoding computer 102, as well as a receiving device or decoder, can correspond to a wide variety of computers. For example, the encoding computer 102 can be a microprocessor or processor (hereinafter referred to as processor) controlled device, including, but not limited to a terminal device, such as a personal computer, a workstation, a server, a client, a mini computer, a main-frame computer, a laptop computer, a network of individual computers, a mobile computer, a palm top computer, a hand held computer, a set top box for a TV, an interactive television, an interactive kiosk, a personal digital assistant, an interactive wireless communications device, a mobile browser, a Web enabled cell phone, a personal digital assistant (PDA) or a combination thereof. By way of example, an encoder computer may also be included in the camera 106, the cell phone 112, the PDA 114, the laptop computer 116, and/or the desktop computer 118. The computer 102 may further possess input devices such as a keyboard, a mouse, a trackball, a touch pad, or a touch screen and output devices such as a computer screen, printer, speaker, or other input devices now in existence or later developed.
[0062] The encoding computer 102, as well as a decoder computer, can correspond to a uniprocessor or multiprocessor machine. Additionally, the encoder and decoder computers can include an addressable storage medium or computer accessible medium, such as random access memory (RAM), an electronically erasable programmable read-only memory (EEPROM), masked read-only memory, one-time programmable memory, hard disks, floppy disks, laser disk players, digital video devices, Compact Disc ROMs, DVD-ROMs, other optical media, video tapes, audio tapes, magnetic recording tracks, electronic networks, and other techniques to transmit or store electronic content such as, by way of example, programs and data. In one embodiment, the encoding and decoding computers are equipped with a network communication device such as a network interface card, a modem, Infra-Red (IR) port, a wireless network interface, or other network connection device suitable for connecting to a network. Furthermore, the computers execute an appropriate operating system, such as Linux, Unix, Microsoft® Windows® 3.1, Microsoft® Windows® 95, Microsoft® Windows® 98, Microsoft® Windows® NT, Microsoft® Windows® 2000, Microsoft® Windows® Me, Microsoft® Windows® XP, Apple® MacOS®, IBM® OS/2®, Microsoft® Windows® CE, or Palm OS®. As is conventional, the appropriate operating system may advantageously include a communications protocol implementation, which handles all incoming and outgoing message traffic passed over the network, which can include a wireless network. In other embodiments, while the operating system may differ depending on the type of computer, the operating system may continue to provide the appropriate communications protocols necessary to establish communication links with the network.
[0063] Figure IB illustrates an example encoding system 100B in accordance with an embodiment of the present invention. The term encoding system, as used herein, includes one or more encoders. The encoding system 100B comprises, by way of example, one or more of processors, program logic, or other substrate configurations representing data and instructions, which operate as described herein. In other embodiments, the encoding system 100B can comprise controller circuitry, integrated circuits, gate arrays, application specific circuits, processor circuitry, processors, general purpose single-chip or multi-chip microprocessors, digital signal processors, embedded microprocessors, microcontrollers and the like, executing software code, including instructions and data stored in computer readable memory. By way of example and not limitation, the encoding system 100B can be housed in one or more leaded, leadless, or ball grid array semiconductor packages, on one or more circuit boards, and/or using one or more hybrid packages. All or portions of the encoding system 100B may be included in a fixed terminal, such as a desktop computer, or in a portable terminal, such as a cellular phone, portable computer, personal digital assistant, video camera, or the like. The encoding system 100B can, in an example embodiment, correspond to the encoding computer 102. By way of further example, an encoding system in accordance with the present invention can be used to conduct video conferencing, to aid in the storage and transmission of movies or other images, and the like.
[0064] The encoding system 100B encodes and compresses video information for transmission to a decoder. The encoding system 100B includes a preprocessing module or circuit 102B, a bit allocation module or circuit 104B, and an encoder module or circuit 106B. The preprocessing module or circuit 102B, including a video sequence analyzer, is used to detect when a scene change has taken place and to determine how a given frame, VOP or picture, is to be encoded.
[0065] A video object layer contains a sequence of 2D representations of arbitrary shape at different time intervals that is referred to in MPEG-4 as a video object plane (VOP). Each of the VOP regions can be non-rectangular and may correspond to particular image or video content of interest, such as physical objects within a scene. Video object planes (VOPs) are divided into macroblocks of size 16x16. A macroblock is encoded in six blocks, four for luminosity and two for chromaticity, of size 8x8. To obtain a macroblock structure from an arbitrary shaped VOP, the bounding box of the VOP is calculated and extended to multiples of the macroblock size.
[0066] However, for most current applications, and in particular for wireless applications using the so-called "simple profile," there is generally only 1 VOP per frame, which is a rectangular VOP. For clarity, the term frame, as used herein, can also include a VOP, such as an MPEG-4 VOP, or a picture. Similarly, the term VOP, as used herein, can also refer to a frame. In MPEG-4, the VOPs can be structured in groups of video object planes (GOV). Using MPEG-2 terminology, frames or pictures can be arranged in groups of pictures (GOPs). For clarity, the term "scene" as used herein, may also refer to a GOV or a GOP and visa versa.
[0067] A frame or video object may be encoded as an intracoded frame (an "I- frame" or "I- VOP"), as a predicted frame (a "P-frame" or "PNOP"), or as a bi-directional frame (a "B-frame" or "BNOP"). MPEG-1 also provides for a D-frame. A D-frame is a frame that has no motion vectors, so that a zero vector is assumed, and has texture DCT data. To exploit spatial redundancy, a Discrete Cosine Transformation (DCT) is performed on the encoded frames and the resulting coefficients are quantized.
[0068] The MPEG-4 simple profile does not support B-frames or B-VOPs. However, the simple profile does support frame skipping. A video frame can be skipped in order to reduce the resulting file size or bit-rate. Because the MPEG-4 simple profile does not support B-frames or D-frames, the following discussions will not focus on such frames. Nonetheless, embodiments of the present invention can be used with B-frames and D- frames in accordance with other profiles and other standards.
[0069] The term frame can correspond to either an interlaced frame or to a noninterlaced frame, i.e., a progressive frame. In an interlaced frame, each frame is made of two separate fields, which are interlaced together to create the frame. Such interlacing is not performed in a non-interlaced or progressive frame. While illustrated in the context of non-interlaced or progressive video, one or ordinary skill in the field will appreciate that the principles and advantages described herein are applicable to both interlaced video and noninterlaced video. In addition, while embodiments of the invention are described in the context of MPEG-4, aspects of the principles and advantages described herein are also applicable to other video standards, including, by way of example, MPEG-1, MPEG-2, H.261, H.262, H.263, H.263+, H.263++, and H.26L, as well as video standards yet to be developed.
[0070] An intracoded I-frame typically only includes information from the image itself and thus an I-frame can be decoded independently of other frames. P and B frames are also referred to as intercoded frames because they are encoded based on data from other frames. The preprocessing module 102 generates a file, referred to as an input frame-type file, containing the frame-type designations corresponding to the frames. In other embodiments, the frame-type information is passed to other portions of the encoding system 100B using variables and the like. While the preprocessing module 102B is illustrated in Figure IB as being included in the encoding system 100B, the preprocessing module 102B can be physically separate from the other portions of the encoding system 100B. In such an embodiment, the preprocessing module 102B can produce a text file that includes frame-type designation that is then input by the remainder of the encoding system 100B.
[0071] Many standard MPEG-4 encoders can only handle one scene, that is, one I-frame followed by P-frames or B-frames, or they introduce regular I-frames every k frames, as is commonly done in MPEG-2 encoding. These approaches make implementation of the encoder simpler, however they place the burden on the user to determine how clips of multiple scenes are to be handled. To adequately improve the coding efficiency, the number of I-frames should be reduced or minimized. In the absence of error conditions, I-frames are preferably used in scene-changes only. Thus, it is advantageous to correctly and accurately detect scene changes.
[0072] An example scene change detection process in accordance with an embodiment of the present invention will now be described. In the example embodiment, the process operates on YUV-4:2:0 files and produces a text file as an output. In the example embodiment, YUV-4:2:0 files are header-less files with concatenated frames, where, for each frame, the (luminosity) Y-pixel values are provided first, followed by the (Chromaticity-blue) Cb-values, and then the (Chromaticity-red) Cr-values. The term "4:2:0" indicates that chromaticity values are subsampled by a factor 4 with respect to luminosity. In particular, if the size of a frame (in pixels) is W x H (W: width, H: height), there are W*H Y-values (1 for each frame pixel), (W/2)*(H/2) Cb-values and (W/2)*(H/2) Cr-values for each frame. That gives a total of 3*W*H/2 bytes as the frame buffer size needed to store a frame of size W x H. The subsampling for chromaticity components is achieved by subsampling along the vertical and horizontal dimensions by a factor of 2. Thus, a 2 x 2 block has 4 luminosity values, and 1 chromaticity-blue and 1 chromaticity- red. In other embodiments, other formats for storing image data can be used.
[0073] The preprocessing module 102B will now be described in greater detail. The preprocessing module 102B performs frame evaluation and encoding designation. As will be described below, each frame is designated by the preprocessing module 102B as an I-frame, a P-frame, or as a skipped frame. In other embodiments, the preprocessing module 102B may also designate frames as B-frames or D-frames. B-frame encoding may be performed if there is sufficient computational power, available bandwidth (B-frames take much more bandwidth than skipped frames), and if allowed by the corresponding standard. For example, the MPEG-4 simple-profile syntax, used in wireless networks, does not allow for B-frames. The example file format generated by the preprocessing module 102B includes a line per input frame, with a frame-type designation character on each line: 0, 1 or 2. A "0" indicates an I-frame, a "1" indicates a P-frame, and a "2" indicates a skipped frame. In other embodiments, designations can be provided for a bidirectional frame and a D-frame.
[0074] As previously discussed, scene change frames are generally intracoded. To locate the scene change frames, the preprocessing module's scene change analysis performs a color-weighted Root Mean Squared (RMS) calculation and a Mean Absolute Differences (MAD) calculation between the i th frame E, and the k th frame Fk . The RMS can be defined as:
Equation 1
Figure imgf000021_0001
[0075] where F(x, y) denotes the (x, y) . th pixel in frame F, and w and h are the width and height of the frame, respectively. Y(x, y) indicates the luminance value, while U(x, y) and V(x, y) are the two chromaticity components. The coefficients α, β and γ are weighting coefficients for the luminosity, chromaticity-blue and chromaticity-red components correspondingly. To ease computations, the weighting coefficients can be fixed. For example, the weighting coefficients can be set as follows: α = β = γ = 1.
[0076] The Mean Absolute Difference (MAD) measure can be defined as follows:
MAD(F, ,Fk) (x, y) - Yk (x, y)\ Equation 2
Figure imgf000021_0002
[0077] In this example, the MAD does not need to include the two chromaticity components. [0078] If MAD(Fl ,Fk) and/or RMS(Fl,Fk) are large or greater than a selected criteria, this indicates that the content of E, is substantially different from Fk . Thus in one embodiment, if the MAD between consecutive frames, MAD(Fl_l , E, ) , is larger than a pre- specified threshold, E, is designated a scene change frame. An example threshold value for designating a scene change frame is approximately 25.
[0079] Optionally, a second temporal derivative of the RMS can be used to determine if a frame is scene change frame, as follows:
^^(t) = R S(E,.1 ,E, )- 2R S(E, ,E,+1)-f R S(E,+1,E,+2) dt
Equation 3
[0080] As defined by Equation 3, the second temporal derivative of the RMS is based on the RMS value for the previous frame F,-ι relative to the current frame F„ the RMS value of the current frame F, relative to the next frame F,+1, and the RMS value of the next frame F,+ι to the subsequent frame F1+2.
[0081] The second temporal derivative of the RMS value will be negative with relatively high amplitude when Ft is a scene-change frame, as illustrated in Figure 3. Thus, if the absolute value of the second temporal derivative of the RMS value is larger than a pre-specified threshold, E, is designated a scene change frame. As illustrated in Figure 3, there is a correlation between the RMS values, indicated by the diamonds, and the second derivative of RMS, indicated by the triangles. Thus, both the RMS values the values of the second derivative of RMS generally provide a correct indication of a scene change. An example second derivative of RMS threshold value for determining a scene change is -6.5.
[0082] While the second derivative of the RMS is a good peak detector, it is somewhat sensitive to noise. To better increase the accuracy of the scene change determination, in one embodiment only if both the temporal activity measures of MAD and the second-order derivative of RMS indicate that the corresponding thresholds are met or passed, then a frame is designated a scene change frame. As will be described below, scene changes frames will be intracoded as I-frames or I-VOPs.
[0083] In particular, in one example a frame is designated as a scene change, and thus will be coded in INTRA mode, when its MAD is greater than 20 and the second derivative of RMS is negative and has an absolute value of greater than 4. In another example, a frame is designated as a scene change, and thus will be coded in TNTRA mode, when its RMS is greater than 40 and/or when the second derivative of RMS is negative and has an absolute value of greater than 8. In other embodiments other thresholds can be used. Alternatively or in addition, a second derivative of MAD can be used, as similarly described above with respect to the second derivative of RMS, as a further indication of whether a frame corresponds to a scene change or not.
[0084] An additional criterion can be used to determine when a scene change has occurred. For example, in one embodiment, a determination is made as to whether the MAD value is a local maximum, that is, has increased from a previous frame to the frame at issue, and then decreased from the frame at issue to the next frame. If so, this indicates that it is likely the frame at issue is a scene change frame and should be intracoded. In addition, a similar determination may be made for the RMS value. For example, a determination is made as to whether the RMS value is a local maximum, that is, has increased from a previous frame to the frame at issue, and then decreased from the frame at issue to the next frame. If so, this too indicates that it is likely the frame at issue is a scene change frame and should be intracoded.
[0085] Optionally, a voting process can be used, wherein if at least two of the RMS, the second derivative of the RMS, and the MAD, meet corresponding criteria, then a frame is designated as a scene change that is to be intracoded. In another embodiment, if the RMS and second derivative of the RMS meet the corresponding criteria, and if the MAD is a local maximum, then the frame is designated as a scene change frame. In still another embodiment, if the RMS and MAD meet the corresponding criteria, and if the MAD is a local maximum, then the frame is designated as a scene change frame.
[0086] To further improve coding efficiency and to meet target bit rates, the number of frames that needs to be encoded per second is preferably reduced as much as acceptable. One technique used to reduce the number of frames encoded per second is to skip-frames in the encoding process. Two example frame-skipping techniques are fixed frame skipping and adaptive frame skipping. Conventional encoders drop frames to reduce the frame rate according to a simple skipping algorithm. For example, a conventional encoder will drop every 4 of 5 frames in a video clip to convert the video clip from a 30 frames per second rate to a 6 frames per second rate. [0087] As will be discussed below, fixed frame skipping tends to provide better error resiliency in noisy environments, while adaptive frame skipping tends to provide better visual results in low noise environments. Optionally, the preprocessing module 102B calculates, based on the bit-rate/frame-rate formula defined in Equation 7 below, the target encoding frame-rate and then switches between adaptive and fixed skipping in order to meet a target encoding frame rate.
[0088] In fixed frame skipping, the input video frame sequence is subsampled along the time axis, by keeping 1 in every k frames, where k is the subsampling factor. For example, if: k = 5 and the original video sequence has a frame rate = 25 frames-per-second (fps); then the subsampled sequence has a frame rate = 5 fps.
[0089] In adaptive frame skipping, as in fixed frame skipping, the input video frame sequence is subsampled along the time axis in order to achieve a desired or predetermined average frame rate. However, rather than skip-frames in a fixed manner, using adaptive frame skipping the rate of frame skipping can be irregular and can vary along the sequence length. Preferably, low activity frames are identified and skipped, and scene-change frames are kept and intracoded. Non-scene changes having some activity frame are intercoded. Because the skipped frames are intelligently selected based on changes on visual activity, the visual result when reproduced by the decoder will be better than with fixed frame skipping, assuming no or relatively few errors occur.
[0090] In one embodiment, the preprocessing module 102B codes skipped frames using a "not_coded" bit-flag or indicator set in the video object plane (VOP) header in an MPEG bit stream. An MPEG-4 video packet starts with the VOP header or the video packet header, followed by motion_shape_texture(), and ends with next_resync_marker() or next_start_code). Thus, a VOP specifies particular image sequence content and is coded into a separate video object layer by coding contour, motion and texture information.
[0091] In another embodiment, a skipped frame is skipped altogether, without inserting VOP-header information in the bitstream. A skipped frame may be recreated by a decoder using interpolation or by repeating a previous frame. The decoder may perform interpolation by pixel averaging between a preceding frame and a subsequent decoded frame, weighted by their time difference. [0092] The information regarding the video sequence frame rate is normally carried in the Video-Object-Layer (VOL) header. In particular, a parameter, referred to as vop_time_increment_resolution, determines the number of time units for each encoding cycle. The vop_time_increment value in the Video-Object-Plane (VOP) header carries the time stamp for each frame.
[0093] The vop_time_increment_resolution may be, by way of example, a 16 bit unsigned integer value. For example, for 25 frames per second (fps), vop_time_increment_resolution = 25, and vop_time_increment cycles through the values 0...24. For 7.5 fps, vop_time_increment_resolution=75, and vop_time_increment cycles through the values 0, 10, 20, 30, 40, 50, 60, 70, 5, 15, 25, 35, 45, 55, 65. Thus, successful decoding of the first 2 frames of a sequence can yield the exact frame rate. But, given the error-prone environment of wireless communication or other error-prone environments, the successful reception and decoding of any 2 consecutive frames cannot be guaranteed. Thus, the decoder may incorrectly determine the frame rate of the entire sequence. For that reason, in one embodiment, the "fixed_vop_rate" flag is set by the encoder module 106B in the VOL header, which then provides (via the value of fixed_vop_time_increment) the default frame rate. This technique better ensures the successful decoding or determination of the frame rate upon the successful decoding of the VOL header. The fixed_vop_time_increment value can later be stored in a global variable of the decoder, which will use the value to determine whether certain frames need to be interpolated or not. The frames to be interpolated can either be frames skipped by the encoder, or lost during transmission. Thus, the error-resilience performance of the MPEG-4 decoder will be enhanced because it will decode the correct number of frames, thereby avoiding loss-of- sync problems with the audio stream.
[0094] As discussed above, for error-prone environments, error resiliency can be enhanced by utilizing fixed frame skipping rather then adaptive frame skipping. Fixed frame skipping enables the decoder to better determine when a frame has been dropped or skipped. Another approach to enhancing error resiliency is to use adaptive frame skipping, but provide a VOP-header with the not coded flag set for a skipped frame. One drawback of this approach is that it results in a slight increase in bit rate due to the more frequent VOP-headers. [0095] Figure 4A illustrates one example process 400 of adaptive frame skipping. The process is iterative in that a frame is selectively dropped from the sequence of frames by computing a mean of absolute differences between the frames adjacent to the frame of interest, and by weighting the computation with a temporal parameter, wherein the frame having the least impact on the scene is dropped. This procedure is repeatedly iterated until a target frame rate, which is related to the desired bit rate and frame size, is achieved.
[0096] Beginning at start state 402, the process 400 proceeds to state 404. The desired frame rate is set or specified. The desired frame rate may be user specified or may be dynamically determined. Proceeding to state 406, the cost function, or adverse impact, that would result from dropping a particular frame is calculated for each frame between the first and last frame in a scene. As described in greater detail below, the cost function can be based at least in part on the mean absolute differences (MAD) between frames closely or most closely bracketing or bounding the particular frame of interest, or on the sum of the mean absolute differences (SMAD). Alternatively or in addition, the cost function can be based on sums of RMS (SRMS) for frames bracketing the particular frame of interest.
[0097] Proceeding to state 408, the frame associated with the lowest cost, that is, having the least adverse impact on visual quality, is skipped or dropped. At state 410 a determination is made as to whether the remaining frames will allow the target frame rate to be met. If the target frame rate can now be met, the adaptive frame rate skipping process 400 proceeds to the end state 414. Otherwise, the process 400 proceeds to state 412, and a remaining frame having the lowest cost will be dropped. The cost of all the frames remaining between the first and last frames may be recalculated at state 412 based on the frames that are currently remaining, and the frame with the lowest cost will be dropped. The process 400 repeats states 410 and 412 until the target frame rate has been met, or the number of already consecutively skipped frames is at a specified maximum. While the process 400 is described with respect to selecting which frames to skip, the process 400 can be similarly used to decide what frames should be bi-directionally encoded.
[0098] One example embodiment of an adaptive frame skipping selection process using the weighted sum of mean absolute differences will now be described in greater detail. Given an input sequence of a frame rate rori and the desired frame rate rdes
(where rorig > rdes ), the video sequence analyzer skips frames in a "greedy" way, that is, until rdes is met. The weighted sum of MAD and the time difference between two frames are used in specifying a cost function for identifying the next frame to be skipped. The frame whose cost function meets predetermined criteria or is minimum among remaining candidates for skipping is skipped.
[0099] By way of example, referring to Figure 4B, if using the greedy approach E3 , E5 , E6 , and E8 have already been skipped, E4 is now considered as a candidate for the next skipped frame. In order to determine if E4 is to be skipped, a cost function is calculated assuming that E4 has been skipped. Upon skipping E4 , E2 and E7 will be the left and right frames bounding the skipped segment of E3 - E6 . The cost function is then defined as:
29 97 [0100] Cost = MAD(F2 , E7 ) + λ — TD Equation r on ■g
5
[0101] Where the term
29.97 r ong is used to normalize the original frame rate r0ng with respect to the NTSC frame rate of
29.97 frames/second and where TD denotes the time difference measure. Of course other or different normalizations may be used as well.
[0102] Thus, in this example TD is 5 (=7-2), and λ is a weighting coefficient.
In this example, the weighting coefficient value λ = 5.0, determined experimentally, provides a suitable result. Alternatively, the weighting coefficient value λ may be dynamically determined. At the current state, the frame whose cost function is minimum among the candidates is skipped. This process is iteratively performed until the desired frame rate is met. The time difference measure TD can be modified such that no more than n consecutive frames are skipped. For example, if a predetermined maximum number of consecutive frames that can be skipped simultaneously is set to 4, TD can be modified as follows
Figure imgf000027_0001
where ∞ is infinity [0103] Optionally, the complexity for the MAD calculation can be reduced, though the accuracy will be reduced, if only even (or only odd) coordinate pixels are used in the calculation. For example:
MAD(Ft ,Fk) = - Yk (2x,2y)\ Equation 6a
Figure imgf000028_0001
[0104] Another example process to adaptively determine which frame to skip estimates the total induced distortion, both spatial and temporal, for each candidate frame for skipping, and then skips the frame whose absence would result in the least distortion. As described below, the process utilizes sums of MAD (SMAD) or sums of RMS (SRMS). Advantageously, all the MADs do not have to be recalculated. Instead, the already calculated appropriate MADs are summed differently, depending on which frame is being considered for skipping.
[0105] In one embodiment, the process is performed as follows. Assume that Ft is the frame currently being considered for skipping. For example, assume Ft is frame
E4 in Figure 4B. Then let F} indicate the previous non-skipped frame, E2 in this example, and let frame Fk indicate the next non-skipped frame, E7 in our example. Then the cost function can be defined as follows:
Cost =
29 97 I SMADesl (FJ ,Fk) - SMADest (FJ , E, ) - SMADesl (F„Fk ) \ +λ — TD r ong
Equation 6b
[0106] where SMADesl(Ft ,Fk) is the estimated spatial distortion when skipping frames (i+1), ... (k-1).
[0107] As can be seen from Equation 6b, the cost function subtracts out the contribution from previously skipped frames. The distortion is calculated as follows:
SMADest(Fl ,Fk) = φ(k - i - l) - ,FJ )
Figure imgf000028_0002
Equation 6c
[0108] where φ(n) is a coefficient that depends on the number (h) of consecutive skipped frames and takes into account how much, on the average or based on a statistical sampling, interpolated frames at the decoder are different from the original frames. The following table provides example coefficient values determined experimentally using typical video sequences:
Figure imgf000029_0001
[0109] As can be seen, φ (n) increases as the number of skipped frames increases.
[0110] To reduce the computational complexity and resources needed, optionally only the MAD between consecutive frames (j-1 and j) is used to estimate the distortion of a complete segment of skipped frames, as in Equation 6b above. The additional distortion when skipping a certain frame is calculated and the temporal component is added.
[0111] In one embodiment, a user specifies the desired encoding frame rate. The desired frame rate can be based on the video sequence statistics, such as temporal and spatial complexity, frame size, frame rate and target bit rate or compression ratio. In another embodiment, a heuristic can be used to select the desired frame rate. The following is an example heuristic equation for calculating the frame rate, normalized with respect to the Quarter Common Intermediate Format (QCIF) frame size, containing 144 lines and 176 pixels per line:
target bit rate in kbps
Encoding frame rate = Equation 7
176 *144
8 w *h
[0112] where w and h are the frame dimensions. In order for the encoding frame rate to remain within reasonable bounds, the encoding frame rate should preferably be in the range of :
[1 up to, and including, the source frame rate]
[0113] In addition, in order to make adaptive frame skipping independent of the temporal complexity of a specific scene, the weighting coefficient λ is optionally set equal to the average RMS or MAD of the designated entire sequence. [0114] Thus, to determine which frames are scene change frames, and to adaptively select which frames are to be skipped, the video sequence analyzer is provided with the video sequence, the frame width, frame height, the source frame rate, the target bit rate, and the setting of the error resilience flag in the following format:
<input file> <width> <height> <source frame rate> <target bit rate> <error resilience flag>
[0115] The error resilience flag is set by the user to switch between adaptive frame skipping, which has less error resiliency but a better visual result when there are no or few errors, and fixed frame skipping, which provides better error resiliency with a lesser visual result.
[0116] In one embodiment, as illustrated in Figure 1C, an RMS circuit 102C is used to calculate RMS values as described above, a Second Derivative of RMS circuit 104C is used to calculate the second derivative of RMS as described above, a MAD circuit 108C is used to calculate the MAD values as described above, a SUM OF MAD circuit HOC is used to calculate the SUM of MAD values as described above, and a Second Derivative of MAD circuit 114C is used to calculated the Second Derivative of MAD as described above. An Evaluator circuit 112C coupled to the outputs of the RMS circuit 102C, the Second Derivative of RMS circuit 104C, the MAD circuit 108C, and the SUM OF MAD circuit HOC, and the Second Derivative of MAD circuit 114C, is used to determine when a scene change has occurred and what frames to skip, based on one or more of the outputs, as discussed above. Of course different embodiments need not include all or any portion of the circuits illustrated in Figure lC.
[0117] The bit allocation module or circuit 104B will now be described. The bit allocation module or circuit 104B provides for bit allocation on a scene, frame, and/or macroblock level. The bit allocation module 104B reads the file or otherwise receives the information generated by the preprocessing module 102B, including the frame-type designations, and calculates a bit budget for each scene, GOV or GOP based on the coded frames. The bit allocation module determines an appropriate distribution of a fixed bit budget.
[0118] As will be described below with respect to an example implementation, a first intracoded frame defines a beginning of a scene. A weight is assigned to the scene based on the number of intracoded frames and the number of intercoded frames, where intracoded frames are weighted more heavily then intercoded frames to account for the greater number of bits needed to encode an intraframe. The bit allocation module distributes the fixed bit budget within a scene by comparing the current bit usage and target bit usage, and based on the comparison, adjusts a quantization parameter or step size for the current frame.
[0119] In particular, the bit-allocation module 104B first parses the input frame- type file from the preprocessing module 102B. The number or quantity of GOVs is then calculated. Based on the calculated bit budget, the encoder module 106B then encodes each GOV using the novel rate control process in accordance with an encoder parameter file, discussed below.
[0120] For a given GOV or scene, the number of coded frames and the first and end frame are known. Using the following definitions, the bit budget for a GOV or scene is calculated using Equation 8 below:
[0121] Nci = number of coded P-VOPs (predicted, intercoded VOPs) in scene, or GOVi.
B, = bit budget for scene i.
B = bit budget for a clip including one or more scenes
Nc = number of coded frames for the clip
Ns = number of scenes in clip. Usually, in the absence of the inclusion of consecutive I-frames for error resilience purposes, Ns = the number of I-
VOPs (intracoded VOPs) Tc = equivalent total number of VOPs in clip
[0122] An example method of determining the bit-budget for a scene or GOV is as follows. B, bits are allocated for each GOV (i). In this example allocation, an assumption is made that the bit usage for one I-frame or I- VOP is close to or equal to the bit-usage of ten P-frames or P-VOPs (Ratio_Of_I_to_P = 10). However, the method is not limited to using the 1:10 ratio. B, and Tc are then determined as follows:
B, = B * (NC1 + Ratio_Of_I_to_P)/ Tc Equation 8 and
Tc = Nc + (Ratio_Of_I_to_P - 1 )*NS Equation 9 [0123] As defined by Equation 8, the bit allocation for a given scene is based on the total number of frames in the scene, wherein an intracoded frame is normalized to be the equivalent of several predicted frames. In order to reduce the complexity and computational overhead, this example bit allocation formula does not take into account the spatial and temporal complexity of each GOV or GOP. In other embodiments, if sufficient computational and time resources are available, the bit allocation formula takes the temporal and spatial complexity into consideration to provide a still more intelligent bit allocation for each GOV.
[0124] For example, in one embodiment a two-pass encoding process is used that takes into account spatial and temporal complexity. The first pass detects scene changes and collects frame complexity. The second pass performs the actual encoding using complexity guided bit allocation.
[0125] The first pass process will now be described in greater detail. A new GOV or GOP is started from a scene change instance. The two-pass rate control process provides substantially uniform quality for each temporally segmented GOV so that quality variation is better limited to GOV or GOP boundaries. This approach is taken because minimizing quality variation, as measured by the Peak Signal to Noise Ratio (PSNR), the Root Mean Square Error or other image fidelity metric, among different scenes provides less benefit with respect to human visual perception.
[0126] In characterizing the relative frame complexity, the complexity measure in accordance with one embodiment of the present invention is relatively invariant with the quantization parameter (QP) used. In particular, generally the bit count for non-texture information, such as frame headers/syntax and motion vectors, denoted by H„ is constant or almost constant in terms of quantization parameter (QP) change. This is in contrast to the bit count of the texture information, which does vary with a change in QP. By way of example, based on the MPEG-4 VM R-Q model [MPEG4VM], if the total number of bits used for coding the current frame i is R„ then the texture bits T,=R,-H, can be represented as:
R. - H, a, a2 „ ,. , n
= — H — ^- Equation 10
M. Q. Q [0127] where M, is the MAD computed with a motion-compensated residual that is substantially invariant with respect to the QP (i.e., Q,), and ai and a2 are Taylor expansion coefficients of texture bits T, over QP. The coefficients \ and a2 are normally of the same order, that is, have similar values. As can be seen, the lower the QP, the greater then number of texture bits needed to encode a given frame.
[0128] The complexity measure Cg ! addresses both the motion and texture bit count, and is substantially QP invariant. In one embodiment, Cg>1 is defined by the ratio of the texture bit count for a given frame to the average texture bit count, and the ratio of the motion vector bit count for the given frame to the average motion vector bit count, as follows: , = ( R&, - Hg,, ) IRg - H + MV(g, /MVg Equation 11
[0129] where the MV(g)1) is the motion vector bit count for frame(g,i), MV is the average motion vector bit count, and Rg - Hg is the average texture bit count. Because the obtained complexity measure Cg), is substantially QP invariant, the frame complexity can be generated in one pass, that is, with one QP. The calculated frame complexity will be utilized in the second encoding pass as will now be described.
[0130] As illustrated in Figure 9, the rate control process 900 in the second pass consists of a three level hierarchy, scene (GOV or GOP)-level bit allocation 902, frame- level bit allocation 904, and macroblock-level QP adjustment 906 which utilizes the obtained frame complexity values Cg),.
[0131] Complexity-guided scene, GOV, or GOP, level bit allocation will now be described with reference to Figure 10. To assign bits to each scene, the complexity measure is defined for each GOV or GOP by calculating the average spatial complexity
C (8) as follows:
C (ε) = ∑ Cg , / Ng Equation 12
[0132] Then, the GOV or GOP level recursive bit allocation process 1000 is applied, as illustrated in Figure 10. At state 1002, an initialization process is performed, with the following assignments: λ=0 Bit budget Br (bit allocation for a given time window corresponding to a certain number of GOVs or GOPs) = B
The initial transmit buffer fullness βi =Ta x R
Start from GOV or GOP of index 1, that is, the first GOV or GOP.
[0133] At state 1004, bits are assigned to the scene (GOV or GOP) of index g according to the following formula:
C x N Bt(g) = λ x (R/F) x Ν(g) + (1-λ) x = — x Br Equation 13
Σ,C, x N.
[0134] Where:
R = the channel rate
F = the selected frame rate
N(g) = the number of frames in GOV or GOP of index g λ = the weighting factor between the buffer variation and complexity demands. and
defines the total scene complexity for the given time window corresponding to the GOVs or GOPs under consideration
[0135] The case of λ=0 represents the bit allocation scheme that directly follows the frame complexity, which is preferred if the transmit buffer constraints can be met. The assignment λ=1.0 represents the case where the bit budget is evenly distributed without considering the frame complexity. In this case, little pre-loading and a small capacity of the decoder buffer are needed as only the first frame needs to be pre-fetched. The case with 0<λ<1.0 represents a bit-allocation tradeoff between the buffer and the quality constraints.
[0136] At state 1006, the buffer status is inspected with the tentatively assigned bit budget Bt(g), if βg.i + Bt(g)-(R/F) x N(g) < MarginFactorl xβmax where, by way of example, MarginFactorl =0.8, which provides a safe margin (0.8 of the maximum buffer size) for buffer regulation.
[0137] then the allocation is accepted, and the process 1000 proceeds to state 1008. Otherwise, the value of λ is adjusted upwards by a factor, such as 0.1, and the process 1000 proceeds back to state 1004. [0138] At state 1008 the buffer status is updated as follows: βg = βg.1 + Bt(g) - (RT) χ N(g), and the remaining budget Br is adjusted as follows:
Figure imgf000035_0001
[0139] The process 1008 then proceeds back to state 1002 and the value of λ is set equal to 0. The allocation for the next GOV or GOP of index g+1 is then performed.
[0140] The GOV or GOP scene-level bit allocation advantageously suitably allocates the bit budget to each GOV or GOP while meeting both the buffer and the quality constraints. However, to obtain a constant or consistent quality within each GOV or GOP, it is preferable to allocate the bit budget according to frames within each GOV or GOP based on frame complexity and while still meeting buffer constraints. The frame level bit allocation process 904 illustrated in Figure 9 performs such a frame bit allocation process, and is similar to the process 1000 for the GOV or GOP level bit allocation. However, rather then using variables corresponding to a GOP or a GOV, the variables corresponding to frames are utilized. Thus, for frames
Bf = λ x (R/F) + ( 1 -λ) x C(g,,) x BgV(Cg.) Equation 14 where:
Bg' = running bits = (Bit allocation for current GOP - used bits for frames already coded)
Cg' = running complexity = (complexity for current GOP - complexity of frames already coded)
R = target bit rate
F = frame rate
[0141] In one embodiment, the macroblock level QP adjustment 906 can be alternatively performed in accordance with the following description. In the scene and the frame level bit allocation processes described above, a safe margin (0.8 of the maximal buffer by way of example) is provided for buffer regulation. To reduce computational complexity, in one alternative embodiment, all the macroblocks are quantized with the same quantization parameter (QP) using a one-pass rate control. The QP value is determined as part of the frame level rate control using the following iterative process:
[0142] UBactual(i)>1.15*Bt(i), then QP,+ι = QP.+l. lfBπctual(i) ≤ 1.15*Bt(i) then ifBactual(i)<0.85* Bt(i) QP1+1 = QP, - 1 else QP,+ι = QP,. //To ensure that QP1+1 is within the valid QP range of 1-31, the following clipping operation is performed QP1+ι = max(QP,+1, 1); QP1+1 = min (QP,+1, 31).
[0143] However, if the buffer is relatively small, then the macroblock level rate control process as described below can be used.
[0144] Suppose NMB is the number of macroblocks in one frame, MADk is the mean absolute difference of MB k, and QPk-i is the QP for a previous macroblock. Then the QP for the current MB k can be in the range of [QPk-ι-2, QPk-t+2], as determined in accordance with the following rules:
QPk = Equation 15
Figure imgf000036_0001
[0145] Optionally, in high noise environment, the macroblock bit allocation process can be disabled or not used so that the decoder can assume that the QP is the same for each macroblock. This helps prevent the decoder from using the wrong QP when portions of a frame have been corrupted or lost.
[0146] After determining the bit budget for each GOV or GOP, a novel rate control process, also referred to as IMP4, is used to meet or substantially meet the calculated bit budget. Conventional MPEG-4 rate control does provide adequate performance for many applications. For example, many conventional rate control processes do not explicitly support multiple scenes. Instead, these conventional rate control processes assume that an entire sequence comprises a single scene, and therefore fail to provide for satisfactory rate control. By contrast, an embodiment of the present invention takes into account when scene changes occur, and so provides enhanced rate control.
[0147] In one embodiment of the present invention, a self-converging rate control processed is used to meet the bit budget of each GOV by adjusting the quantization parameter QP of each frame, where QP is equal to half the quantization step size. In MPEG-4, the quantizer parameter QP can have 31 values [1-31]. The rate control process determines the QP based on past bit usage, the number of un-coded frame and the rest bits for a given GON. Thus, if the current bit usage exceeds the assigned bit budget by more than a certain amount or percentage, the quantization parameter, and therefore the quantization step size, are increased. If, instead, the current bit usage falls beneath the assigned bit budget by more than a certain amount or percentage, the quantization parameter, and therefore the quantization step size, are decreased. The process can be performed in accordance with the following example pseudo-code:
if (current bit usage > assigned bit budget *Marginl)
QPnext = min (31, IΝT(QPcur+StepUpPrcnt* QPcur));
Else if (current bit usage < assigned bit budget*Margin2)
QPnext = max (1, INT(QPcur-StepDwnPrcnt* QPcur));
End if.
Where:
Margin 1 is a constant that allows the current bit usage to exceed the assigned bit budget, so that the system has an opportunity to stabilize. For example, Marginl can be set equal to 1.15, allowing the current bit usage to exceed the assigned bit budget by 15%.
Margin2 is a constant that allows the current bit usage to under-run the assigned bit budget, so that the system has an opportunity to stabilize. For example, Margin2 can be set equal to 0.85, allowing the current bit usage to under-run the assigned bit budget by 15%.
StepUpPrcnt is constant related to how much the quantization parameter is to be increased. For example, StepUpPrcnt may be set equal to 0.1.
StepDwnPrcnt is constant related to how much the quantization parameter is to be increased. For example, StepDwnPrcnt may be set equal to 0.1.
[0148] After encoding each GOV, the actual bit usage Bact is compared with the pre-assigned bit budget Bass, and if the actual bit usage varies by more then a certain amount or percentage (Margin3, Margin 4) from the budgeted bit usage, the quantization parameter QP for an I-frame (QPI) is adjusted up or down by a certain amount (StepUp, StepDwn) or percentage as needed. The quantization parameter adjustment process can be performed in accordance with the following example pseudo-code: If (Bact > Margin3 * Bass)
QPI = QPI + StepUp; Else if (Bact < Margin4 * Bass)
QPI = QPI - StepDwn; End if
Where, by way of example: Margin3 = 1.2 Margin4 = 0.8 StepUp = 2 StepDwn = -2
[0149] The initial value of QPI may be set to 10, while the initial value of QP for a P-VOP may be set to 12. Note that when the quantization parameter QPI for I- VOP is changed, the QP assignment of the following P-frames may also change.
[0150] In one embodiment, the encoder module 106B then performs adaptive motion change detection to efficiently reduce large propagation errors. In particular, adaptive intra refresh (AIR) is used to reduce error propagation in an MPEG data stream by the selective intra-coding of macroblocks in p-frames. Thus, AIR is used to help determine how many macroblocks should be intra-encoded in the detected motion region of a frame. While the performance of macroblock intra-refresh increases error resilience of the compressed bitstream, increasing the number of intra-coded macroblocks correspondingly increase the number of bits used to encode these macroblocks. Further, if there is a fixed bit rate, the quantization error has to increase for the other, non-intracoded, macroblocks. Thus, preferably, bandwidth and the bit error probability (BER) are taken into account to determine the percentage or number of macroblocks that are to be intracoded.
[0151] In addition, the encoder module 106B optionally used Cyclic Intra Refresh (CIR) to encode a predetermined number of macroblocks (MBs) in each frame. Thus, CIR provides periodic intra-refresh to constrain possible error propagation.
[0152] In one embodiment, Cyclic Intra Refresh (CIR) and Adaptive Intra Refresh (AIR) are performed as follows. The number of Intra macroblocks in a VOP is specified by the user in the encoder parameter file. The number of Intra macroblocks in a VOP depends on target bit rate, frame rate, bit buffer usage, channel noise feedback, and other transmission related parameters. The encoder module 106B estimates the amount of motion for each macroblock and selects heavy motion area to be encoded in INTRA mode to enhance error resiliency. The results of the estimation are recorded a refresh map at the macroblock level.
[0153] An example conventional refresh map 202 is illustrated in Figure 2 A. The encoder module 106B refers to the refresh map and selectively determines whether to encode a given macroblock of the current VOP in TNTRA mode or not. The estimation of motion is performed by comparing SAD (Sum of the Absolute Difference) and SADth- SAD refers to the Sum of the Absolute Differences value between the current macroblock and the macroblock in the same location of the previous VOP. Advantageously, the SAD is already calculated when performing motion estimation. Therefore, the SAD calculation does not have to be repeated as part of the AIR process. SADth is used as a threshold value in determining whether a given macroblock is a motion area. If the SAD of the current macroblock is larger then SADth, this macroblock is regarded as motion area.
[0154] Once a macroblock is regarded as a motion area, it remains as a candidate motion area until it is encoded in Intra mode a predetermined number of times. In the example refresh map 202 illustrated in Figure 2A, the value for this "predetermined number of times" is set "1," in other embodiments, the predetermined number of times can be set equal to 2, or a higher value. Horizontal scanning is used to resolve among macroblocks that are candidates to be encoded in Intra mode within the moving area as illustrated in map 204 in Figure 2B.
[0155] With reference to Figure 8 A, conventional AIR processing, as applied to four sample sequential VOPs, will now be explained in greater detail. The AIR refresh rate, that is, the fixed number of Intra macroblocks in a VOP is preferably determined in advance. In this example, the number of Intra macroblocks in a VOP is set to "2" .
[0156] [1] 1st VOP - Figure 8 A [a] and [b]
[0157] The first VOP is a scene change frame containing elements 802, 804. Therefore, all macroblocks in the 1st VOP are encoded in Intra mode, as illustrated in Figure 8 A [a]. As illustrated in Figure 8 A [b], the refresh map is set to "0", where a 0 indicates that an Intra refresh is not to be performed and a 1 indicates that an Intra refresh is to be performed, because the 1st VOP is encoded without reference to a previous VOP.
[0158] [2] 2nd VOP - Figure 8A [c] - [f] [0159] The 2nd VOP is intercoded as a P-VOP. Elements 802, 804 have moved down one macroblock and to the right by one macroblock. Intra refresh is not performed in this VOP, because all values in the refresh map are still zero, as illustrated in Figure 8A [c]. The encoder module 106B estimates motion of each macroblock. If the SAD for a given macroblock is larger than SADth, the given macroblock is regarded as motion area, illustrated by the hatched area in Figure 8 A [e]; thus, the refresh map is updated as illustrated in Figure 8 A [fj, where the refresh map entry corresponding to a motion macroblock is set to 1.
[0160] [3] 3rd VOP - Figure 8A [g] - [k]
[0161] Elements 802, 804 have moved down by an additional macroblock and to the right by an additional macroblock. When the 3rd VOP is encoded, the encoder module 106B refers to the Refresh Map illustrated in Figure 8 A [g]. If the refresh map indicates that a macroblock is be Intra refreshed, the macroblock is encoded in Intra mode, as illustrated by the macroblocks containing an "X" in Figure 8 A [h]. The corresponding refresh map value for an intracoded macroblock is decreased by 1 as illustrated in Figure 8A [i].
[0162] If the decreased value is 0, the corresponding macroblock is not regarded as a motion area. Proceeding, the processing is substantially the same as that for the 2nd VOP as illustrated in Figure 8A [j]-[k], where if the SAD for a given macroblock is larger than SADth, the given macroblock is regarded as motion area. The refresh map is updated as illustrated in Figure 8 A [k], where the refresh map entry corresponding to a motion macroblock is set to 1.
[0163] [4] 4th VOP - Figure 8 A [1] - [p]
[0164] The processing is substantially the same as for the 3rd VOP. If a current macroblock has a 1 associated with it in the refresh map, it is encoded in Intra mode as illustrated by the macroblocks containing an "X" in Figure 8 A [m]. The corresponding refresh map value for an intracoded macroblock is decreased by 1 as illustrated in Figure 8A[n]. [0165] If the decreased value is 0, the corresponding macroblock is not regarded as a motion area. If the SAD for a given macroblock is larger than SADth, the given macroblock is regarded as motion area. The refresh map is updated as illustrated in Figure 8A [p].
[0166] In another embodiment, a novel enhanced AIR process is performed as follows to select which macroblocks are to be intracoded in a predicted frame. An intercode distortion value and an intracode distortion value are calculated, as are an intercode bit rate and an intracode bit rate. Based on a comparison of the calculated intercode distortion value and the intracode distortion value, and on a comparison of the intercode bit rate and the intracode bit rate for each macroblock, a decision is made as to which macroblocks are to be intracoded. The enhanced AIR process will now be described in greater detail.
[0167] In order to select which prediction frame macroblock is to be intracoded, the expected distortion that would result if the macroblock were lost or corrupted is estimated. For a predicted or intracoded macroblock, the distortion can be reduced if the reference macroblock in the prediction is intracoded.
[0168] With reference to Figure 8B, recursive tracking in conjunction with the prediction path can be used to determine the expected distortion of the macroblock. The dashed lines 804B to 818B, 806B to 820B, 820B to 826B, 812B to 822B, 814B to 824B, 822B to 828B, and 828B to 830B, indicate motion vectors (MV) that are part of the encoded bitstream from a macroblock in the previous frame to a macroblock in the current frame. The angled solid lines, such as those from 802B to 818B, 806B to 820B, 810B to 822B, 816B to 824B, 818B to 826B, 824B to 828B, and 826B to 830B, indicate a zero- motion vector, where a lost motion vector is set to zero. Zero motion vectors are used by a decoder in an error condition, wherein the decoder, for error concealment, replaces an unrecoverable macroblock with a corresponding macroblock from a previous frame. Note that this is just one of the available error concealment strategies, which is termed "basic concealment." Other concealment strategies, such as temporal concealment or supplemental motion vector concealment can be performed as well. Optionally, the effects of these other error concealment strategies are considered separately, in parallel, when performing the recursive tracking. Referring to Figure 8B, "p" is the packet loss probability or rate, and q = (l-p).
[0169] While encoding a given current macroblock, the encoder module 106B performs a motion search on the previous frame and locates a macroblock that most closely matches the current macroblock or is otherwise determined to be a good prediction frame. This located macroblock from the previous frame, depicted by a non-hatched circle, such as macroblocks 802B, 806B, 81 OB, 816B, 818B, 820B, 826B, is called a prediction macroblock. After the motion search is performed, a residual error is calculated and further encoded using the Discrete Cosine Transform (DCT), then quantized using a selected quantization step or quantization parameter (QP), and entropy coded using variable length coding (VLC). The encoded bitstream consists of motion vector information, entropy coded quantized DCT coefficients for the residual error, and corresponding header information.
[0170] When the decoder receives the encoded bitstream, the decoder processes the coded information and reconstructs the macroblocks. When information for a macroblock is missing, which may be due to packet loss or other error conditions, the decoder preferably conceals the corresponding macroblock using one or more error- concealment strategies, such as the basic concealment discussed above. As discussed above, when a macroblock is missing, basic concealment copies a macroblock at the same spatial location from the previous frame. This is equivalent to receiving a zero-motion vector and zero DCT coefficients.
[0171] In order to determine which macroblocks should be intracoded, in one embodiment the encoder system 100 includes corresponding decoder circuitry so that it can mimic the decoder process and reconstruct what the decoder will reconstruct both in the absence of errors, and in the presence of one or more errors, such as a single error affecting just the current macroblock ("MBC"). By way of example, the difference between the error-free reconstruction and the reconstruction assuming one error is termed "concealment error" or EC. EC is defined as follows:
[0172] EC = MBQ-MBC Equation 16
[0173] Where MBQ is the error free reconstruction, and MBC is a single error reconstruction [0174] When a given macroblock is used as a prediction macroblock for the next frame, an error present on the given macroblock will propagate to those macroblocks in the next frame that use the given macroblock for prediction purposes, even when there is no further error in motion vectors and DCT coefficients for those next-frame macroblocks. The mechanism with which error propagates from a macroblock in a given frame to other macroblocks in the next frame is termed "the error propagation model."
[0175] Error attenuation occurs when half-pixel accuracy is used for prediction either in the vertical or horizontal direction or in both the vertical and the horizontal directions. Error attenuation, comparable to a low pass filter, occurs as a result of the low- pass frequency characteristic of the pixel averaging operation applied when half-pixel motion is used. Thus, given the concealment error EC calculated at the encoder system 100B, the propagated error via half-pixel motion in the horizontal direction ECh/2, the propagated error via half-pixel motion in the vertical direction ECv/2, and the propagated error via half-pixel motion in the horizontal and vertical direction EChv/2, can be determined.
[0176] Half pixel interpolation is illustrated in Figure 8D, showing integer pixel locations, half-pixel locations in the horizontal direction, half-pixel locations in the vertical direction, and half-pixel locations in the horizontal and vertical dimension.
[0177] The half-pixel averaging filter that is normally applied to pixel values can be applied to the concealment error, EC, to define four types of propagated error arrays:
[0178] ECO = EC
[0179] ECh/2 = error through horizontal half-pixel motion (value calculated on crosses "x" in Figure 8D)
[0180] ECv/2 = error through vertical half-pixel motion (value calculated on diamonds in Figure 8D)
[0181] EChv/2 = error through horizontal and vertical half-pixel motion (value calculated on squares in Figure 8D)
[0182] For each of the four error arrays, the corresponding energy, which approximates the error variance under the hypothesis of zero mean, is calculated.
[0183] The four error variances for these four cases can correspondingly be defined as:
(Equation 17) σ Ec ' " Ech/2 ' u Ecv/2 and σ 2 Echvll
[0184] The following four transition or strength factors can then be defined as:
2 2 2 2 y& = -|L -l = 0, y* 2 = -T- -l » ^2 = 0^ -! and Ykvl2 = —*—\ Equation σ £C σ Λ/2 σ v/2 σ A 2
18 which correspond to the four possible cases of motion for the current macroblock. These quantities are saved, together with the motion vector that is used to encode the current macroblock, rnx,my ), the initial error energy σE 2 c , and the coding mode (Intra/Inter), in a table, file or other record.
[0185] The half-pixel horizontal and vertical propagation strength can be approximated as follows: ϊkvii = rh/2 + yv,2 + „/2Y 2 Equation 19
[0186] using the transition factors of half-pixel horizontal and half-pixel vertical motion, thereby reducing the computation time and resources needed to calculate half-pixel horizontal and vertical propagation strength or transition factor. In addition, the propagation strengths should be positive. Therefore, a negative propagation strength result will be rounded or set to zero.
[0187] As illustrated in Figure 8E, a motion vector MV can map macroblocks in the current frame Framen, aligned with a grid of 16-pixel rows and columns, into 16x16 pixels in the predicted frame Framen.ι that are not necessarily aligned on the same grid. Indeed, as illustrated in figure 8E, a macroblock in Framen can map to portions of up to four macroblocks in the predicted frame Frame„-ι .
[0188] An error present on one or more of the four possible macroblocks from the previous frame used for prediction for a macroblock in the current frame will be reflected in the macroblock in the current frame. The error relationship can be proportional to the overlap area. For example the error relationship can be proportional or based on the number of pixels that they overlap. Thus, for each macroblock in the current frame, the up to four prediction macroblocks are identified that would be used when encoding the macroblock in Inter mode. Using the corresponding motion vector information, the overlapping area is determined, and a weighting factor equal or related to that area is used to normalize the overlap area to the total macroblock area, 256 (=16x16) for example, as defined by the following equation:
Figure imgf000045_0001
that estimates the expecting distortion on macroblock y in current frame due to macroblock i in previous frame. Note that γ . . is one of the transition factors γ EC , γΛ/2 , γv/2 and γΛv/2 defined previously, depending on the type of motion (half or integer pixel motion vector)
along horizontal, vertical or both directions and w(i, j) = is the weighting factor
256 relating the area of overlap (wl x hi) between macroblock y and macroblock i. The term σ I (ϊ) is the concealment error σ E 2 C for macroblock i.
[0189] With reference to Figure 8B, beginning with the macroblock 830B in current Framen, there are two macroblocks in Framen-i that may be used by a decoder to recreate macroblock 830B, a macroblock 826B used for normal decoding, and a macroblock 828B used for concealment. Each of the macroblocks 826B, 828B in Framen-i may correspond to up to 4 aligned macroblocks, as discussed above. The same "decode or conceal" strategy can be recursively applied for the two macroblocks 826B, 828B in Framen.ι to locate 4 macroblocks 818B, 824,B, 822B, 829B in Frame n-2, and then reach Frames with 8 macroblocks 802B, 804B, 806B, 808B, 810B, 812B, 814B, 816B, and so on. Each of the 8 macroblocks in Framen_3 has a probability of appearing in Framen at the current macroblock, if a certain series of errors/packet loss occurs during transmission. The probability of each one of these 8 paths is the product of the corresponding branch probability (plq), where * is the packet loss probability and q = (1-p). The probability of a particular path to the macroblock 830B occurring can be determined by multiplying the p and q values along the path. Thus, there exist paths that have probability p2 , such as those where two packet losses in a row occur, and a path defined by 812B-830B with probability
P3 .
[0190] Assuming a relatively small probability of error (e.g. 0.1), the higher- order (with respect to p) paths, such as those with a probability of p2 or /?3 , can be neglected, and Figure 8B can thereby be simplified to the paths illustrated in Figure 8F. The reductions in paths in Figure 8B is based on an assumption that a macroblock that is to be used for concealment is not itself corrupted, that is, the probability is neglected of multiple error/packet loss on a certain path between two macroblocks. While this assumption may not always be true, it will most often be true.
[0191] Based on this simplified macroblock relationship, the expected distortion for the current macroblock in Framen can be estimated using the propagation model described above. The expected distortion is defined using the following equation:
D(n) = pσE 2 C(n) + qD'(n -l) Equation 20
[0192] where D'(n -1) is the expected distortion for the reference macroblocks in Framen-ι, as modified by the transition factors to take into account the possible half-pixel motion from Framen-i to Framen. Expanding this formula for the reference macroblock in Framen-i, expected distortion is defined as follows:
D(n) = pσE 2 C(n) + - 2)) Equation 21
Figure imgf000046_0001
γ ((1_, n) is one of the 4 transition factors (γ EC , γ Λ/2 , γ v/2 and γ Λv/2 ) for the reference macroblock in Frame n-ι, depending on the motion vector from Framen-i to Framen. As similarly described above, D"(n - 2) is the expected distortion for the reference macroblocks in Framen_2 as modified by the transition factors to take into account the possible half-pixel motion from Framen-2 to Framen-i and from Framen-i to Framen. Expanding this term further, the expected distortion is defined as:
D(n) = pσE 2 C (n) _ 3))) Equation 22
Figure imgf000046_0002
[0193] If Framen-3 is an I-frame or if the frame buffer is limited or restricted to 3 frame, then Dm(n -3) is equal to zero. Otherwise, the same procedure is recursively applied to previous frame macroblocks. Similarly, if an Intra macroblock is encountered during the recursive processes, it is assumed that the distortion is equal to pσ c , because there is no motion vector, and thus no error-propagation term.
[0194] For the above distortion equations, contributions from each one of a maximum of 4 prediction macroblocks in each frame are summed and multiplied by the corresponding weighting factor that relates the area of overlap between each one of these macroblocks with the target macroblock in Framen.
[0195] The information stored for each macroblock of previous frames can be utilized to calculate the expected distortion for each macroblock for the current Framen. Note that this expected distortion is due to errors in transmission and is not correlated to the distortion due to quantization for each macroblock. Therefore, the expected distortion term needs to be added to the quantization error to determine the total distortion for each macroblock. This total distortion is referred to as "total Inter-mode distortion," or DTINTER , as it relates to Inter mode encoding of macroblocks.
[0196] For Intra mode, the expected distortion reduces to the first term as follows:
D(n) = pσE 2 C (n) Equation 23
[0197] reflecting the expected error distortion that needs to be added to the corresponding Intra-mode quantization distortion in order to get the "total Intra-mode distortion" for the macroblock, also referred to as DTINTRA .
[0198] A certain number of bits are needed for the Inter mode encoding and the Intra mode encoding, respectively referred to as RTINTER and RTINTRA . The difference between these bit rates, ΔR = R,NTRA - RmTER , together with the difference in total distortion, AD = DlNTRA - DINTER can be used to select the best coding mode.
[0199] If the encoder has sufficient resources and capability, a full Rate- Distortion optimization can be performed that involves determination of an optimal weighting factor λ to be used for evaluating a cost function for each macroblock, given by:
C = D + λR Equation 24 and thus obtain the following Intra/Inter decision rule:
[0200] Choose Intra mode, when
• Δ£> < 0 , if ΔR = 0
• — < -λ , if ΔR > 0 ΔR
• — > -λ , if ΔR < 0 ΔR
Else select Inter mode
[0201] Note that determination of the optimal λ parameter is optionally achieved by trying all possible QP and λ combinations. The particular combination that results in the least distortion among all combinations that produce bitstreams below or optionally equal to the desired bit rate is then selected. [0202] Alternatively, the encoder system 100 can first choose macroblocks that satisfy the first of the above listed cases ( ΔZ) < 0 , if ΔR = 0 ), since it is applicable to all
values of λ , and also all macroblocks that satisfy the condition > 0 , if ΔR < 0 , which
ΔR automatically applies to the third case. Then macroblocks with ΔR > 0 are grouped
together and ordered in increasing order with respect to the ratio . Similarly, the
macroblocks with ΔR < 0 are grouped together and ordered in decreasing order with
• ΔE» respect to the same ratio, .
ΔR
[0203] This is shown by the graph illustrated in Figure 8G, representing the
value of the ratio for each macroblock, where "x"s indicate those macroblock with
ΔR
ΔR > 0 and "o"s indicate those with ΔR < 0.
[0204] The "x"s that have the most negative values, or negative values that meet corresponding defined criteria, and the "o"s that have the least negative values, or negative values that meet corresponding defined criteria, are selected as candidates for Intra Refresh. Note that "o"s with a positive value have already been chosen for Intra coding, and "x"s with positive value are excluded altogether as they are automatically intracoded. Experimental results indicates that macroblocks with ΔR > 0 , indicated by an "x", are the most common ones, because generally Intra mode costs more, as measured in bits, compared to Inter mode. Thus, optionally only macroblocks with ΔR > 0 will be considered for Intra refresh. The so-called refresh rate, specifying how many additional macroblocks are to encoded in intra mode, then dictates how many of the candidate macroblocks are eventually chosen.
[0205] A less resource intensive process is to calculate the expected distortion for each macroblock due to transmission error, while ignoring or excluding quantization error. Then, the differential between the expected distortion for Intra and Inter mode can be used as the criterion for selecting macroblocks to be Intra coded by ordering them according to this criterion.
[0206] Thus, Adaptive Intra Refresh (AIR) can be used to help determine how many macroblocks should be intra-encoded in the detected motion region of a frame. AIR can be enabled and disabled in the encoder parameter file using an AIR bit set by a user and read by the encoder module 106B. When AIR is enabled, the user also specifies another parameter, the AIR refresh rate. The AIR refresh rate determines how many macroblocks should be intra-coded in the detected motion region of one frame. Adaptive motion change detection can efficiently reduce the large propagation error, even when the error occurs in the motion region.
[0207] Figure 8H illustrates an embodiment of the E-AIR process. At state 802H, the motion vector or vectors for the current macroblock of interest in Frame (n) are received. At state 804H the motion vector is used to locate which macroblocks from a previous Frame (n-1) are to be used in predicating the current macroblock. At state 806H a determination is made as to how much, in terms of area or pixels, of each of the located macroblocks in Frame (n-1) will be used in generating the current macroblock. At state 808H, the error variances (σ EC> σ Ech/2, σ ECV/2, σ Echv/2) are calculated, including the overlap
weighting ( w(i, j) ). At state 808H the propagation strength transition quantities are
1 +Y.,, calculated based on the error variances. At state 812H, the Intra Error Distortion DINTRA for Frame (n) is calculated. At state 814H, the Inter Error Distortion DΓNTER for Frame (n) is recursively calculated. As discussed above, the recursive calculation can include the error distortion and quantization distortion from previous frames, such as Frame (n-1), Frame (n- 2), and so on, whose errors may propagate to Frame (n). The recursion may be limited to a predetermined number of frame generations, until all or a predetermined amount of the frame buffer is being used, or the recursion may stop when an Intra frame is reached.
[0208] At state 816, the value DeltaD is calculated by taking the difference between DINTRA and DINTER, or by otherwise comparing D1NTRA and DΓNTER- At state 818H the bit quantity or bit rate RINTRA and RINTER for intracoding Frame (n) and for intercoding Frame (n) respectively are determined. At state 820H a comparison of RINTRA and RINTER is made by calculating the difference DeltaR. At state 822H the decision to intracode or to intercode is made based on DeltaR, DeltaD and Lambda using the illustrated criteria. Alternatively, those macroblocks having a DeltaD may be chosen for intracoding. For example, after DeltaD is calculated for all of Frame (n)'s macroblocks, the two macroblocks having the largest DeltaD are intracoded. The macroblock intracode selection may also be based on a cost calculation where the Cost = Rate + lambda*D, or D + lambda*R, and choose the highest N (=ATR rate) [0209] Figure 8C illustrates experimental results comparing the use of Cyclic Intra Refresh, trace 802C, with the use of the enhanced ATR method described immediately above, trace 804C. The overall gain is approximately 1 dB in the PSNR. In one embodiment, the additional computational load is approximately 10%.
[0210] In order to utilize ATR more effectively, conventional Cyclic Intra Refresh (CIR) is combined with AIR. The number of the Intra Refresh macroblocks in a VOP is defined as the summation of the ATR refresh rate and the CTR_refresh_rate. ATR_refresh_rate macroblocks are encoded in AIR mode and CTR_refresh_rate macroblock are encoded in the conventional CIR mode. These values are user definable in the encoder parameter file. When the channel degrades, higher CIR and AIR rates should preferably be assigned. In addition, when the distance between I-frames is large, higher CIR and ATR rates should preferably be assigned. These rates are preferably varied adaptively with changing channel conditions as well as with the coding parameters to improve the tradeoff between error resilience and coding efficiency.
[0211] The encoder parameter file specifies many different encoding parameters, including those discussed above. The encoder parameter file can be used in conjunction with the preprocessing module output by reading the frame- type file, which specifies the encoding type are determined by preprocessing process described above. By way of example, the encoder parameter file includes fields to enable/disable AIR, CIR, and SMC, to specify the ATR and CIR refresh rates, and a flag used to enable or disable the inclusion of two I-frames at the beginning of each scene, GOV or GOP.
The encoder parameter file has the following parameters or fields:
Figure imgf000051_0001
Table 1 Encoder parameter specification.
[0212] To further increase error resiliency, a Header Extension Code (HEC) is included by the encoder module 106B in every packet in a sequence of video packets or in every video packet, and not just on the first video packet following the VOP header as with conventional encoders. This better ensures that even if a packet is lost or corrupted, subsequent packets can still be decoded and used. Further, even typical conventional decoders will be able to handle the inclusion of the enhanced use of HECs as the use of additional HECs is compatible with the MPEG-4 bitstream syntax. Adding a header, including sequence information, to all packets increases overhead by only about 40 bits per packet, or about 0.2%, but results in a noticeable improvement in decoding.
[0213] Further, Second-order Motion Compensation (SMC) is optionally provided to enhance error resiliency. The SMC process is performed by the encoder module 106B and generates supplemental motion vectors so that each predicted frame can be predicted separately from two preceding frames. Sequence 502 of Figure 5 illustrates the SMC process, in which kth frame has motion vectors from both from the (k-l)th frame and the (k-2)tn frame. Therefore, even if the motion vectors from the (k-l)th frame are corrupted, or the (k-l)th frame is itself corrupted, the kth frame can still be predicted from the (k-2)th frame using the corresponding motion vectors. Thus, by inserting the redundant motion vectors, also termed second-order motion vectors, from the (k-2) h frame to kth frame, the scene image quality at the decoder-side will be better protected from transmission errors. For example, even if all the information for ktn frame is corrupted during transmission, the use of SMC can effectively suppress error propagation by excluding kth frame from being used in any later prediction as illustrated in Figure 5 by sequence 504.
[0214] To perform SMC, a frame buffer is included in the encoder module 106B to store the previously decoded frame at time (t-2). This previously decoded frame is used to calculate the second order motion vectors. In one embodiment, these redundant motion vectors are not used in the encoder to produce residuals. The decoder uses the second order motion vectors when the bitstream is corrupted during the transmission and the first order motion vectors or corresponding frame is corrupted. Otherwise, the second order motion vectors need not be used in the decoder. Optionally, full, unrestricted motion search can be performed in order to determine these second-order motion vectors. The information regarding motion between frames (t-2) and (t-1) and between frames (t-1) and (t) can be combined in order to estimate these second-order motion vectors. For this reason, the SMC data can optionally be included, via the "user data mechanism", as explained below, for the first P-VOP following a scene change or for each P-VOP. The advantage of having SMC on only the first P-VOP is that bandwidth is not wasted when there is no error, thereby providing better coding efficiency. However, providing SMC for every P-VOP or for many P-VOPs enhances the robustness and decoding ability of the decoder, especially for cases of severe error conditions.
[0215] An additional video packet, referred to as "User data" or an "SMC" video packet, for each P-VOP is used to transmit these second-order motion vectors. This packet contains, in the same predictive fashion and using the same variable-length codes as in the standard motion vectors, a motion vector for each macroblock or selected macroblocks of the current P-VOP. An HEC is included in this special SMC video packet, which allows the SMC video packet to be decoded even if other packets for this P-VOP are lost. In one embodiment, this packet is positioned in the bitstream at the end of each P- NOP. A user can enable or disable the use of SMC by setting to 1 or 0 the corresponding option in the encoder parameter file. Figure 6 illustrates an example packetized bitstream showing the relative position of packets in the bitstream, including the SMC packet 602.
[0216] Optionally, in order to make SMC packets compliant with the MPEG-4 syntax, a so-called "User data start code" (hex code B2) or the like, including other unique identifier codes to be assigned in the future by MPEG-standards committee and the like, precedes the HEC and motion vector information. The user data start code signals standard decoders not capable of using the second order motion vectors to ignore all bits following it until the next start code in the bitstream, which in this will be a VOP start code. In one embodiment, the encoder includes a unique 16-bit identifier in order not to confuse the SMC user data extensions with data that other people may decide to include in the bitstream following the same convention.
[0217] To further enhance error resiliency, two consecutive I-frames are inserted upon a scene change, even if the second of the two frames is not a scene change frame and has low enough relative motion that it would normally be intercoded, as illustrated in Figure 7. That is, the scene change frame 702 is intracoded, and the next frame 704 is then automatically intracoded. Thus, the loss of one I-frame will not prevent the decoder from decoding the predicted frames 706, 706, 710, 712 that are predicted from frame 702 as well as frame 704. Frames subsequent to the second I-frame may be encoded as intracoded frames, such as P-frames or B-frames.
[0218] The use of two consecutive I-frames advantageously prevents the prediction of a frame in the current scene using scene content from other scenes, without degrading the performance of the SMC. Because the first two consecutive frames in a scene are intracoded, neither first nor second-order motion vectors are inserted into the I- frames.
[0219] The inclusion of the consecutive I-frames can be under the control of the preprocessing module 102B which can designate both a scene change frame and the next frame as intracoded frames. Alternatively, the encoder module 106B can automatically intracode a frame following a frame designated as an intracoded frame by the preprocessing module 102B.
[0220] While the insertion of two consecutive I-frames increases the bit-budget and thus decreases coding efficiency for a certain transmission bit-rate, in error-prone environments this inefficiency is more then compensated for by the additional error- resilience thereby provided. A "consecutive I-frame" flag is provided in the encoder parameter file that can be independent of the SMC flag. Advantageously, the presence of two consecutive I-frames at the beginning of each scene can be used for the decoder to conceal transmission errors more efficiently, even if the SMC mode is not turned on, or when it is turned on for just the first P-VOP following the (double) I-frame after a scene change.
[0221] Adaptive Intra Refresh (AIR) is optionally supported by the encoder module 106B as a by-product of SMC. This mode, enabled when selecting SMC for just the first P-VOP or for every P-VOP, encodes in INTRA mode those macroblocks that have as prediction macroblocks from frames (t-1) and (t-2) two significantly different macroblocks, as measured by the MAD distance measure. An example threshold is 20. Thus, if the MAD between the two prediction macroblocks for a given macroblock in the current frame is greater than 20, this macroblock is intracoded.
[0222] The encoder module 106B also performs general encoder functions, such as motion estimation, residual calculation, and the like. The encoder output can be stored for later transmission or can be transmitted in substantially real-time to a receiving terminal, such as a cellular phone, containing an appropriate decoder.
[0223] To increase error resilience, in one embodiment Intra_dc_vlc_thr is set to "0", so that all DC coefficients are coded using DC VLC in a frame or VOP. In addition, the ac_pred_flag may be disabled for all Intra macroblocks. Both these options are permitted by the syntax and so are supported by standard decoders, and can result in higher quality for the case of error in transmission. This improvement can be on the order of 0.1- 0.2dB in PSNR. In particular, when data partitioning is enabled, the DC coefficient of each 8x8 DCT block of an TNTRA macroblock can either be coded together with the 63 AC DCT coefficients, using what is known as an "INTER VLC" table, or separately, using what is known as an "TNTRA VLC" table.
[0224] Using the INTRA VLC table results in separating the corresponding bits for the DC coefficient from those of the rest 63 AC coefficients as follows:
[0225] For an I- VOP, where the macroblocks are all intracoded, the DC data bits are located before the DC marker (DCM), together with the header bits, while the data bits for AC coefficients are placed after the DC marker.
[0226] For a P-VOP, the DC data bits are placed immediately after the Motion Marker (MM), together with other crucial or important information, and the data bits for AC coefficients follow.
[0227] This separation of DC and AC information enhances error resilience, since the DC data can be decoded and trusted even for corrupted packets, if the DCM/MM markers are correctly hitting during the decode process. Further, with respect to P-VOPs, Reversible Variable Length Code (RVLC) forward/backward decoding can reveal at least one good first part.
[0228] To control whether the DC data is coded together or separate from AC coefficients, the flag called "intra_dc_vlc_thr" is used for each VOP that maps, according to the QP value, each macroblock. Setting this value = 0 means that all macroblocks, regardless of their QP should use an TNTRA DC table and thus separate DC data from AC coefficients. This is a standard syntactic element in MPEG-4 and thus supported by standard decoders.
[0229] The ac_pred_flag is another option that indicates whether for a specific block of an TNTRA macroblock, the top row and first column DCT coefficients, are coded independently or differentially with respect to the neighboring blocks. To enhance error resilience it is preferable to set ac_pred_flag to 0.
[0230] In one embodiment, error-correction is supported at the source level by using Forward Error Correction (FEC). In particular, Bose-Chaudhuri-Hocquenghem (BCH) codes, including Reed-Solomon, are supported. As is well known in the art, BCH is an error detection and correction technique based on Cyclic Redundancy Code. For any positive integers m, m > 3, and t < 2m' there is a binary BCH code with a block length n equal to 2m - 1 and n - k < mt parity check bits, where k is the number of information bits. The BCH code has a minimum distance of at least 2t + 1. Each binary BCH code (n, k, t) can correct up to t bit errors, and thus it is also referred to as a t-error-correcting code.
[0231] Different block sizes may be used. In one embodiment, a block size of 511 is used. FEC is performed at a packetizer level of the elementary video bitstream, which can be considered as source-level error correction. By contrast, channel level error- correction introduces redundancy at the bit-level after multiplexing.
[0232] FEC can provide significant error resilience, at the cost of some bit budget. Figure 11 illustrates an example graph of Forward Error Correction overhead vs. average BER correction capability. As illustrated, there is a close relation between FEC redundancy and error correcting capability, which is a strong indicator of error resilience. Preferably, at least double the expected BER is provided for.
[0233] In one embodiment of the present invention, rather then apply FEC to all packet data, a more efficient process is performed that reduces the number of error correction bits generated as compared to conventional approaches, while still providing significant error correcting capability. One embodiment of the FEC process optionally generates FEC bits only for selected portions of the packets, and in particular, for those portions that are considered more essential or important for purposes of reproducing a frame sequence by the decoder. In addition, the FEC process provides a systematic code, that is, the FEC correction or parity bits are separate from the original uncoded data bits. Thus, even if all the FEC bits are lost, the original selected portions of the packet are still potentially decodable. Further, in one embodiment the FEC data is encoded and transmitted in an MPEG-4 compliant manner as explained below. Thus, even if a decoder that is not equipped to process the FEC data receives the FEC packet, the decoder will still be able to process the frame motion and texture data.
[0234] In particular, FEC is efficiently applied to important data, such as motion vectors, DC coefficients and header information, and FEC bits are not generated for unimportant or less important data. This more important data may be located in a packet between a packet resync field and a motion marker. In particular, for a given frame or VOP, the selected bits targeted for FEC coding are concatenated together with those from other frame packets and the FEC code bits are generated for the concatenated bits. [0235] In one embodiment, rather than including the FEC bits in the same packet or packets as the frame data, for a given frame or VOP, the resulting FEC bits are placed in an additional packet after the regular frame or VOP packets to ensure MPEG compatibility.
[0236] In addition, to better allow the decoder to recover in cases were a data packet is lost or has a motion marker missing, for each standard packet, in the FEC packet a packet identifier is stored in association with a corresponding value indicating how many bits and/or which bits where used to generate FEC bits.
[0237] In order to maintain compatibility with reference or conventional MPEG-4 decoders, this additional FEC packet further includes a user data identifier code, "user_data_start_code," used to identify user defined data, and as such will be ignored by conventional decoders not equipped to process the FEC packet. In addition, where there is no error, the FEC packet will not be used by decoders equipped to handle the FEC bits. But when errors do occur, FEC decoding will help recover data that will allow for decoding even under severe error conditions.
[0238] Thus, as described above, by providing error resiliency and enhanced compression, embodiments of the present invention advantageously enable the transmission of video information even in low-bit rate, high noise environments. For example, embodiments of the present invention enable video transmission to be successfully performed over cellular networks and the like.
[0239] Although this invention has been described in terms of certain preferred embodiments, other embodiments that are apparent to those of ordinary skill in the art are also within the scope of this invention.

Claims

WHAT IS CLAIMED IS:
1. A method of enhancing error resiliency in an encoded video transmission, the method comprising: receiving an indication that a first frame corresponds to a scene change; intracoding the first frame at least partly in response to receiving the indication that the first frame corresponds to a scene change; automatically intracoding the next consecutive frame; and transmitting the intracoded first frame and the intracoded next consecutive frame.
2. The method as defined in Claim 1, wherein the transmission of the intracoded first frame and the intracoded next consecutive frame is MPEG-4 compliant.
3. The method as defined in Claim 1, wherein the first frame is designated as an I-frame.
4. The method as defined in Claim 1, wherein the next consecutive frame is designated as an I-frame.
5. The method as defined in Claim 1, wherein the next consecutive frame is not a scene change frame.
6. The method as defined in Claim 1, wherein a frame immediately subsequent to the next consecutive frame is intercoded.
7. The method as defined in Claim 1, wherein the indication that that first frame corresponds to a scene change is read from a file.
8. A video coding apparatus, comprising: a first instruction stored in a processor readable memory, the first instruction configured to receive an indication that a first frame is to be intracoded; a second instruction stored in processor readable memory, the second instruction configured to intracode the first frame at least partly in response to the indication; and a third instruction stored in processor readable memory, the third instruction configured to automatically intracode the frame immediately after the first frame as a result of the indication that the first frame is to be intracoded.
9. The video coding apparatus as defined in Claim 8, wherein the video coding apparatus is an integrated circuit.
10. The video coding apparatus as defined in Claim 8, wherein the video coding apparatus is a cellular phone.
11. The video coding apparatus as defined in Claim 8, wherein the video coding apparatus is a desktop computer.
12. The video coding apparatus as defined in Claim 8, wherein the video coding apparatus is a personal digital assistant.
13. The video coding apparatus as defined in Claim 8, wherein the first frame is a scene change frame.
14. The video coding apparatus as defined in Claim 8, further comprising a file used to store the indication that the first frame is to be intracoded.
15. A method of encoding video frames, the method comprising: receiving a first instruction to intracode a first frame; intracoding the first frame in response to the first instruction; and intracoding a second frame as a result of the first instruction to intracode the first frame.
16. The method as defined in Claim 15, further comprising transmitting the intracoded first frame and the intracoded next frame.
17. The method as defined in Claim 15, wherein the first frame is a scene change frame.
18. The method as defined in Claim 15, wherein the second frame has an activity level that would cause it to be intercoded in the absence of the first instruction.
19. The method as defined in Claim 15, further comprising coding a third frame based at least in part on its relative visual motion, the third frame immediately subsequent to the second frame.
20. An encoding apparatus, comprising: a means for receiving a first indication to intracode a first frame; a means for intracoding the first frame in response to the first instruction; and a means for intracoding a second frame as a result of the first instruction to intracode the first frame.
21. The encoding apparatus as defined in Claim 20, further comprising a means for providing the first intracoded frame and the second intracoded frame.
22. The encoding apparatus as defined in Claim 20, wherein the encoding apparatus is included in an integrated circuit.
23. A method of detecting a scene change in a digital video sequence having a plurality of frames, the method comprising: calculating a first root mean squared (RMS) value for a first frame relative to a second frame and the second frame relative to a third frame; calculating a first mean absolute difference (MAD) value for the first frame relative to the second frame; determining if the first RMS value meets a first criterion; determining if the first MAD value meets a second criterion; and designating the second frame as a scene change frame at least partly in response to determining that both the first RMS value meets the first criterion and the first MAD value meets the second criterion.
24. The method as defined in Claim 23, wherein the first RMS value is color weighted.
25. The method as defined in Claim 23, further comprising storing an I-frame designation in a file for the second frame and storing a P-frame designation for a third frame in the file.
26. The method as defined in Claim 23, wherein the first RMS value is based at least in part on pixel luminance values and chrominance values of the first and second frames.
27. The method as defined in Claim 23, wherein the first RMS value is defined as a 1 w
RMS(F F ) = — ∑ ∑ \Y (x,y) - Y (x,y)\
' * a + β+y ]l wx h x = ,l vy == 0l κ '
Figure imgf000060_0001
where F; is the first frame, F is the second frame F(x,y) denotes the
(x,y) th pixel in frame E, w is a frame width and Λ is a frame height, Y(x, y) corresponds to a pixel luminance value, and U(x, y) and V(x, y) corresponds to chromaticity components, and α, β and γ are weighting coefficients for luminosity, chromaticity-blue and chromaticity-red components correspondingly.
28. The method as defined in Claim 27, wherein α = β = γ = 1.
29. The method as defined in Claim 23, wherein the first MAD value is calculated using luminance value and excluding chromaticity components.
30. The method as defined in Claim 23, wherein the first criterion is a first threshold and the second criterion is a second threshold.
31. A method of detecting a scene change in a digital video sequence, the method comprising: calculating a second temporal derivative RMS value for a first frame relative to a second frame and the second frame relative to a third frame; and based at least in part on the second derivative value, determining that the second frame is a scene change frame.
32. The method as defined in Claim 31, wherein the determination that the second frame is a scene change frame is further based upon a mean absolute difference value calculated using at least luminosity information for the first and the second frames.
33. The method as defined in Claim 31, wherein the determination that the second frame is a scene change frame is further based upon both an RMS value meeting a first criterion and the second temporal derivative RMS value meeting a second criterion.
34. The method as defined in Claim 31, wherein the second temporal derivative RMS value is greater than or equal to a first threshold.
35. The method as defined in Claim 31, wherein the second frame is designated as a scene change frame when the second temporal derivative RMS value is negative and has a greater absolute value than a first value.
36. The method as defined in Claim 31, further comprising calculating a first RMS value, wherein the first RMS value is color weighted and the second temporal derivative RMS value is based only on temporal components.
37. The method as defined in Claim 31, wherein the second temporal derivative RMS value is equal to (RMS(F,-ι, F.) - 2RMS(F„ F,+ι) + RMS(F,+ι, FI+2)), where F,-ι is the first frame, F, is the second frame, F,+ι is a third frame, and F,+2 is a fourth frame.
38. An apparatus for identifying a scene change in a video sequence, the apparatus comprising: a first instruction stored in processor readable memory, the first instruction configured to calculate a first root mean squared (RMS) value for a first portion of a video sequence relative to a second portion of the video sequence; a second instruction stored in processor readable memory, the second instruction configured to calculate a second temporal derivative RMS value; and a third instruction configured to cause the second portion of the video sequence to be intracoded based at least in part on the second derivative RMS value.
39. The apparatus as defined in Claim 31, wherein the third instruction is further configured to cause the second portion of the video sequence to be intracoded based upon a mean absolute difference value calculated using at least luminosity information for first portion and the second portion.
40. The apparatus as defined in Claim 31, wherein the third instruction is further configured to cause the second portion of the video sequence to be intracoded at least partly in response to both the RMS value meeting a first criterion and the second temporal derivative RMS value meeting a second criterion.
41. The apparatus as defined in Claim 31, wherein the third instruction is further configured to store in processor readable memory a flag indicating that the second portion is to be coded as an I-frame.
42. The apparatus as defined in Claim 31, wherein the first portion is a first frame and the second portion is a second frame.
43. The method as defined in Claim 31, wherein the third instruction identifies the first portion as including a scene change when the second derivative value is negative and has a greater absolute value than a first value.
44. The method as defined in Claim 31, wherein the first RMS value is color weighted.
45. The apparatus as defined in Claim 31, wherein the apparatus is an integrated circuit.
46. A method of determining which portions of a video sequence are to be intracoded, the method comprising: calculating a first root mean squared (RMS) value for a first portion of the video sequence; calculating a first mean absolute difference (MAD) value for the first portion of the video sequence; determining if the first RMS value meets a first criterion; determining if the first MAD value meets a second criterion; determining if the first MAD value meets a third criterion; and causing an intracoding operation to be performed at least partly in response to at least two of the first, second and third criteria being met.
47. The method as defined in Claim 46, wherein the third criterion is that the MAD value is a local maximum.
48. The method as defined in Claim 46, wherein the first portion of the video sequence includes a first frame.
49. The method as defined in Claim 46, wherein the first portion of the video sequence includes a first GOV.
50. The method as defined in Claim 46, wherein the first portion of the video sequence includes a first GOP.
51. A scene change detection apparatus, comprising: an RMS circuit having a first frame information input and an RMS output, the RMS circuit configured to provide at the RMS output a value corresponding to root mean squared differences information between at least two frames based on frame information received on the first frame information input; a MAD circuit having a second frame information input and a MAD output, the MAD circuit configured to provide at the MAD output a value corresponding to mean absolute differences information between at least two frames based on frame information received on the second frame information input; and an evaluator circuit coupled to the RMS output and the MAD output, the evaluator circuit configured to detect a scene change frame based at least on the value corresponding to root mean squared differences information and the value corresponding to mean absolute differences information, and to provide a scene change designation.
52. The apparatus as defined in Claim 50, further comprising a second derivative RMS circuit having a third frame information input and a second derivative RMS output, the second derivative RMS circuit configured to provide at the second derivative RMS output a value corresponding second derivative root mean squared differences information based on frame information received on the third frame information input, wherein the second derivative RMS output is coupled to the evaluator circuit.
53. A bit allocation method for use with video sequences, the method comprising : receiving a clip bit budget for at least a first clip; determining a scene quantity for the first clip; determining a quantity of predicted frames in a first scene in the first clip; and calculating a bit budget for the first scene based at least in part on the scene quantity for the first clip, the first clip bit budget, a quantity of intracoded frames in the first scene, and the quantity of predicted frames in the first scene.
54. The bit allocation method as defined in Claim 53, wherein each scene in the first clip begins with an intracoded frame.
55. The bit allocation method as defined in Claim 53, wherein the bit budget is calculated based at least in part by multiplying the clip bit budget by a sum of the number of predicted frames and a first constant, and dividing by a sum of the quantity of intracoded and predicted frames in the clip and the number of clip scenes multiplied by a second constant.
56. The bit allocation method as defined in Claim 53, wherein the first scene is a GOV.
57. The bit allocation method as defined in Claim 53, wherein the first scene is a GOP.
58. The bit allocation method as defined in Claim 53, wherein the predicted frames are P-VOPs.
59. The bit allocation method as defined in Claim 53, further comprising adjusting a quantization parameter for a first predicted frame in the first scene based on the bit budget for the first scene and current bit usage.
60. The bit allocation method as defined in Claim 59, wherein the adjustment of the quantization parameter is limited to a first range.
61. The bit allocation method as defined in Claim 53, further comprising adjusting a quantization parameter for a first predicted frame in the first scene upwards at least partly in response to determining that current bit usage is greater then a value related to the bit budget for the first scene.
62. The bit allocation method as defined in Claim 53, further comprising adjusting a quantization parameter for a first predicted frame in the first scene downwards at least partly in response to determining that current bit usage is less then a value related to the bit budget for the first scene.
63. The bit allocation method as defined in Claim 53, further comprising adjusting a quantization parameter for a macroblock based at least in part on a channel rate.
64. A bit allocation method for use with video clip scenes, the method comprising : calculating a bit budget for a first scene; calculating bit budgets for corresponding frames, including at least a first frame, within the first scene; and calculating bits budgets corresponding to macroblocks within the first frame.
65. The bit allocation method as defined in Claim 64, wherein the first scene is a GOP.
66. The bit allocation method as defined in Claim 64, wherein the first scene is a GOV.
67. The bit allocation method as defined in Claim 64, wherein the bit budget for the first scene is based at least in part on a complexity determination for the first scene.
68. The bit allocation method as defined in Claim 64, wherein the bit budget for the first scene is based at least in part on a complexity determination for the first scene and an average complexity of a plurality of scenes.
69. The bit allocation method as defined in Claim 64, wherein the bit budget for the first scene is based at least in part on a buffer status.
70. The bit allocation method as defined in Claim 64, wherein the bit budget for the first frame is based on quantization parameter-invariant criteria.
71. The bit allocation method as defined in Claim 64, wherein the bit budget for the first frame is based at least in part on how may texture bits are in the first frame and how many motion vector bits there are for the first frame.
72. The bit allocation method as defined in Claim 71, wherein the bit budget for the first frame is further based on a mean absolute difference value corresponding to the first frame relative to a second frame.
73. The bit allocation method as defined in Claim 64, wherein the bit budget for the first frame is based at least in part on the bit budget for the first scene, a quantity of bits used for already coded frames in the first scene, a complexity of the first scene, and complexities of already coded frames in the first scene.
74. The bit allocation method as defined in Claim 64, wherein all the macroblocks within the first frame are quantized using one quantization parameter value.
75. The bit allocation method as defined in Claim 64, wherein a quantization parameter is varied for the first frame macroblocks based at least in part on current bit usage and budgeted bit usage.
76. An apparatus for allocating bits, comprising: a first instruction configured to assign a first bit budget for a first scene based at least in part on how many intracoded frames are in the first scene, a complexity determination for the first scene, and how many intercoded frames are in the first scene; and a second instruction configured to allocate bit budgets to frames within the first scene based at least in part on a current bit usage and a target bit usage.
77. The apparatus as defined in Claim 76, further comprising varying quantization of macroblocks within the frames based at least in part of the current bit usage and the target bit usage.
78. The apparatus as defined in Claim 76, wherein the first frame in the first scene is intracoded.
79. The apparatus as defined in Claim 76, wherein the first instruction assigns a greater weight to the intracoded frames then to the intercoded frames.
80. The apparatus as defined in Claim 76, wherein the apparatus is an integrated circuit.
81. A bit budgeting apparatus, comprising : a first instruction configured to receive a bit budget for at least a first video sequence; a second instruction configured to determine a scene quantity for the first video sequence; a third instruction configured to determine a quantity of predicted frames in a first scene in the first video sequence; and a fourth instruction configured to determine a bit budget for the first scene based at least in part on the scene quantity for the first clip, the first clip bit budget, a quantity of intracoded frames in the first scene, and the quantity of predicted frames in the first scene.
82. The apparatus as defined in Claim 81, wherein the first scene is a GOV.
83. The apparatus as defined in Claim 81, wherein the bit budget for the first scene is based at least in part on a buffer status.
84. The apparatus as defined in Claim 81, further comprising a fifth instruction configured to determine a bit budget for a first frame based at least in part on how may texture bits are in the first frame and how many motion vector bits there are for the first frame.
85. The apparatus as defined in Claim 81, further comprising a fifth instruction configured to determine quantization parameters for macroblocks in a first frame in the first scene.
86. The apparatus as defined in Claim 81, wherein the apparatus is an integrated circuit.
87. A method of selecting which video sequence frames to skip during an encoding process, the method comprising: receiving a sequence of frames; for each frame in at least a portion of the sequence of frames, determining a corresponding mean of absolute differences value weighted by a temporal parameter between frames bracketing said each frame; and at least partly based on the corresponding mean of absolute differences value weighted by a temporal parameter calculated for each frame, skipping a first frame in the sequence of frames.
88. The method as defined in Claim 87, further comprising: receiving a target frame rate; and determining if the target frame rate is met with the first frame skipped.
89. The method as defined in Claim 88, wherein at least partly in response to determining that the target frame rate is not met with the first frame skipped, selecting a second frame for skipping based at least in part on a second corresponding mean of absolute differences value weighted by a second temporal parameter between frames bracketing the second frame.
90. The method as defined in Claim 87, further comprising limiting to a first amount how many consecutive frames can be skipped.
91. The method as defined in Claim 87, wherein a first intracoded frame is excluded from being skipped.
92. The method as defined in Claim 87, wherein a last frame in the sequence of frames is excluded from being skipped.
93. The method as defined in Claim 87, further comprising providing a header for the first skipped frame and setting a flag indicating that the first frame is not coded.
94. A method of determining which frames to skip in a video sequence, the method comprising: receiving a first frame rate goal; selecting a first potential frame to be skipped, the first potential frame between a first frame and a last frame in a video sequence scene; calculating a first cost value for the first potential frame to be skipped, the first cost value related to the visual impact that would result from skipping the first potential frame; storing the first cost value in computer readable memory; selecting a second potential frame to be skipped, the second potential frame between the first frame and the last frame; calculating a second cost value for the second potential frame to be skipped; storing the second cost value in computer readable memory; selecting a third potential frame to be skipped, the third potential frame between the first frame and the last frame; calculating a third cost value for the third potential frame to be skipped; storing the third cost value in computer readable memory; determining which one of the first, second, and third cost values has the least cost; skipping the one of the one of the first, second, and third potential frames corresponding to the least cost value; determining if the first frame rate goal is met; and at least partly in response to determining the first frame rate goal is not met, recalculating cost values associated with frames remaining from the first, second and third potential frames, and skipping the one of the remaining frames having the least recalculated cost value.
95. The method as defined in Claim 94, wherein the first cost value is based at least in part on a mean average difference (MAD) value and a time difference between the first frame and the second potential frame to be skipped.
96. The method as defined in Claim 95, wherein only a portion of pixels in the first frame and the second potential frame to be skipped are evaluated in determining the MAD value.
97. The method as defined in Claim 95, wherein the time difference is normalized to an NTSC frame rate.
98. The method as defined in Claim 94, wherein the number of frames that are allowed to be consecutively skipped is limited to a first number.
99. The method as defined in Claim 94, further comprising providing a header for each skipped frame.
100. The method as defined in Claim 94, further comprising transmitting a Header Extension Code (HEC), including sequence information, in every packet in a sequence of video packets.
101. A method of selecting image frames to be skipped during an encoding process, the method comprising: receiving a sequence of frames including a first frame; calculating a first cost function value for the first frame, wherein the cost function includes a first temporal parameter related to a time difference and luminosity differences between frames bounding the first frame; calculating a second cost function value for a second frame, wherein the cost function includes a second temporal parameter related to a time difference between frames and luminosity differences bounding the second frame; and selecting for skipping one of the first frame and the second frame based at least in part on the first cost function value and the second cost function value.
102. The method as defined in Claim 101, further comprising: receiving a target frame rate; and determining if the target frame rate is met with one of the first frame and the second frame being skipped.
103. The method as defined in Claim 101, wherein the luminosity differences between frames bounding the first frame are defined using a mean of average difference function.
104. The method as defined in Claim 101, further comprising limiting to a first amount how many consecutive frames can be skipped.
105. The method as defined in Claim 101, wherein the first temporal parameter is normalized to a first frame rate.
106. A processor circuit configured to select video frames to skip in a sequence of frames, the processor circuit comprising: a first instruction configured to calculate for each frame in at least a portion of the sequence of frames a corresponding mean of absolute differences value weighted by a temporal parameter between frames bracketing said each frame in the portion of the sequence of frames; and a second instruction configured to designate a first frame to be skipped based at least in part on the corresponding mean of absolute differences value weighted by the temporal parameter.
107. The processor circuit as defined in Claim 106, further comprising a third instruction configured to determine if a target frame rate is met assuming the first frame is skipped.
108. The processor circuit as defined in Claim 106, further comprising a third instruction configured to limit to a first amount how many consecutive frames can be skipped.
109. A frame rate control apparatus, comprising: a first instruction configured to receive a frame rate target; a second instruction configured to calculate a first cost function value for a first frame in a sequence of frames, wherein the first cost function value is based at least in part on a first temporal parameter related to a time difference and luminosity differences between frames in the sequence of frames bounding the first frame; a third instruction configured to calculate a second cost function value for a second frame in the sequence of frames, wherein the second cost function based at least in part on a second temporal parameter related to a time difference and luminosity differences between frames in the sequence of frames bounding the second frame; a fourth instruction configured to select for skipping one of the first frame and the second frame based at least in part on the first cost function value and the second cost function value; and a fifth instruction configured to determine if the target frame rate is met with the selected one of the first frame and the second frame skipped.
110. The frame rate control apparatus as defined in Claim 109, wherein the frame rate control apparatus is an integrated circuit.
111. The frame rate control apparatus as defined in Claim 109, wherein the luminosity differences between frames bounding the first frame are defined using a mean of average difference function.
112. A method of providing forward error correction (FEC) on a plurality of frame packets, the method comprising: concatenating selected portions of packet data corresponding to a plurality of frame packets for a first frame; generating forward error correction bits for the concatenated selected portions of packet data; and transmitting the forward error correction bits in a separate packet identified with a user data identifier code.
113. The method as defined in Claim 112, wherein the transmission of the forward error correction bits in the separate packet is MPEG-4 compliant.
114. The method as defined in Claim 112, wherein the separate packet is transmitted after the plurality of frame packets.
115. The method as defined in Claim 112, wherein the forward error correction bits are generated using a BCH code.
116. The method as defined in Claim 112, wherein the forward error correction bits are generated using a systematic code.
117. The method as defined in Claim 112, wherein the selected portions of packet data includes motion vector data and DCT data.
118. The method as defined in Claim 112, wherein the selected portions of packet data includes only header data, motion vector data and DCT data.
119. The method as defined in Claim 112, wherein the selected portions of packet data corresponds to packet data located between a resync field and a motion marker.
120. The method as defined in Claim 112, further comprising: setting a flag indicating that a fixed Video Object Plane (VOP) increment is to be used; and providing a corresponding fixed time increment value.
121. The method as defined in Claim 112, further comprising transmitting in the separate packet a value for at least a first of the plurality of frame packets related to a quantity of bits within the first packet for which forward error correction bits were generated.
122. An error correction generation circuit, comprising: a first instruction stored in processor readable memory configured to generate forward error correction data for selected portions of packet data that are to be transmitted in a corresponding plurality of frame packets; a second instruction stored in processor readable memory configured to store the forward error correction data in a first packet separate from the plurality of frame packets; and a third instruction stored in processor readable memory configured to identify the first packet with a first data identifier code.
123. The error correction generation circuit as defined in Claim 122, further comprising a fourth instruction configured to concatenate selected portions of packet data before the first instruction generates the forward error correction data.
124. The error correction generation circuit as defined in Claim 122, further comprising a fourth instruction configured to set a flag indicating that a fixed Video Object Plane (VOP) increment is to be used and to provide a corresponding fixed time increment value.
125. The error correction generation circuit as defined in Claim 122, further comprising a fourth instruction configured to provide a Header Extension Code (HEC) in a every packet in a first sequence of packets.
126. The error correction generation circuit as defined in Claim 122, wherein the error correction generation circuit is an integrated circuit.
127. The error correction generation circuit as defined in Claim 122, wherein the first packet is MPEG-4 compliant.
128. The error correction generation circuit as defined in Claim 122, wherein the forward error correction data is generated using a BCH code.
129. The error correction generation circuit as defined in Claim 122, wherein the forward error correction data is generated using a systematic code.
130. The error correction generation circuit as defined in Claim 122, wherein the selected portions of packet data includes motion vector data and DCT data.
131. The error correction generation circuit as defined in Claim 122, wherein the selected portions of packet data includes only header data, motion vector data and DCT data.
132. The error correction generation circuit as defined in Claim 122, wherein the selected portions of packet data corresponds to packet data located between a resync filed and a motion marker.
133. An encoder circuit, comprising: a means for generating forward error correction data for selected portions of packet data from a plurality of frame packets; a means for storing the forward error correction data in a first packet separate from the plurality of frame packets; and a means for identifying the first packet with a first data identifier code.
134. The encoder as defined in Claim 133, further comprising a means for concatenating the selected portions of packet data.
135. The encoder as defined in Claim 133, further comprising a means for transmitting in the first packet at least a first value related to a quantity of bits within the first packet for which forward error correction bits were generated.
136. A method of performing adaptive intra refresh, the method comprising: receiving a packet loss probability value; receiving a motion vector for a first macroblock in a first frame; mapping the motion vector onto portions of a plurality of other macroblocks in a previous frame; calculating at least a first transition factor value based at least in part on the mapping; calculating an estimated inter distortion value for the first macroblock based on at least the packet loss probability value and the first transition factor value; calculating an estimated intra distortion value for the first macroblock based on at least the packet loss probability value; receiving a first bit quantity value corresponding to a bit quantity used to intracode the first macroblock; receiving a second bit quantity value corresponding to a bit quantity used to intercode the first macroblock; and based at least in part on the estimated inter distortion value, the estimated intra distortion value, the first bit quantity value, and the second bit quantity value, transmitting the first macroblock as one of an intra macroblock and an inter macroblock.
137. The method as defined in Claim 136, further comprising transmitting the first macroblock as one of an intra macroblock and an inter macroblock based in part on a quantization distortion value.
138. The method as defined in Claim 136, further comprising determining a first difference value based on a difference between the estimated intra distortion value and the estimated inter distortion value, and a second difference value based on a difference between the first bit quantity value and the second bit quantity value.
139. The method as defined in Claim 137, wherein the first macroblock is transmitted as an intra macroblock at least partly in response to the second difference value being positive, and the first difference value divided by the second difference value being more negative than a first threshold.
140. The method as defined in Claim 137, wherein the first macroblock is transmitted as an intra macroblock at least partly in response to the second difference value being zero, and the first difference value being negative.
141. The method as defined in Claim 137, wherein the first macroblock is transmitted as an intra macroblock at least partly in response to the second difference value being negative, and the first difference value divided by the second difference value being greater than a first threshold.
142. The method as defined in Claim 141, wherein the first threshold is a weighting factor.
143. The method as defined in Claim 137, wherein the first macroblock is transmitted as an inter macroblock at least partly in response to the second difference value being zero, and the first difference value being positive.
144. The method as defined in Claim 137, wherein the first macroblock is transmitted as an intra macroblock at least partly in response to the second difference value being negative, and the first difference value divided by the second difference value being less than a first threshold.
145. The method as defined in Claim 137, wherein the first macroblock is transmitted as an intra macroblock at least partly in response to the second difference value being positive, and the first difference value divided by the second difference value being less than a first threshold.
146. The method as defined in Claim 136, wherein the estimated intra distortion value is further based on an initial error energy.
147. The method as defined in Claim 136, wherein the estimated intra distortion value is further based on an initial error energy squared multiplied by the packet loss probability value.
148. The method as defined in Claim 136, wherein the estimated inter distortion value is recursively calculated to include distortion propagated from a plurality of previous frames.
149. The method as defined in Claim 136, wherein the first transition factor corresponds to a first half-pixel horizontal and vertical propagation strength.
150. The method as defined in Claim 136, wherein the first transition factor corresponds to a first half-pixel horizontal strength.
151. The method as defined in Claim 136, wherein the first transition factor corresponds to a first half-pixel vertical propagation strength.
152. The method as defined in Claim 136, further comprising limiting how many macroblocks can be intracoded in the first frame to a first amount.
153. A method of selectively intracoding macroblocks, the method comprising: receiving a packet loss probability value; receiving a motion vector for a first macroblock in a first frame; based at least in part on the motion vector, determining which portions of macroblocks in a previous frame would be used in predicting the first macroblock; calculating at least a first propagation strength value based at least in part on determining which portions of macroblocks in the previous frame would be used in predicting the first macroblock; calculating an estimated inter distortion value for the first macroblock based on at least the packet loss probability value and the first propagation strength value; calculating an estimated intra distortion value for the first macroblock based on at least the packet loss probability value; calculating a quantization distortion value for the first macroblock; receiving a first bit quantity value corresponding to a bit quantity used to intracode the first macroblock; receiving a second bit quantity value corresponding to a bit quantity used to intercode the first macroblock; and based at least in part on the estimated inter distortion value, the estimated intra distortion value, the quantization distortion value, the first bit quantity value, and the second bit quantity value, providing for transmission the first macroblock as one of an intra macroblock and an inter macroblock.
154. The method as defined in Claim 153, wherein the intra distortion value is further based on an initial error energy.
155. The method as defined in Claim 153, wherein the estimated inter distortion value is recursively calculated to include distortion propagated from a plurality of previous frames.
156. The method as defined in Claim 153, wherein the intra distortion value is further based on an initial error energy squared multiplied by the packet loss probability value.
157. The method as defined in Claim 153, wherein the first macroblock is provided as one of an intra macroblock and an inter macroblock further based upon a weighting factor.
158. The method as defined in Claim 157, wherein the weighting factor is generated at least in part by calculating a plurality of intra distortions values using corresponding different quantization parameters, and selecting a weighting factor value that results in the lowest distortion value meeting a first bitrate criterion.
159. A method of selectively intracoding macroblocks in a plurality of macroblocks in a first frame, the method comprising: receiving a packet loss probability value; receiving a corresponding motion vector for each macroblock in the plurality of macroblocks; based at least in part on the corresponding motion vector, determining which portions of macroblocks in a previous frame would be used in predicting said each macroblock; calculating for each of said macroblocks at least a first corresponding propagation strength value based at least in part on determining which portions of macroblocks in the previous frame would be used in predicting said each macroblock; calculating for each of said macroblocks an estimated inter distortion value based upon at least the packet loss probability value and the corresponding at least first propagation strength value; calculating for each of said macroblocks an estimated intra distortion value based upon at least the packet loss probability value; calculating for each of said macroblocks a quantization distortion value; and based at least in part on the estimated inter distortion, the estimated intra distortion, and the quantization distortion values for each macroblock in the plurality of macroblocks, designating a subset of the plurality of macroblocks to be intracoded.
160. The method as defined in Claim 159, wherein the subset is limited to a predetermined number of macroblocks.
161. The method as defined in Claim 159, further comprising: for each of said macroblocks, receiving a first bit quantity value corresponding to a bit quantity used to intracode said each macroblock; and for each of said macroblocks, receiving a second bit quantity value corresponding to a bit quantity used to intercode said each macroblock, wherein the subset is designated based in part on the first bit quantity values and the second bit quantity values.
162. A circuit configured to selectively intracode macroblocks, the circuit comprising: a first instruction configured to receive a packet loss probability value; a second instruction configured to receive a motion vector for a first macroblock in a first frame; a third instruction configured to determine, based at least in part on the motion vector, which portions of macroblocks in a previous frame would be used in predicting the first macroblock ; a fourth instruction configured to calculate at least a first propagation strength value based at least in part on the determination of which portions of macroblocks in the previous frame would be used in predicting the first macroblock; a fifth instruction configured to calculate an estimated inter distortion value for the first macroblock based on at least the packet loss probability value and the first propagation strength value; a sixth instruction configured to calculate an estimated intra distortion value for the first macroblock based on at least the packet loss probability value; and a seventh instruction configured to selectively provide for transmission the first macroblock as one of an intra macroblock and an inter macroblock based at least in part on the estimated inter distortion value and the estimated intra distortion value.
163. The circuit as defined in Claim 162, wherein the intra distortion value is further based upon an initial error energy.
164. The circuit as defined in Claim 162, further comprising an eighth instruction configured to calculate a quantization distortion value for the first macroblock, wherein the seventh instruction is further configured to selectively provide for transmission the first macroblock as one of an intra macroblock and an inter macroblock based at least in part on the quantization distortion value.
165. The circuit as defined in Claim 162, wherein the fifth instruction is further configured to recursively calculate the estimated inter distortion value to thereby include distortion propagated from a plurality of previous frames.
166. The circuit as defined in Claim 162, wherein the intra distortion value is further based on an initial error energy squared multiplied by the packet loss probability value.
167. The circuit as defined in Claim 162, wherein the seventh instruction is further configured to selectively provide for transmission the first macroblock as one of an intra macroblock and an inter macroblock based at least in part on a weighting factor.
PCT/US2002/006726 2001-03-05 2002-03-05 Systems and methods for error resilient encoding WO2002071639A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2002570429A JP2004532540A (en) 2001-03-05 2002-03-05 System and method for error resilient coding
EP02721263A EP1374430A4 (en) 2001-03-05 2002-03-05 Systems and methods for error resilient encoding

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US27344301P 2001-03-05 2001-03-05
US60/273,443 2001-03-05
US27585901P 2001-03-14 2001-03-14
US60/275,859 2001-03-14
US28628001P 2001-04-25 2001-04-25
US60/286,280 2001-04-25

Publications (2)

Publication Number Publication Date
WO2002071639A1 true WO2002071639A1 (en) 2002-09-12
WO2002071639A8 WO2002071639A8 (en) 2003-10-30

Family

ID=27402570

Family Applications (3)

Application Number Title Priority Date Filing Date
PCT/US2002/006880 WO2002071640A1 (en) 2001-03-05 2002-03-05 Systems and methods for encoding and decoding redundant motion vectors in compressed video bitstreams
PCT/US2002/006865 WO2002071736A2 (en) 2001-03-05 2002-03-05 Systems and methods of error resilience in a video decoder
PCT/US2002/006726 WO2002071639A1 (en) 2001-03-05 2002-03-05 Systems and methods for error resilient encoding

Family Applications Before (2)

Application Number Title Priority Date Filing Date
PCT/US2002/006880 WO2002071640A1 (en) 2001-03-05 2002-03-05 Systems and methods for encoding and decoding redundant motion vectors in compressed video bitstreams
PCT/US2002/006865 WO2002071736A2 (en) 2001-03-05 2002-03-05 Systems and methods of error resilience in a video decoder

Country Status (5)

Country Link
US (23) US7003033B2 (en)
EP (3) EP1374429A4 (en)
JP (10) JP2004532540A (en)
AU (1) AU2002245609A1 (en)
WO (3) WO2002071640A1 (en)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1251701A2 (en) * 2001-04-16 2002-10-23 Mitsubishi Denki Kabushiki Kaisha Encoding a video with a variable frame-rate while minimizing total average distortion
WO2004043074A1 (en) * 2002-11-04 2004-05-21 Tandberg Telecom As Inter-network and inter-protocol video conference privacy method, apparatus and computer program product
EP1439714A2 (en) * 2002-12-23 2004-07-21 Polycom, Inc. Dynamic intra-coded macroblock refresh interval for video error concealment
SG106261A1 (en) * 2002-11-22 2005-06-29 Toshiba Kk Moving picture encoding/decoding method and device
WO2006075070A1 (en) * 2005-01-07 2006-07-20 France Telecom Video encoding method and device
FR2881013A1 (en) * 2005-01-14 2006-07-21 Canon Kk Continuous video transmission method for e.g. Internet, involves interlacing transmission packets, created with elementary entities of images` spatial zones, so that two packets transport data of two different images and two different zones
JP2006521064A (en) * 2003-03-17 2006-09-14 クゥアルコム・インコーポレイテッド Partial frame encoding system and method for wireless multimedia communication
WO2006126694A2 (en) * 2005-05-27 2006-11-30 Canon Kabushiki Kaisha Video coding using an alternative reference frame for motion compensated prediction
EP2194719A2 (en) * 2008-10-30 2010-06-09 ViXS Systems Inc. Video transcoding system with quality readjustment based on scene change detection
EP2654298A3 (en) * 2012-04-19 2014-10-01 ViXS Systems Inc. Detection of video feature based on variance metric
US9077991B2 (en) 2002-12-10 2015-07-07 Sony Computer Entertainment America Llc System and method for utilizing forward error correction with video compression
US9084936B2 (en) 2002-12-10 2015-07-21 Sony Computer Entertainment America Llc System and method for protecting certain types of multimedia data transmitted over a communication channel
CN105847796A (en) * 2016-03-31 2016-08-10 乐视控股（北京）有限公司 Bit distribution method and device used for video coding
US9420283B2 (en) 2002-12-10 2016-08-16 Sony Interactive Entertainment America Llc System and method for selecting a video encoding format based on feedback data
US9661376B2 (en) 2005-07-13 2017-05-23 Polycom, Inc. Video error concealment method
DE102015121148A1 (en) * 2015-12-04 2017-06-08 Technische Universität München Reduce the transmission time of pictures
US9917874B2 (en) 2009-09-22 2018-03-13 Qualcomm Incorporated Enhanced block-request streaming using block partitioning or request controls for improved client-side handling
US11477253B2 (en) 2006-06-09 2022-10-18 Qualcomm Incorporated Enhanced block-request streaming system using signaling or block creation
JP2023517003A (en) * 2020-07-01 2023-04-21 ▲騰▼▲訊▼科技（深▲セン▼）有限公司 Image processing method, device, electronic device and computer program

Families Citing this family (531)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6104754A (en) * 1995-03-15 2000-08-15 Kabushiki Kaisha Toshiba Moving picture coding and/or decoding systems, and variable-length coding and/or decoding system
US6563953B2 (en) 1998-11-30 2003-05-13 Microsoft Corporation Predictive image compression using a single variable length code for both the luminance and chrominance blocks for each macroblock
IL134182A (en) 2000-01-23 2006-08-01 Vls Com Ltd Method and apparatus for visual lossless pre-processing
US6753929B1 (en) 2000-06-28 2004-06-22 Vls Com Ltd. Method and system for real time motion picture segmentation and superposition
SG146434A1 (en) 2000-11-29 2008-10-30 British Telecomm Transmitting and receiving real-time data
US6765964B1 (en) 2000-12-06 2004-07-20 Realnetworks, Inc. System and method for intracoding video data
US20020126759A1 (en) * 2001-01-10 2002-09-12 Wen-Hsiao Peng Method and apparatus for providing prediction mode fine granularity scalability
EP1374429A4 (en) * 2001-03-05 2009-11-11 Intervideo Inc Systems and methods for encoding and decoding redundant motion vectors in compressed video bitstreams
KR100425676B1 (en) * 2001-03-15 2004-04-03 엘지전자 주식회사 Error recovery method for video transmission system
US20020180891A1 (en) * 2001-04-11 2002-12-05 Cyber Operations, Llc System and method for preconditioning analog video signals
US7151831B2 (en) * 2001-06-06 2006-12-19 Sony Corporation Partial encryption and PID mapping
US7895616B2 (en) 2001-06-06 2011-02-22 Sony Corporation Reconstitution of program streams split across multiple packet identifiers
US7110525B1 (en) 2001-06-25 2006-09-19 Toby Heller Agent training sensitive call routing system
US7266150B2 (en) * 2001-07-11 2007-09-04 Dolby Laboratories, Inc. Interpolation of video compression frames
KR100388612B1 (en) * 2001-07-25 2003-06-25 엘지전자 주식회사 Method of Compressing Packing In Switching System
US7039117B2 (en) * 2001-08-16 2006-05-02 Sony Corporation Error concealment of video data using texture data recovery
JP2003153254A (en) * 2001-08-31 2003-05-23 Canon Inc Data processing apparatus and method, as well as program, and storage medium
US8923688B2 (en) * 2001-09-12 2014-12-30 Broadcom Corporation Performing personal video recording (PVR) functions on digital video streams
EP1428357A1 (en) * 2001-09-21 2004-06-16 British Telecommunications Public Limited Company Data communications method and system using receiving buffer size to calculate transmission rate for congestion control
US6956902B2 (en) * 2001-10-11 2005-10-18 Hewlett-Packard Development Company, L.P. Method and apparatus for a multi-user video navigation system
EP1442600B1 (en) * 2001-10-16 2010-04-28 Koninklijke Philips Electronics N.V. Video coding method and corresponding transmittable video signal
US7120168B2 (en) * 2001-11-20 2006-10-10 Sony Corporation System and method for effectively performing an audio/video synchronization procedure
PT2268034T (en) * 2001-11-22 2016-11-18 Godo Kaisha Ip Bridge 1 Variable length coding method and variable length decoding method
EP1449331B1 (en) * 2001-11-30 2007-09-19 British Telecommunications Public Limited Company Data transmission
WO2003053066A1 (en) 2001-12-17 2003-06-26 Microsoft Corporation Skip macroblock coding
KR100460950B1 (en) * 2001-12-18 2004-12-09 삼성전자주식회사 Transcoder and transcoding method
FR2833796B1 (en) * 2001-12-19 2004-04-09 Thomson Licensing Sa METHOD AND DEVICE FOR COMPRESSING VIDEO DATA ENCODED BY VIDEO PACKETS
US7292690B2 (en) * 2002-01-02 2007-11-06 Sony Corporation Video scene change detection
US7302059B2 (en) * 2002-01-02 2007-11-27 Sony Corporation Star pattern partial encryption
US7155012B2 (en) 2002-01-02 2006-12-26 Sony Corporation Slice mask and moat pattern partial encryption
US8027470B2 (en) * 2002-01-02 2011-09-27 Sony Corporation Video slice and active region based multiple partial encryption
US7376233B2 (en) * 2002-01-02 2008-05-20 Sony Corporation Video slice and active region based multiple partial encryption
US8051443B2 (en) * 2002-01-02 2011-11-01 Sony Corporation Content replacement by PID mapping
US7215770B2 (en) 2002-01-02 2007-05-08 Sony Corporation System and method for partially encrypted multimedia stream
US7823174B2 (en) 2002-01-02 2010-10-26 Sony Corporation Macro-block based content replacement by PID mapping
US7765567B2 (en) 2002-01-02 2010-07-27 Sony Corporation Content replacement by PID mapping
JP4114859B2 (en) 2002-01-09 2008-07-09 松下電器産業株式会社 Motion vector encoding method and motion vector decoding method
FI114527B (en) * 2002-01-23 2004-10-29 Nokia Corp Grouping of picture frames in video encoding
US7660474B2 (en) * 2002-01-23 2010-02-09 Siemens Aktiengesellschaft Coding of image sequences with a plurality of image blocks and reference images
CN1288915C (en) * 2002-01-23 2006-12-06 诺基亚有限公司 Grouping of image frames in video coding
EP1472880B1 (en) * 2002-01-24 2007-02-28 Koninklijke Philips Electronics N.V. Error correction of stream data
US7003035B2 (en) 2002-01-25 2006-02-21 Microsoft Corporation Video coding methods and apparatuses
US20050094648A1 (en) * 2002-02-20 2005-05-05 Van Den Heuvel Sebastiaan Antonius F.A. Video information stream distribution unit
GB2386275B (en) * 2002-03-05 2004-03-17 Motorola Inc Scalable video transmissions
KR100846770B1 (en) * 2002-03-05 2008-07-16 삼성전자주식회사 Method for encoding a moving picture and apparatus therefor
KR100850705B1 (en) * 2002-03-09 2008-08-06 삼성전자주식회사 Method for adaptive encoding motion image based on the temperal and spatial complexity and apparatus thereof
EP1345451A1 (en) * 2002-03-15 2003-09-17 BRITISH TELECOMMUNICATIONS public limited company Video processing
ATE363809T1 (en) * 2002-03-27 2007-06-15 British Telecomm DATA STRUCTURE FOR A DATA TRANSMISSION SYSTEM
AU2003215752A1 (en) * 2002-03-27 2003-10-13 British Telecommunications Public Limited Company Video coding and transmission
EP1359722A1 (en) 2002-03-27 2003-11-05 BRITISH TELECOMMUNICATIONS public limited company Data streaming system and method
US7151856B2 (en) * 2002-04-25 2006-12-19 Matsushita Electric Industrial Co., Ltd. Picture coding apparatus and picture coding method
JP4135395B2 (en) * 2002-04-26 2008-08-20 日本電気株式会社 Encoded packet transmission receiving method and apparatus and program thereof
US7428684B2 (en) * 2002-04-29 2008-09-23 Fraunhofer-Gesellschaft Zur Forderung Der Angewandten Forschung E.V. Device and method for concealing an error
JP2003348594A (en) * 2002-05-27 2003-12-05 Sony Corp Device and method for decoding image
FR2840495B1 (en) * 2002-05-29 2004-07-30 Canon Kk METHOD AND DEVICE FOR SELECTING A TRANSCODING METHOD FROM A SET OF TRANSCODING METHODS
US20040001546A1 (en) 2002-06-03 2004-01-01 Alexandros Tourapis Spatiotemporal prediction for bidirectionally predictive (B) pictures and motion vector prediction for multi-picture reference motion compensation
US7450646B2 (en) * 2002-06-04 2008-11-11 Panasonic Corporation Image data transmitting apparatus and method and image data reproducing apparatus and method
US7471880B2 (en) * 2002-07-04 2008-12-30 Mediatek Inc. DVD-ROM controller and MPEG decoder with shared memory controller
US7944971B1 (en) * 2002-07-14 2011-05-17 Apple Inc. Encoding video
RU2321967C2 (en) * 2002-07-15 2008-04-10 Нокиа Корпорейшн Method for masking errors in video sequences
US7154952B2 (en) 2002-07-19 2006-12-26 Microsoft Corporation Timestamp-independent motion vector prediction for predictive (P) and bidirectionally predictive (B) pictures
US7421129B2 (en) 2002-09-04 2008-09-02 Microsoft Corporation Image compression and synthesis for video effects
US8818896B2 (en) 2002-09-09 2014-08-26 Sony Corporation Selective encryption with coverage encryption
JP2004112593A (en) * 2002-09-20 2004-04-08 Pioneer Electronic Corp Data read method, data read device, and program for data read
US7075987B2 (en) * 2002-09-23 2006-07-11 Intel Corporation Adaptive video bit-rate control
US7321623B2 (en) * 2002-10-01 2008-01-22 Avocent Corporation Video compression system
US20060126718A1 (en) * 2002-10-01 2006-06-15 Avocent Corporation Video compression encoder
US7466755B2 (en) * 2002-10-04 2008-12-16 Industrial Technology Research Institute Method for video error concealment by updating statistics
US7027515B2 (en) * 2002-10-15 2006-04-11 Red Rock Semiconductor Ltd. Sum-of-absolute-difference checking of macroblock borders for error detection in a corrupted MPEG-4 bitstream
TWI220636B (en) * 2002-11-13 2004-08-21 Mediatek Inc System and method for video encoding according to degree of macroblock distortion
US7440502B2 (en) * 2002-11-14 2008-10-21 Georgia Tech Research Corporation Signal processing system
SG111978A1 (en) * 2002-11-20 2005-06-29 Victor Company Of Japan An mpeg-4 live unicast video streaming system in wireless network with end-to-end bitrate-based congestion control
US9138644B2 (en) 2002-12-10 2015-09-22 Sony Computer Entertainment America Llc System and method for accelerated machine switching
US9314691B2 (en) 2002-12-10 2016-04-19 Sony Computer Entertainment America Llc System and method for compressing video frames or portions thereof based on feedback information from a client device
US8964830B2 (en) * 2002-12-10 2015-02-24 Ol2, Inc. System and method for multi-stream video compression using multiple encoding formats
US20090118019A1 (en) 2002-12-10 2009-05-07 Onlive, Inc. System for streaming databases serving real-time applications used through streaming interactive video
US20040125237A1 (en) * 2002-12-31 2004-07-01 Intel Corporation Fast slope calculation method for shot detection in a video sequence
AU2003300445A1 (en) * 2003-01-10 2004-08-10 Thomson Licensing S.A. Technique for defining concealment order to minimize error propagation
CN1323553C (en) * 2003-01-10 2007-06-27 汤姆森许可贸易公司 Spatial error concealment based on the intra-prediction modes transmitted in a coded stream
US7256797B2 (en) * 2003-01-31 2007-08-14 Yamaha Corporation Image processing device with synchronized sprite rendering and sprite buffer
US9818136B1 (en) 2003-02-05 2017-11-14 Steven M. Hoffberg System and method for determining contingent relevance
DE10310023A1 (en) * 2003-02-28 2004-09-16 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Method and arrangement for video coding, the video coding comprising texture analysis and texture synthesis, as well as a corresponding computer program and a corresponding computer-readable storage medium
KR20040079084A (en) * 2003-03-06 2004-09-14 삼성전자주식회사 Method for adaptively encoding motion image based on the temperal complexity and apparatus thereof
GB0306296D0 (en) * 2003-03-19 2003-04-23 British Telecomm Data transmission
US7292692B2 (en) * 2003-03-25 2007-11-06 Sony Corporation Content scrambling with minimal impact on legacy devices
US7551671B2 (en) * 2003-04-16 2009-06-23 General Dynamics Decision Systems, Inc. System and method for transmission of video signals using multiple channels
DE10318068B4 (en) * 2003-04-17 2009-08-27 Phoenix Contact Gmbh & Co. Kg Method and device for packet-oriented transmission of security-relevant data
US20040218669A1 (en) * 2003-04-30 2004-11-04 Nokia Corporation Picture coding method
US8824553B2 (en) 2003-05-12 2014-09-02 Google Inc. Video compression method
US7499104B2 (en) * 2003-05-16 2009-03-03 Pixel Instruments Corporation Method and apparatus for determining relative timing of image and associated information
KR100584422B1 (en) * 2003-06-04 2006-05-26 삼성전자주식회사 Method and device for compressing image data
US7408986B2 (en) 2003-06-13 2008-08-05 Microsoft Corporation Increasing motion smoothness using frame interpolation with motion analysis
US7558320B2 (en) 2003-06-13 2009-07-07 Microsoft Corporation Quality control in frame interpolation with motion analysis
US20040260827A1 (en) * 2003-06-19 2004-12-23 Nokia Corporation Stream switching based on gradual decoder refresh
US20040258154A1 (en) * 2003-06-19 2004-12-23 Microsoft Corporation System and method for multi-stage predictive motion estimation
US7313183B2 (en) * 2003-06-24 2007-12-25 Lsi Corporation Real time scene change detection in video sequences
JP2007525063A (en) * 2003-06-26 2007-08-30 トムソン ライセンシング How to control multipath video rate to match sliding window channel limit
US7751474B2 (en) * 2003-06-30 2010-07-06 Mitsubishi Denki Kabushiki Kaisha Image encoding device and image encoding method
FR2857205B1 (en) * 2003-07-04 2005-09-23 Nextream France DEVICE AND METHOD FOR VIDEO DATA CODING
EP1499131A1 (en) * 2003-07-14 2005-01-19 Deutsche Thomson-Brandt Gmbh Method and apparatus for decoding a data stream in audio video streaming systems
US7609763B2 (en) * 2003-07-18 2009-10-27 Microsoft Corporation Advanced bi-directional predictive coding of video frames
WO2005017781A1 (en) * 2003-07-25 2005-02-24 Sony Electronics Inc. Video content scene change determination
US9560371B2 (en) * 2003-07-30 2017-01-31 Avocent Corporation Video compression system
US7324592B2 (en) * 2003-08-13 2008-01-29 Mitsubishi Electric Research Laboratories, Inc. Resource-constrained encoding of multiple videos
US7284072B2 (en) * 2003-08-13 2007-10-16 Broadcom Corporation DMA engine for fetching words in reverse order
US7489726B2 (en) * 2003-08-13 2009-02-10 Mitsubishi Electric Research Laboratories, Inc. Resource-constrained sampling of multiple compressed videos
JP2005065122A (en) * 2003-08-19 2005-03-10 Matsushita Electric Ind Co Ltd Dynamic image encoding device and its method
KR100640498B1 (en) * 2003-09-06 2006-10-30 삼성전자주식회사 Apparatus and method for concealing error of frame
US7092576B2 (en) * 2003-09-07 2006-08-15 Microsoft Corporation Bitplane coding for macroblock field/frame coding type information
US7724827B2 (en) * 2003-09-07 2010-05-25 Microsoft Corporation Multi-layer run level encoding and decoding
US8064520B2 (en) * 2003-09-07 2011-11-22 Microsoft Corporation Advanced bi-directional predictive coding of interlaced video
US7286667B1 (en) 2003-09-15 2007-10-23 Sony Corporation Decryption system
GB2406184B (en) * 2003-09-17 2006-03-15 Advanced Risc Mach Ltd Data processing system
WO2005029833A2 (en) * 2003-09-21 2005-03-31 Servision Ltd. Deriving motion detection information from motion-vector-search type video encoders
US7573872B2 (en) * 2003-10-01 2009-08-11 Nortel Networks Limited Selective forwarding of damaged packets
WO2005046072A1 (en) * 2003-10-09 2005-05-19 Thomson Licensing Direct mode derivation process for error concealment
KR20050040448A (en) * 2003-10-28 2005-05-03 삼성전자주식회사 Method for video decording with error detection, and apparatus for the same
US7853980B2 (en) 2003-10-31 2010-12-14 Sony Corporation Bi-directional indices for trick mode video-on-demand
US7394855B2 (en) * 2003-11-20 2008-07-01 Mitsubishi Electric Research Laboratories, Inc. Error concealing decoding method of intra-frames of compressed videos
US7370125B2 (en) * 2003-11-25 2008-05-06 Intel Corporation Stream under-run/over-run recovery
US7796499B2 (en) 2003-12-05 2010-09-14 Telefonaktiebolaget L M Ericsson (Publ) Method of and system for video fast update
US8472792B2 (en) 2003-12-08 2013-06-25 Divx, Llc Multimedia distribution system
US7519274B2 (en) 2003-12-08 2009-04-14 Divx, Inc. File format for multiple track digital data
US8717868B2 (en) * 2003-12-19 2014-05-06 Rockstar Consortium Us Lp Selective processing of damaged packets
US7889792B2 (en) * 2003-12-24 2011-02-15 Apple Inc. Method and system for video encoding using a variable number of B frames
EP1551185A1 (en) * 2004-01-05 2005-07-06 Thomson Licensing S.A. Encoding method, decoding method, and encoding apparatus for a digital picture sequence
US7606313B2 (en) * 2004-01-15 2009-10-20 Ittiam Systems (P) Ltd. System, method, and apparatus for error concealment in coded video signals
EP1719081B1 (en) * 2004-01-30 2013-09-04 Telefonaktiebolaget L M Ericsson (Publ) Prioritising data elements of a data stream
WO2005076632A2 (en) * 2004-01-30 2005-08-18 Thomson Licensing Encoder with adaptive rate control for h.264
US7697608B2 (en) * 2004-02-03 2010-04-13 Sony Corporation Scalable MPEG video/macro block rate control
US20050169473A1 (en) * 2004-02-03 2005-08-04 Candelore Brant L. Multiple selective encryption with DRM
US20050169369A1 (en) * 2004-02-03 2005-08-04 Sony Corporation Scalable MPEG video/macro block rate control
US7986731B2 (en) 2004-02-06 2011-07-26 Apple Inc. H.264/AVC coder incorporating rate and quality controller
US7869503B2 (en) * 2004-02-06 2011-01-11 Apple Inc. Rate and quality controller for H.264/AVC video coder and scene analyzer therefor
US7492820B2 (en) 2004-02-06 2009-02-17 Apple Inc. Rate control for video coder employing adaptive linear regression bits modeling
EP2770694A1 (en) 2004-02-12 2014-08-27 Core Wireless Licensing S.a.r.l. Classified media quality of experience
JP2007524309A (en) * 2004-02-20 2007-08-23 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ Video decoding method
WO2005094086A1 (en) * 2004-02-27 2005-10-06 Thomson Licensing Error concealment technique using weighted prediction
US7586924B2 (en) 2004-02-27 2009-09-08 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Apparatus and method for coding an information signal into a data stream, converting the data stream and decoding the data stream
US7599565B2 (en) * 2004-03-10 2009-10-06 Nokia Corporation Method and device for transform-domain video editing
US20050201469A1 (en) * 2004-03-11 2005-09-15 John Sievers Method and apparatus for improving the average image refresh rate in a compressed video bitstream
US20050201470A1 (en) * 2004-03-12 2005-09-15 John Sievers Intra block walk around refresh for H.264
US20050207501A1 (en) * 2004-03-18 2005-09-22 Sony Corporation Method of and system for video bit allocation for scene cuts and scene changes
KR100647948B1 (en) * 2004-03-22 2006-11-17 엘지전자 주식회사 Method for refreshing of adaptative intra macro block
CA2563107C (en) 2004-03-29 2014-03-04 Nielsen Media Research, Inc. Methods and apparatus to detect a blank frame in a digital video broadcast signal
JP4031455B2 (en) * 2004-03-29 2008-01-09 株式会社東芝 Image encoding device
JP4020883B2 (en) * 2004-04-20 2007-12-12 株式会社東芝 Video decoding device
US7818444B2 (en) 2004-04-30 2010-10-19 Move Networks, Inc. Apparatus, system, and method for multi-bitrate content streaming
US7882421B2 (en) 2004-05-06 2011-02-01 Seyfullah Halit Oguz Method and apparatus for joint source-channel map decoding
EP1757100B1 (en) * 2004-06-15 2008-08-27 NTT DoCoMo INC. Apparatus and method for generating a transmit frame
US7457461B2 (en) * 2004-06-25 2008-11-25 Avocent Corporation Video compression noise immunity
US20070058614A1 (en) * 2004-06-30 2007-03-15 Plotky Jon S Bandwidth utilization for video mail
US7903902B2 (en) 2004-07-26 2011-03-08 Sheraizin Semion M Adaptive image improvement
US7639892B2 (en) * 2004-07-26 2009-12-29 Sheraizin Semion M Adaptive image improvement
US8861601B2 (en) * 2004-08-18 2014-10-14 Qualcomm Incorporated Encoder-assisted adaptive video frame interpolation
US20060045190A1 (en) * 2004-09-02 2006-03-02 Sharp Laboratories Of America, Inc. Low-complexity error concealment for real-time video decoder
EP1790091A4 (en) * 2004-09-02 2009-12-23 Univ California Content and channel aware object scheduling and error control
JP2006079779A (en) * 2004-09-13 2006-03-23 Matsushita Electric Ind Co Ltd Demultiplexer
JP2006086670A (en) * 2004-09-15 2006-03-30 Hitachi Ltd Data recording device
US20060062304A1 (en) * 2004-09-17 2006-03-23 Shih-Chang Hsia Apparatus and method for error concealment
US20060062312A1 (en) * 2004-09-22 2006-03-23 Yen-Chi Lee Video demultiplexer and decoder with efficient data recovery
US7474701B2 (en) * 2004-09-23 2009-01-06 International Business Machines Corporation Single pass variable bit rate control strategy and encoder for processing a video frame of a sequence of video frames
US7679627B2 (en) * 2004-09-27 2010-03-16 Qualcomm Mems Technologies, Inc. Controller and driver features for bi-stable display
US8514938B2 (en) * 2004-10-07 2013-08-20 Hewlett-Packard Development Company L.P. Picture coding apparatus for a still picture sequence and picture decoding apparatus for a still picture sequence
US8948266B2 (en) * 2004-10-12 2015-02-03 Qualcomm Incorporated Adaptive intra-refresh for digital video encoding
WO2006044260A1 (en) * 2004-10-18 2006-04-27 Thomson Licensing Film grain simulation method
US7382381B2 (en) * 2004-10-22 2008-06-03 Hewlett-Packard Development Company, L.P. Graphics to video encoder
US7587091B2 (en) * 2004-10-29 2009-09-08 Intel Corporation De-interlacing using decoder parameters
EP2202982A3 (en) * 2004-11-12 2012-10-10 Thomson Licensing Film grain simulation for normal play and trick mode play for video playback systems
TWI248312B (en) * 2004-11-16 2006-01-21 Aiptek Int Inc Method for locating the partitions of a video image
TWI246862B (en) * 2004-11-16 2006-01-01 An Lnternet Products & Technol Video coding/decoding buffering apparatus and buffering method thereof
PT1812904E (en) 2004-11-16 2012-05-22 Thomson Licensing Film grain simulation method based on pre-computed transform coefficients
EP1813119B1 (en) 2004-11-16 2019-01-09 Thomson Licensing Film grain sei message insertion for bit-accurate simulation in a video system
EP1825388A4 (en) * 2004-11-17 2010-07-28 Univ California System and method for providing a web page
PT1812905T (en) 2004-11-17 2019-08-06 Interdigital Vc Holdings Inc Bit-accurate film grain simulation method based on pre-computed transformed coefficients
KR101208158B1 (en) * 2004-11-22 2012-12-05 톰슨 라이센싱 Methods, apparatus and system for film grain cache splitting for film grain simulation
US7650031B2 (en) * 2004-11-23 2010-01-19 Microsoft Corporation Method and system for detecting black frames in a sequence of frames
KR20060059782A (en) * 2004-11-29 2006-06-02 엘지전자 주식회사 Method for supporting scalable progressive downloading of video signal
US20060120406A1 (en) * 2004-12-03 2006-06-08 Chao-Hung Wu Internet A/V data imaging results & transmission rate improvement methodology
US7895617B2 (en) 2004-12-15 2011-02-22 Sony Corporation Content substitution editor
US8041190B2 (en) 2004-12-15 2011-10-18 Sony Corporation System and method for the creation, synchronization and delivery of alternate content
GB0428160D0 (en) * 2004-12-22 2005-01-26 British Telecomm Variable bit rate processing
GB0428156D0 (en) * 2004-12-22 2005-01-26 British Telecomm Buffer overflow prevention
GB0428155D0 (en) * 2004-12-22 2005-01-26 British Telecomm Buffer underflow prevention
US20060140591A1 (en) * 2004-12-28 2006-06-29 Texas Instruments Incorporated Systems and methods for load balancing audio/video streams
JP4367337B2 (en) * 2004-12-28 2009-11-18 セイコーエプソン株式会社 Multimedia processing system and multimedia processing method
KR20070097548A (en) * 2004-12-29 2007-10-04 코닌클리케 필립스 일렉트로닉스 엔.브이. Method and apparatus for encoding video data stream
US7415041B2 (en) * 2004-12-31 2008-08-19 Motorola, Inc. Method and apparatus for decoding data in a wireless communication system
FR2880462A1 (en) * 2005-01-06 2006-07-07 Thomson Licensing Sa METHOD FOR REPRODUCING DOCUMENTS INCLUDING ALTERED SEQUENCES AND RELATED REPRODUCTION DEVICE
GB0500332D0 (en) * 2005-01-08 2005-02-16 Univ Bristol Enhanced error concealment
US8780957B2 (en) 2005-01-14 2014-07-15 Qualcomm Incorporated Optimal weights for MMSE space-time equalizer of multicode CDMA system
KR100598119B1 (en) * 2005-01-17 2006-07-10 삼성전자주식회사 Display apparatus and control method thereof
EP1839445A2 (en) * 2005-01-18 2007-10-03 Thomson Licensing Method and apparatus for estimating channel induced distortion
KR100755688B1 (en) * 2005-02-02 2007-09-05 삼성전자주식회사 Apparatus and method for Error concealment
US7526142B2 (en) * 2005-02-22 2009-04-28 Sheraizin Vitaly S Enhancement of decompressed video
US8514933B2 (en) * 2005-03-01 2013-08-20 Qualcomm Incorporated Adaptive frame skipping techniques for rate controlled video encoding
US20060198441A1 (en) * 2005-03-02 2006-09-07 Hua-Chang Chi Motion detection method for detecting motion objects in video frames generated from a video surveillance system
AR052601A1 (en) 2005-03-10 2007-03-21 Qualcomm Inc CLASSIFICATION OF CONTENTS FOR MULTIMEDIA PROCESSING
EP1703513A1 (en) * 2005-03-15 2006-09-20 Deutsche Thomson-Brandt Gmbh Method and apparatus for encoding plural video signals as a single encoded video signal, method and and apparatus for decoding such an encoded video signal
US8223845B1 (en) * 2005-03-16 2012-07-17 Apple Inc. Multithread processing of video frames
US20060217027A1 (en) * 2005-03-25 2006-09-28 Martuccio Michael C Method and apparatus for fan expressing participation in sporting events
US7982757B2 (en) * 2005-04-01 2011-07-19 Digital Multitools Inc. Method for reducing noise and jitter effects in KVM systems
US20060230428A1 (en) * 2005-04-11 2006-10-12 Rob Craig Multi-player video game system
US20060233237A1 (en) * 2005-04-15 2006-10-19 Apple Computer, Inc. Single pass constrained constant bit-rate encoding
US20060268996A1 (en) * 2005-05-13 2006-11-30 Sethi Sumeet S Error recovery using in band error patterns
WO2006123331A2 (en) * 2005-05-16 2006-11-23 Human Monitoring Ltd Monitoring method and device
JP2008541649A (en) * 2005-05-20 2008-11-20 エヌエックスピー ビー ヴィ Image coding apparatus using refresh map
US8442126B1 (en) 2005-06-14 2013-05-14 Apple Inc. Synchronizing audio and video content through buffer wrappers
US8118676B2 (en) * 2005-07-08 2012-02-21 Activevideo Networks, Inc. Video game system using pre-encoded macro-blocks
US8270439B2 (en) * 2005-07-08 2012-09-18 Activevideo Networks, Inc. Video game system using pre-encoded digital audio mixing
US8284842B2 (en) * 2005-07-08 2012-10-09 Activevideo Networks, Inc. Video game system using pre-encoded macro-blocks and a reference grid
US9061206B2 (en) * 2005-07-08 2015-06-23 Activevideo Networks, Inc. Video game system using pre-generated motion vectors
US7587098B2 (en) * 2005-07-14 2009-09-08 Mavs Lab. Inc. Pixel data generating method
US8774272B1 (en) * 2005-07-15 2014-07-08 Geo Semiconductor Inc. Video quality by controlling inter frame encoding according to frame position in GOP
US8074248B2 (en) 2005-07-26 2011-12-06 Activevideo Networks, Inc. System and method for providing video content associated with a source image to a television in a communication network
US7944967B2 (en) * 2005-07-28 2011-05-17 Delphi Technologies, Inc. Technique for addressing frame loss in a video stream
US20070030894A1 (en) * 2005-08-03 2007-02-08 Nokia Corporation Method, device, and module for improved encoding mode control in video encoding
US7286498B1 (en) * 2005-08-09 2007-10-23 H-Itt, Llc Validation method and data structures for wireless communications
JP4264656B2 (en) * 2005-08-11 2009-05-20 ソニー株式会社 Encoding apparatus and method, program, and recording medium
US9077960B2 (en) 2005-08-12 2015-07-07 Microsoft Corporation Non-zero coefficient block pattern coding
US20070036227A1 (en) * 2005-08-15 2007-02-15 Faisal Ishtiaq Video encoding system and method for providing content adaptive rate control
JP2009507412A (en) * 2005-09-01 2009-02-19 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ Method and apparatus for video error resilience encoding / decoding
US7992074B2 (en) 2005-09-13 2011-08-02 Mitsubishi Electric Corporation Decoding device
US20070120969A1 (en) * 2005-09-15 2007-05-31 Alpha Omega International Audio visual communication system and method
US7676591B2 (en) * 2005-09-22 2010-03-09 Packet Video Corporation System and method for transferring multiple data channels
US8879857B2 (en) 2005-09-27 2014-11-04 Qualcomm Incorporated Redundant data encoding methods and device
US8427578B2 (en) * 2005-10-14 2013-04-23 Broadcom Corporation Method and system for frame rate adaptation
US8654848B2 (en) * 2005-10-17 2014-02-18 Qualcomm Incorporated Method and apparatus for shot detection in video streaming
US8948260B2 (en) 2005-10-17 2015-02-03 Qualcomm Incorporated Adaptive GOP structure in video streaming
US7916796B2 (en) * 2005-10-19 2011-03-29 Freescale Semiconductor, Inc. Region clustering based error concealment for video data
WO2007050680A2 (en) * 2005-10-25 2007-05-03 William Marsh Rice University Method and apparatus for on-line compressed sensing
CN100466725C (en) * 2005-11-03 2009-03-04 华为技术有限公司 Multimedia communication method and terminal thereof
US9210447B2 (en) * 2005-12-07 2015-12-08 Thomson Licensing Llc Method and apparatus for video error concealment using reference frame selection rules
US20070140353A1 (en) * 2005-12-19 2007-06-21 Sharp Laboratories Of America, Inc. Intra prediction skipping in mode selection for video compression
EP1969857B1 (en) * 2006-01-05 2012-03-28 Telefonaktiebolaget LM Ericsson (publ) Media container file management
WO2007081150A1 (en) * 2006-01-09 2007-07-19 Electronics And Telecommunications Research Institute Method defining nal unit type and system of transmission bitstream and redundant slice coding
TWI309529B (en) * 2006-01-19 2009-05-01 Avermedia Tech Inc Multi-bit stream of multimedia data processing
US8861585B2 (en) * 2006-01-20 2014-10-14 Qualcomm Incorporated Method and apparatus for error resilience algorithms in wireless video communication
US8325822B2 (en) * 2006-01-20 2012-12-04 Qualcomm Incorporated Method and apparatus for determining an encoding method based on a distortion value related to error concealment
US7979059B2 (en) * 2006-02-06 2011-07-12 Rockefeller Alfred G Exchange of voice and video between two cellular or wireless telephones
KR100846787B1 (en) * 2006-02-15 2008-07-16 삼성전자주식회사 Method and apparatus for importing transport stream
US7555570B2 (en) 2006-02-17 2009-06-30 Avocent Huntsville Corporation Device and method for configuring a target device
US8718147B2 (en) * 2006-02-17 2014-05-06 Avocent Huntsville Corporation Video compression algorithm
US8185921B2 (en) 2006-02-28 2012-05-22 Sony Corporation Parental control of displayed content using closed captioning
US8189686B2 (en) * 2006-03-03 2012-05-29 David John Boyes Systems and methods for visualizing errors in video signals
FR2898459B1 (en) * 2006-03-08 2008-09-05 Canon Kk METHOD AND APPARATUS FOR RECEIVING IMAGES HAVING FOUND LOSS DURING TRANSMISSION
US9131164B2 (en) 2006-04-04 2015-09-08 Qualcomm Incorporated Preprocessor method and apparatus
EP1843587A1 (en) 2006-04-05 2007-10-10 STMicroelectronics S.r.l. Method for the frame-rate conversion of a digital video signal and related apparatus
US7577898B2 (en) * 2006-04-10 2009-08-18 At&T Intellectual Property I, L.P. System and method of correcting video data errors
JP4730183B2 (en) * 2006-04-17 2011-07-20 株式会社日立製作所 Video display device
US7714838B2 (en) * 2006-04-27 2010-05-11 Research In Motion Limited Handheld electronic device having hidden sound openings offset from an audio source
TW200743386A (en) * 2006-04-27 2007-11-16 Koninkl Philips Electronics Nv Method and apparatus for encoding/transcoding and decoding
CA2650663A1 (en) * 2006-04-28 2007-11-08 Avocent Corporation Dvc delta commands
US8798172B2 (en) * 2006-05-16 2014-08-05 Samsung Electronics Co., Ltd. Method and apparatus to conceal error in decoded audio signal
JP4692388B2 (en) * 2006-05-24 2011-06-01 ソニー株式会社 Data processing apparatus and data processing method
CN100548051C (en) * 2006-05-25 2009-10-07 联想(北京)有限公司 Video decoding/encoding device and method and system
US20080034396A1 (en) * 2006-05-30 2008-02-07 Lev Zvi H System and method for video distribution and billing
GB2438660B (en) * 2006-06-02 2011-03-30 Tandberg Television Asa Recursive filter system for a video signal
GB2438905B (en) * 2006-06-07 2011-08-24 Tandberg Television Asa Temporal noise analysis of a video signal
US8559501B2 (en) * 2006-06-09 2013-10-15 Thomson Licensing Method and apparatus for adaptively determining a bit budget for encoding video pictures
FR2903270B1 (en) * 2006-06-30 2008-08-29 Canon Kk METHOD AND DEVICE FOR ENCODING AN IMAGE SEQUENCE, TELECOMMUNICATION SYSTEM COMPRISING SUCH A DEVICE AND PROGRAM USING SUCH A METHOD
FR2903556B1 (en) * 2006-07-04 2008-10-03 Canon Kk METHODS AND DEVICES FOR ENCODING AND DECODING IMAGES, A TELECOMMUNICATIONS SYSTEM COMPRISING SUCH DEVICES AND COMPUTER PROGRAMS USING SUCH METHODS
KR100790149B1 (en) * 2006-07-27 2008-01-02 삼성전자주식회사 Rate control of scene-changed video encoder
KR100834625B1 (en) * 2006-07-27 2008-06-02 삼성전자주식회사 Real-time scene-change detection for rate control of video encoder
KR100790150B1 (en) * 2006-07-28 2008-01-02 삼성전자주식회사 Video encoder and method for encoding video data frame
JP2008042332A (en) * 2006-08-02 2008-02-21 Toshiba Corp Interpolation frame preparation method and interpolation frame preparation device
US8238442B2 (en) * 2006-08-25 2012-08-07 Sony Computer Entertainment Inc. Methods and apparatus for concealing corrupted blocks of video data
US8699561B2 (en) * 2006-08-25 2014-04-15 Sony Computer Entertainment Inc. System and methods for detecting and handling errors in a multi-threaded video data decoder
US8135063B2 (en) * 2006-09-08 2012-03-13 Mediatek Inc. Rate control method with frame-layer bit allocation and video encoder
US8379733B2 (en) * 2006-09-26 2013-02-19 Qualcomm Incorporated Efficient video packetization methods for packet-switched video telephony applications
EP2067358A2 (en) * 2006-09-28 2009-06-10 Thomson Licensing Method for rho-domain frame level bit allocation for effective rate control and enhanced video coding quality
US8509313B2 (en) * 2006-10-10 2013-08-13 Texas Instruments Incorporated Video error concealment
JP4851911B2 (en) * 2006-10-23 2012-01-11 富士通株式会社 Encoding apparatus, encoding program, and encoding method
US8218641B2 (en) * 2006-10-31 2012-07-10 Sony Computer Entertainment Inc. Picture encoding using same-picture reference for pixel reconstruction
JPWO2008053557A1 (en) * 2006-11-02 2010-02-25 パイオニア株式会社 Moving image re-encoding device, moving image re-encoding method, moving image re-encoding program, and recording medium storing moving image re-encoding program
US20090180546A1 (en) 2008-01-09 2009-07-16 Rodriguez Arturo A Assistance for processing pictures in concatenated video streams
US8873932B2 (en) 2007-12-11 2014-10-28 Cisco Technology, Inc. Inferential processing to ascertain plural levels of picture interdependencies
US20080115175A1 (en) * 2006-11-13 2008-05-15 Rodriguez Arturo A System and method for signaling characteristics of pictures' interdependencies
TWI339073B (en) * 2006-11-13 2011-03-11 Univ Nat Chiao Tung Video coding method using image data skipping
US8875199B2 (en) 2006-11-13 2014-10-28 Cisco Technology, Inc. Indicating picture usefulness for playback optimization
US8416859B2 (en) 2006-11-13 2013-04-09 Cisco Technology, Inc. Signalling and extraction in compressed video of pictures belonging to interdependency tiers
EP1924097A1 (en) * 2006-11-14 2008-05-21 Sony Deutschland Gmbh Motion and scene change detection using color components
WO2008079503A2 (en) * 2006-12-19 2008-07-03 Motorola, Inc. Method and apparatus for adaptive error resilience for video decoders
FR2910211A1 (en) * 2006-12-19 2008-06-20 Canon Kk METHODS AND DEVICES FOR RE-SYNCHRONIZING A DAMAGED VIDEO STREAM
JP5746811B2 (en) * 2006-12-21 2015-07-08 味の素株式会社 Colorectal cancer evaluation method, colorectal cancer evaluation device, colorectal cancer evaluation method, colorectal cancer evaluation system, colorectal cancer evaluation program, and recording medium
CN102176751B (en) * 2006-12-27 2013-12-25 松下电器产业株式会社 Moving picture decoding apparatus and method
US7895502B2 (en) * 2007-01-04 2011-02-22 International Business Machines Corporation Error control coding methods for memories with subline accesses
EP3145200A1 (en) 2007-01-12 2017-03-22 ActiveVideo Networks, Inc. Mpeg objects and systems and methods for using mpeg objects
US9826197B2 (en) 2007-01-12 2017-11-21 Activevideo Networks, Inc. Providing television broadcasts over a managed network and interactive content over an unmanaged network to a client device
US8494049B2 (en) * 2007-04-09 2013-07-23 Cisco Technology, Inc. Long term reference frame management with error video feedback for compressed video communication
FR2915342A1 (en) * 2007-04-20 2008-10-24 Canon Kk VIDEO ENCODING METHOD AND DEVICE
GB0708440D0 (en) * 2007-05-02 2007-06-06 Film Night Ltd Data transmission
US7978669B2 (en) * 2007-05-09 2011-07-12 Cisco Technology, Inc. Methods and apparatus for efficient MPEG transmission over 802.11
US10715834B2 (en) 2007-05-10 2020-07-14 Interdigital Vc Holdings, Inc. Film grain simulation based on pre-computed transform coefficients
US8300699B2 (en) * 2007-05-31 2012-10-30 Qualcomm Incorporated System, method, and computer-readable medium for reducing required throughput in an ultra-wideband system
JP4833923B2 (en) * 2007-06-15 2011-12-07 富士通セミコンダクター株式会社 Transcoding device, transcoder, decoder and transcoding method
US8171030B2 (en) 2007-06-18 2012-05-01 Zeitera, Llc Method and apparatus for multi-dimensional content search and video identification
US8605779B2 (en) 2007-06-20 2013-12-10 Microsoft Corporation Mechanisms to conceal real time video artifacts caused by frame loss
KR20090000502A (en) * 2007-06-28 2009-01-07 삼성전자주식회사 Method and apparatus adaptive to characteristics of the adjacent blocks of lost block
US7962640B2 (en) * 2007-06-29 2011-06-14 The Chinese University Of Hong Kong Systems and methods for universal real-time media transcoding
US8254455B2 (en) * 2007-06-30 2012-08-28 Microsoft Corporation Computing collocated macroblock information for direct mode macroblocks
WO2009012302A1 (en) * 2007-07-16 2009-01-22 Telchemy, Incorporated Method and system for viewer quality estimation of packet video streams
DE102007035262B4 (en) 2007-07-27 2018-05-24 Texas Instruments Deutschland Gmbh A receiver and method for handling a data packet stream in the event of a physical layer failure
JPWO2009017105A1 (en) * 2007-07-30 2010-10-21 日本電気株式会社 Communication terminal, distribution system, conversion method, and program
US8804845B2 (en) 2007-07-31 2014-08-12 Cisco Technology, Inc. Non-enhancing media redundancy coding for mitigating transmission impairments
US8958486B2 (en) 2007-07-31 2015-02-17 Cisco Technology, Inc. Simultaneous processing of media and redundancy streams for mitigating impairments
US8023562B2 (en) * 2007-09-07 2011-09-20 Vanguard Software Solutions, Inc. Real-time video coding/decoding
US7769015B2 (en) * 2007-09-11 2010-08-03 Liquid Computing Corporation High performance network adapter (HPNA)
US7802062B2 (en) 2007-09-28 2010-09-21 Microsoft Corporation Non-blocking variable size recyclable buffer management
KR100928324B1 (en) * 2007-10-02 2009-11-25 주식회사 아이브이넷 Operation method of frame buffer memory for recovering compressed video and decoding device suitable for this
US20090103617A1 (en) * 2007-10-22 2009-04-23 The Hong Kong University Of Science And Technology Efficient error recovery with intra-refresh
KR20100106327A (en) 2007-11-16 2010-10-01 디브이엑스, 인크. Hierarchical and reduced index structures for multimedia files
AU2007237313A1 (en) * 2007-12-03 2009-06-18 Canon Kabushiki Kaisha Improvement for error correction in distributed vdeo coding
TW200952494A (en) * 2007-12-05 2009-12-16 Onlive Inc Method for multicasting views of real-time streaming interactive video
TW200943975A (en) * 2008-01-09 2009-10-16 Nokia Corp Systems and methods for media container file generation
US7916657B2 (en) * 2008-01-22 2011-03-29 At&T Intellectual Property Ii, L.P. Network performance and reliability evaluation taking into account abstract components
US9357233B2 (en) * 2008-02-26 2016-05-31 Qualcomm Incorporated Video decoder error handling
US7940777B2 (en) * 2008-02-26 2011-05-10 Cisco Technology, Inc. Loss-free packet networks
US8416858B2 (en) 2008-02-29 2013-04-09 Cisco Technology, Inc. Signalling picture encoding schemes and associated picture properties
US20090231439A1 (en) * 2008-03-14 2009-09-17 Arkady Kopansky Method for Propagating Data Through a Video Stream
KR101431545B1 (en) * 2008-03-17 2014-08-20 삼성전자주식회사 Method and apparatus for Video encoding and decoding
US8861598B2 (en) * 2008-03-19 2014-10-14 Cisco Technology, Inc. Video compression using search techniques of long-term reference memory
US8406296B2 (en) 2008-04-07 2013-03-26 Qualcomm Incorporated Video refresh adaptation algorithms responsive to error feedback
US20090268097A1 (en) * 2008-04-28 2009-10-29 Siou-Shen Lin Scene change detection method and related apparatus according to summation results of block matching costs associated with at least two frames
US8254469B2 (en) * 2008-05-07 2012-08-28 Kiu Sha Management Liability Company Error concealment for frame loss in multiple description coding
US7860996B2 (en) 2008-05-30 2010-12-28 Microsoft Corporation Media streaming with seamless ad insertion
WO2009152450A1 (en) 2008-06-12 2009-12-17 Cisco Technology, Inc. Picture interdependencies signals in context of mmco to assist stream manipulation
US8705631B2 (en) 2008-06-17 2014-04-22 Cisco Technology, Inc. Time-shifted transport of multi-latticed video for resiliency from burst-error effects
US8971402B2 (en) 2008-06-17 2015-03-03 Cisco Technology, Inc. Processing of impaired and incomplete multi-latticed video streams
US8699578B2 (en) 2008-06-17 2014-04-15 Cisco Technology, Inc. Methods and systems for processing multi-latticed video streams
US20130022114A1 (en) * 2008-06-23 2013-01-24 Mediatek Inc. Method and related apparatuses for decoding multimedia data
US8494058B2 (en) * 2008-06-23 2013-07-23 Mediatek Inc. Video/image processing apparatus with motion estimation sharing, and related method and machine readable medium
US8347408B2 (en) * 2008-06-30 2013-01-01 Cisco Technology, Inc. Matching of unknown video content to protected video content
US8259177B2 (en) * 2008-06-30 2012-09-04 Cisco Technology, Inc. Video fingerprint systems and methods
US20090327334A1 (en) * 2008-06-30 2009-12-31 Rodriguez Arturo A Generating Measures of Video Sequences to Detect Unauthorized Use
KR20100004792A (en) * 2008-07-04 2010-01-13 삼성전자주식회사 Method for storing defected information, information processing apparatus capable of storing defected information, information storing apparatus for transmitting defected information in storable form, and medium capable of being read by information processing apparatus, in which a software needed for storing defected information is recorded
EP2141703B1 (en) * 2008-07-04 2013-09-04 Samsung Electronics Co., Ltd. Methods and apparatus for copying data
FR2934453B1 (en) * 2008-07-22 2010-10-15 Canon Kk ERROR MASKING METHOD AND DEVICE
JP5164714B2 (en) * 2008-07-24 2013-03-21 キヤノン株式会社 Transmitting apparatus and method, program
ES2387869T3 (en) * 2008-07-30 2012-10-03 FRANCE TéLéCOM Multi-channel audio data reconstruction
WO2010017166A2 (en) 2008-08-04 2010-02-11 Dolby Laboratories Licensing Corporation Overlapped block disparity estimation and compensation architecture
US8254441B2 (en) * 2008-08-18 2012-08-28 Sprint Communications Company L.P. Video streaming based upon wireless quality
US8239900B1 (en) 2008-08-27 2012-08-07 Clearwire Ip Holdings Llc Video bursting based upon wireless device location
US8270307B2 (en) * 2008-09-05 2012-09-18 Cisco Technology, Inc. Network-adaptive preemptive repair in real-time video
US8275046B2 (en) * 2008-09-19 2012-09-25 Texas Instruments Incorporated Fast macroblock structure decision using SAD discrepancy and its prediction mode
US9237034B2 (en) 2008-10-21 2016-01-12 Iii Holdings 1, Llc Methods and systems for providing network access redundancy
US9100535B2 (en) 2008-10-22 2015-08-04 Entropic Communications, Llc Device and method for motion estimation and compensation
US20100104003A1 (en) * 2008-10-24 2010-04-29 Manufacturing Resources International Inc. System and method for securely transmitting video data
EP2356812B1 (en) 2008-11-12 2015-06-10 Cisco Technology, Inc. Processing of a video program having plural processed representations of a single video signal for reconstruction and output
US20100158130A1 (en) 2008-12-22 2010-06-24 Mediatek Inc. Video decoding method
JP4600574B2 (en) 2009-01-07 2010-12-15 日本電気株式会社 Moving picture decoding apparatus, moving picture decoding method, and program
US20100195742A1 (en) * 2009-02-02 2010-08-05 Mediatek Inc. Error concealment method and apparatus
US8189666B2 (en) 2009-02-02 2012-05-29 Microsoft Corporation Local picture identifier and computation of co-located information
US9812047B2 (en) 2010-02-25 2017-11-07 Manufacturing Resources International, Inc. System and method for remotely monitoring the operating life of electronic displays
US8326131B2 (en) 2009-02-20 2012-12-04 Cisco Technology, Inc. Signalling of decodable sub-sequences
US8782261B1 (en) 2009-04-03 2014-07-15 Cisco Technology, Inc. System and method for authorization of segment boundary notifications
US20100269147A1 (en) 2009-04-15 2010-10-21 Echostar Technologies Llc Video stream index generation at a video content transmitter
US20120050475A1 (en) 2009-05-01 2012-03-01 Dong Tian Reference picture lists for 3dv
US8949883B2 (en) 2009-05-12 2015-02-03 Cisco Technology, Inc. Signalling buffer characteristics for splicing operations of video streams
US8279926B2 (en) 2009-06-18 2012-10-02 Cisco Technology, Inc. Dynamic streaming with latticed representations of video
US8665964B2 (en) * 2009-06-30 2014-03-04 Qualcomm Incorporated Video coding based on first order prediction and pre-defined second order prediction mode
US9654792B2 (en) 2009-07-03 2017-05-16 Intel Corporation Methods and systems for motion vector derivation at a video decoder
US20110002387A1 (en) * 2009-07-03 2011-01-06 Yi-Jen Chiu Techniques for motion estimation
US8462852B2 (en) * 2009-10-20 2013-06-11 Intel Corporation Methods and apparatus for adaptively choosing a search range for motion estimation
US8917769B2 (en) * 2009-07-03 2014-12-23 Intel Corporation Methods and systems to estimate motion based on reconstructed reference frames at a video decoder
CN102474384B (en) * 2009-07-15 2016-05-04 诺基亚技术有限公司 A kind of device
US8194862B2 (en) * 2009-07-31 2012-06-05 Activevideo Networks, Inc. Video game system with mixing of independent pre-encoded digital audio bitstreams
US8582952B2 (en) * 2009-09-15 2013-11-12 Apple Inc. Method and apparatus for identifying video transitions
US20110064129A1 (en) * 2009-09-16 2011-03-17 Broadcom Corporation Video capture and generation at variable frame rates
WO2011059419A1 (en) 2009-11-13 2011-05-19 Thomson Licensing Preamble identification in a mobile dtv system
WO2011059420A1 (en) * 2009-11-13 2011-05-19 Thomson Licensing Joint preamble and code rate identifier in a mobile dtv system
EP2323404A1 (en) * 2009-11-17 2011-05-18 Research In Motion Limited Additional information for in-loop video deblocking
GB2475739A (en) * 2009-11-30 2011-06-01 Nokia Corp Video decoding with error concealment dependent upon video scene change.
KR101345098B1 (en) * 2009-12-18 2013-12-26 한국전자통신연구원 Apparatus and method for assessing image quality based on real-time
TWI535028B (en) 2009-12-21 2016-05-21 半導體能源研究所股份有限公司 Thin film transistor
US8588297B2 (en) * 2009-12-23 2013-11-19 Oracle America, Inc. Quantization parameter prediction
US8476744B2 (en) 2009-12-28 2013-07-02 Semiconductor Energy Laboratory Co., Ltd. Thin film transistor with channel including microcrystalline and amorphous semiconductor regions
US8925024B2 (en) 2009-12-31 2014-12-30 The Nielsen Company (Us), Llc Methods and apparatus to detect commercial advertisements associated with media presentations
RU2540846C2 (en) 2010-01-11 2015-02-10 Телефонактиеболагет Л М Эрикссон (Пабл) Video quality assessment technology
KR101675118B1 (en) 2010-01-14 2016-11-10 삼성전자 주식회사 Method and apparatus for video encoding considering order of skip and split, and method and apparatus for video decoding considering order of skip and split
JP2013519295A (en) * 2010-02-03 2013-05-23 トムソン ライセンシング Effective replacement data in encoded video.
WO2011100347A2 (en) * 2010-02-09 2011-08-18 Vanguard Software Solutions, Inc. Video sequence encoding system and algorithms
US8681873B2 (en) * 2010-02-19 2014-03-25 Skype Data compression for video
US9313526B2 (en) * 2010-02-19 2016-04-12 Skype Data compression for video
US9078009B2 (en) * 2010-02-19 2015-07-07 Skype Data compression for video utilizing non-translational motion information
US9819358B2 (en) * 2010-02-19 2017-11-14 Skype Entropy encoding based on observed frequency
US9609342B2 (en) * 2010-02-19 2017-03-28 Skype Compression for frames of a video signal using selected candidate blocks
JP5583992B2 (en) * 2010-03-09 2014-09-03 パナソニック株式会社 Signal processing device
US20110222837A1 (en) * 2010-03-11 2011-09-15 Cisco Technology, Inc. Management of picture referencing in video streams for plural playback modes
US20110255596A1 (en) * 2010-04-15 2011-10-20 Himax Technologies Limited Frame rate up conversion system and method
JP2012010263A (en) * 2010-06-28 2012-01-12 Sony Corp Encoding device, imaging device, encoding/transmitting system and encoding method
WO2012083487A1 (en) 2010-12-21 2012-06-28 Intel Corporation System and method for enhanced dmvd processing
WO2012031269A1 (en) * 2010-09-03 2012-03-08 Loglogic, Inc. Random access data compression
GB2483282B (en) * 2010-09-03 2017-09-13 Advanced Risc Mach Ltd Data compression and decompression using relative and absolute delta values
WO2012030262A1 (en) * 2010-09-03 2012-03-08 Telefonaktiebolaget Lm Ericsson (Publ) Co-compression and co-decompression of data values
US9883161B2 (en) 2010-09-14 2018-01-30 Thomson Licensing Compression methods and apparatus for occlusion data
CA2814070A1 (en) 2010-10-14 2012-04-19 Activevideo Networks, Inc. Streaming digital video between video devices using a cable television system
US8419547B1 (en) * 2010-11-04 2013-04-16 Wms Gaming, Inc. Iterative XOR-matrix forward error correction for gaming
US10802763B2 (en) * 2010-11-29 2020-10-13 Pure Storage, Inc. Remote storage verification
US11307930B1 (en) 2010-11-29 2022-04-19 Pure Storage, Inc. Optimized selection of participants in distributed data rebuild/verification
US8914534B2 (en) 2011-01-05 2014-12-16 Sonic Ip, Inc. Systems and methods for adaptive bitrate streaming of media stored in matroska container files using hypertext transfer protocol
JP5878295B2 (en) * 2011-01-13 2016-03-08 ソニー株式会社 Image processing apparatus, image processing method, and program
US8838680B1 (en) 2011-02-08 2014-09-16 Google Inc. Buffer objects for web-based configurable pipeline media processing
KR101803970B1 (en) * 2011-03-16 2017-12-28 삼성전자주식회사 Method and apparatus for composing content
US9204203B2 (en) 2011-04-07 2015-12-01 Activevideo Networks, Inc. Reduction of latency in video distribution networks using adaptive bit rates
US8681866B1 (en) 2011-04-28 2014-03-25 Google Inc. Method and apparatus for encoding video by downsampling frame resolution
US9106787B1 (en) 2011-05-09 2015-08-11 Google Inc. Apparatus and method for media transmission bandwidth control using bandwidth estimation
WO2012170904A2 (en) * 2011-06-10 2012-12-13 Bytemobile, Inc. Adaptive bitrate management on progressive download with indexed media files
US8780981B2 (en) 2011-06-27 2014-07-15 Panasonic Intellectual Property Corporation Of America Image coding method, image decoding method, image coding apparatus, image decoding apparatus, and image coding-decoding apparatus
WO2013002342A1 (en) * 2011-06-30 2013-01-03 ソニー株式会社 Image processing device and method
US8767824B2 (en) 2011-07-11 2014-07-01 Sharp Kabushiki Kaisha Video decoder parallelization for tiles
US10498359B2 (en) * 2011-07-14 2019-12-03 Microsoft Technology Licensing, Llc Correction data
US8984156B2 (en) * 2011-07-21 2015-03-17 Salesforce.Com, Inc. Multi-party mesh conferencing with stream processing
JP5558431B2 (en) * 2011-08-15 2014-07-23 株式会社東芝 Image processing apparatus, method, and program
US10659724B2 (en) * 2011-08-24 2020-05-19 Ati Technologies Ulc Method and apparatus for providing dropped picture image processing
US9955195B2 (en) * 2011-08-30 2018-04-24 Divx, Llc Systems and methods for encoding and streaming video encoded using a plurality of maximum bitrate levels
US8818171B2 (en) 2011-08-30 2014-08-26 Kourosh Soroushian Systems and methods for encoding alternative streams of video for playback on playback devices having predetermined display aspect ratios and network connection maximum data rates
US9467708B2 (en) 2011-08-30 2016-10-11 Sonic Ip, Inc. Selection of resolutions for seamless resolution switching of multimedia content
US8856624B1 (en) 2011-10-27 2014-10-07 Google Inc. Method and apparatus for dynamically generating error correction
US8693551B2 (en) 2011-11-16 2014-04-08 Vanguard Software Solutions, Inc. Optimal angular intra prediction for block-based video coding
US9490850B1 (en) 2011-11-28 2016-11-08 Google Inc. Method and apparatus for decoding packetized data
WO2013106390A1 (en) 2012-01-09 2013-07-18 Activevideo Networks, Inc. Rendering of an interactive lean-backward user interface on a television
US8850054B2 (en) * 2012-01-17 2014-09-30 International Business Machines Corporation Hypertext transfer protocol live streaming
US9531990B1 (en) 2012-01-21 2016-12-27 Google Inc. Compound prediction using multiple sources or prediction modes
US8737824B1 (en) 2012-03-09 2014-05-27 Google Inc. Adaptively encoding a media stream with compound prediction
US9489827B2 (en) 2012-03-12 2016-11-08 Cisco Technology, Inc. System and method for distributing content in a video surveillance network
US9489659B1 (en) * 2012-04-02 2016-11-08 Cisco Technology, Inc. Progressive sharing during a collaboration session
US9800945B2 (en) 2012-04-03 2017-10-24 Activevideo Networks, Inc. Class-based intelligent multiplexing over unmanaged networks
US9123084B2 (en) 2012-04-12 2015-09-01 Activevideo Networks, Inc. Graphical application integration with MPEG objects
US9185429B1 (en) 2012-04-30 2015-11-10 Google Inc. Video encoding and decoding using un-equal error protection
US20130287100A1 (en) * 2012-04-30 2013-10-31 Wooseung Yang Mechanism for facilitating cost-efficient and low-latency encoding of video streams
US9049349B2 (en) * 2012-05-16 2015-06-02 Cisco Technology, Inc. System and method for video recording and retention in a network
US9532080B2 (en) 2012-05-31 2016-12-27 Sonic Ip, Inc. Systems and methods for the reuse of encoding information in encoding alternative streams of video data
US8819525B1 (en) 2012-06-14 2014-08-26 Google Inc. Error concealment guided robustness
US9185414B1 (en) 2012-06-29 2015-11-10 Google Inc. Video encoding using variance
JP2014027448A (en) 2012-07-26 2014-02-06 Sony Corp Information processing apparatus, information processing metho, and program
US10034023B1 (en) 2012-07-30 2018-07-24 Google Llc Extended protection of digital video streams
US9256803B2 (en) 2012-09-14 2016-02-09 Palo Alto Research Center Incorporated Automatic detection of persistent changes in naturally varying scenes
US9491487B2 (en) * 2012-09-25 2016-11-08 Apple Inc. Error resilient management of picture order count in predictive coding systems
WO2014061925A1 (en) * 2012-09-28 2014-04-24 (주)휴맥스 Method for adaptively transmitting fec parity data using cross-layer optimization
US9386326B2 (en) * 2012-10-05 2016-07-05 Nvidia Corporation Video decoding error concealment techniques
CN103780801A (en) * 2012-10-25 2014-05-07 特克特朗尼克公司 Heuristic method for scene cut detection in digital baseband video
US10015486B2 (en) * 2012-10-26 2018-07-03 Intel Corporation Enhanced video decoding with application layer forward error correction
US10341047B2 (en) * 2013-10-31 2019-07-02 Hewlett Packard Enterprise Development Lp Method and system for controlling the forwarding of error correction data
US9307235B2 (en) * 2012-12-03 2016-04-05 Vixs Systems, Inc. Video encoding system with adaptive hierarchical B-frames and method for use therewith
US10349069B2 (en) * 2012-12-11 2019-07-09 Sony Interactive Entertainment Inc. Software hardware hybrid video encoder
US9106922B2 (en) 2012-12-19 2015-08-11 Vanguard Software Solutions, Inc. Motion estimation engine for video encoding
US9191457B2 (en) 2012-12-31 2015-11-17 Sonic Ip, Inc. Systems, methods, and media for controlling delivery of content
US9628790B1 (en) 2013-01-03 2017-04-18 Google Inc. Adaptive composite intra prediction for image and video compression
US9172740B1 (en) 2013-01-15 2015-10-27 Google Inc. Adjustable buffer remote access
US9146808B1 (en) * 2013-01-24 2015-09-29 Emulex Corporation Soft error protection for content addressable memory
US9311692B1 (en) 2013-01-25 2016-04-12 Google Inc. Scalable buffer remote access
US9225979B1 (en) 2013-01-30 2015-12-29 Google Inc. Remote access encoding
US9177245B2 (en) 2013-02-08 2015-11-03 Qualcomm Technologies Inc. Spiking network apparatus and method with bimodal spike-timing dependent plasticity
JP6182888B2 (en) * 2013-02-12 2017-08-23 三菱電機株式会社 Image encoding device
US9357210B2 (en) 2013-02-28 2016-05-31 Sonic Ip, Inc. Systems and methods of encoding multiple video streams for adaptive bitrate streaming
US8928815B1 (en) * 2013-03-13 2015-01-06 Hrl Laboratories, Llc System and method for outdoor scene change detection
US10275128B2 (en) 2013-03-15 2019-04-30 Activevideo Networks, Inc. Multiple-mode system and method for providing user selectable video content
JP5838351B2 (en) * 2013-03-26 2016-01-06 パナソニックＩｐマネジメント株式会社 Video receiving apparatus and image recognition method for received video
CN103237108B (en) * 2013-05-13 2015-11-25 百度在线网络技术（北京）有限公司 For method of testing and the test terminal of mobile terminal
US9374578B1 (en) 2013-05-23 2016-06-21 Google Inc. Video coding using combined inter and intra predictors
US9219922B2 (en) 2013-06-06 2015-12-22 Activevideo Networks, Inc. System and method for exploiting scene graph information in construction of an encoded video sequence
EP3005712A1 (en) 2013-06-06 2016-04-13 ActiveVideo Networks, Inc. Overlay rendering of user interface onto source video
US9294785B2 (en) 2013-06-06 2016-03-22 Activevideo Networks, Inc. System and method for exploiting scene graph information in construction of an encoded video sequence
US9185275B2 (en) * 2013-07-09 2015-11-10 Lenovo (Singapore) Pte. Ltd. Control flap
EP3028468B1 (en) * 2013-07-30 2020-09-09 Robert Bosch GmbH Adaptive methods for wireless camera communication
US9866653B2 (en) * 2013-08-02 2018-01-09 Hitachi, Ltd. Data transfer system and method
US9609343B1 (en) 2013-12-20 2017-03-28 Google Inc. Video coding using compound prediction
KR102143618B1 (en) * 2014-01-17 2020-08-11 삼성전자주식회사 Method for controlling a frame rate and an electronic device
JP6248671B2 (en) * 2014-02-10 2017-12-20 富士通株式会社 Information processing apparatus, method, program, and information processing system
US9788029B2 (en) 2014-04-25 2017-10-10 Activevideo Networks, Inc. Intelligent multiplexing using class-based, multi-dimensioned decision logic for managed networks
US9723377B2 (en) 2014-04-28 2017-08-01 Comcast Cable Communications, Llc Video management
US9939253B2 (en) 2014-05-22 2018-04-10 Brain Corporation Apparatus and methods for distance estimation using multiple image sensors
US9713982B2 (en) 2014-05-22 2017-07-25 Brain Corporation Apparatus and methods for robotic operation using video imagery
US10194163B2 (en) * 2014-05-22 2019-01-29 Brain Corporation Apparatus and methods for real time estimation of differential motion in live video
US9848112B2 (en) 2014-07-01 2017-12-19 Brain Corporation Optical detection apparatus and methods
US10057593B2 (en) 2014-07-08 2018-08-21 Brain Corporation Apparatus and methods for distance estimation using stereo imagery
AU2015311069B2 (en) * 2014-09-03 2019-12-12 Electrolux Appliances Aktiebolag Method for data communication with a domestic appliance by a mobile computer device, mobile computer device and domestic appliance
US10055850B2 (en) 2014-09-19 2018-08-21 Brain Corporation Salient features tracking apparatus and methods using visual initialization
AT514851B1 (en) 2014-10-23 2019-07-15 Avl List Gmbh A method of reconstructing a data packet erroneously received in a wireless sensor network
KR101690375B1 (en) 2014-11-14 2016-12-27 영남대학교 산학협력단 Controlling apparatus for agricultural products drying machine
US9544615B2 (en) 2014-11-14 2017-01-10 Sony Corporation Method and system for processing video content
US10319408B2 (en) 2015-03-30 2019-06-11 Manufacturing Resources International, Inc. Monolithic display with separately controllable sections
KR20160131526A (en) * 2015-05-07 2016-11-16 삼성전자주식회사 System on chip, display system including the same, and operating method thereof
US10922736B2 (en) 2015-05-15 2021-02-16 Manufacturing Resources International, Inc. Smart electronic display for restaurants
US10269156B2 (en) 2015-06-05 2019-04-23 Manufacturing Resources International, Inc. System and method for blending order confirmation over menu board background
US9848222B2 (en) 2015-07-15 2017-12-19 The Nielsen Company (Us), Llc Methods and apparatus to detect spillover
US10197664B2 (en) 2015-07-20 2019-02-05 Brain Corporation Apparatus and methods for detection of objects using broadband signals
KR102056069B1 (en) 2015-09-10 2020-01-22 매뉴팩처링 리소시스 인터내셔널 인코포레이티드 System and method for systematic detection of display error
KR102453803B1 (en) * 2015-09-10 2022-10-12 삼성전자주식회사 Method and apparatus for processing image
US10756997B2 (en) 2015-09-28 2020-08-25 Cybrook Inc. Bandwidth adjustment for real-time video transmission
US10516892B2 (en) 2015-09-28 2019-12-24 Cybrook Inc. Initial bandwidth estimation for real-time video transmission
US10506257B2 (en) 2015-09-28 2019-12-10 Cybrook Inc. Method and system of video processing with back channel message management
CN105245908B (en) * 2015-10-27 2018-06-29 大连海事大学 A kind of video fault-tolerant coding method based on error correction preferred value feedback
US10506283B2 (en) 2015-11-18 2019-12-10 Cybrook Inc. Video decoding and rendering using combined jitter and frame buffer
US10506245B2 (en) * 2015-11-18 2019-12-10 Cybrook Inc. Video data processing using a ring buffer
US10798396B2 (en) 2015-12-08 2020-10-06 Samsung Display Co., Ltd. System and method for temporal differencing with variable complexity
CN107181968B (en) * 2016-03-11 2019-11-19 腾讯科技（深圳）有限公司 A kind of redundancy control method and device of video data
US10319271B2 (en) 2016-03-22 2019-06-11 Manufacturing Resources International, Inc. Cyclic redundancy check for electronic displays
CA3024512C (en) 2016-05-31 2020-12-29 Manufacturing Resources International, Inc. Electronic display remote image verification system and method
US10148989B2 (en) 2016-06-15 2018-12-04 Divx, Llc Systems and methods for encoding video content
WO2018031717A2 (en) 2016-08-10 2018-02-15 Manufacturing Resources International, Inc. Dynamic dimming led backlight for lcd array
US10785279B2 (en) 2016-12-29 2020-09-22 Facebook, Inc. Video encoding using starve mode
US10868569B2 (en) * 2017-05-08 2020-12-15 Qualcomm Incorporated PBCH signal design and efficient continuous monitoring and polar decoding
CN107169117B (en) * 2017-05-25 2020-11-10 西安工业大学 Hand-drawn human motion retrieval method based on automatic encoder and DTW
US10560910B2 (en) 2017-06-12 2020-02-11 Qualcomm Incoporated Synchronization signal for a broadcast channel
JP2019016850A (en) * 2017-07-04 2019-01-31 ヒロテック株式会社 Video transmission method, video transmission system, transmission device, and reception device
CN109413427B (en) 2017-08-17 2022-04-08 腾讯科技（深圳）有限公司 Video frame coding method and terminal
US10152275B1 (en) 2017-08-30 2018-12-11 Red Hat, Inc. Reverse order submission for pointer rings
CN107948735B (en) * 2017-12-06 2020-09-25 北京乐我无限科技有限责任公司 Video playing method and device and electronic equipment
EP3811690B1 (en) * 2018-06-20 2022-10-19 Telefonaktiebolaget LM Ericsson (publ) Method and apparatus for massive mu-mimo
US10908863B2 (en) 2018-07-12 2021-02-02 Manufacturing Resources International, Inc. System and method for providing access to co-located operations data for an electronic display
US11645029B2 (en) 2018-07-12 2023-05-09 Manufacturing Resources International, Inc. Systems and methods for remotely monitoring electronic displays
CN110972202B (en) 2018-09-28 2023-09-01 苹果公司 Mobile device content provision adjustment based on wireless communication channel bandwidth conditions
CN110972203B (en) 2018-09-28 2023-09-01 苹果公司 Electronic device content provision adjustment based on wireless communication channel bandwidth conditions
WO2020102076A1 (en) * 2018-11-13 2020-05-22 Intel Corporation Beam switching in intra-band non-contiguous carrier aggregation
JP7277586B2 (en) * 2018-12-21 2023-05-19 ホアウェイ・テクノロジーズ・カンパニー・リミテッド Method and apparatus for mode and size dependent block level limiting
WO2020164751A1 (en) 2019-02-13 2020-08-20 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Decoder and decoding method for lc3 concealment including full frame loss concealment and partial frame loss concealment
US11402940B2 (en) 2019-02-25 2022-08-02 Manufacturing Resources International, Inc. Monitoring the status of a touchscreen
US11137847B2 (en) 2019-02-25 2021-10-05 Manufacturing Resources International, Inc. Monitoring the status of a touchscreen
EP3962091B1 (en) * 2020-08-26 2024-08-28 Tata Consultancy Services Limited Methods and systems for maintaining quality of experience in real-time live video streaming
US11368251B1 (en) * 2020-12-28 2022-06-21 Aira Technologies, Inc. Convergent multi-bit feedback system
CN114630122B (en) * 2021-03-19 2023-04-28 杭州海康威视数字技术股份有限公司 Decoding and encoding method based on self-adaptive intra-frame refreshing mechanism and related equipment
KR102620281B1 (en) * 2021-05-14 2023-12-29 연세대학교 산학협력단 Apparatus and Method for Selecting Skip Frames
US11921010B2 (en) 2021-07-28 2024-03-05 Manufacturing Resources International, Inc. Display assemblies with differential pressure sensors
US11965804B2 (en) 2021-07-28 2024-04-23 Manufacturing Resources International, Inc. Display assemblies with differential pressure sensors
CN113630597B (en) * 2021-08-19 2024-01-23 随锐科技集团股份有限公司 Method and system for preventing video from losing packets irrelevant to encoding and decoding
US20230098691A1 (en) * 2021-09-29 2023-03-30 Tencent America LLC Techniques for constraint flag signaling for range extension with extended precision
US11895362B2 (en) 2021-10-29 2024-02-06 Manufacturing Resources International, Inc. Proof of play for images displayed at electronic displays
US11917269B2 (en) * 2022-01-11 2024-02-27 Tencent America LLC Multidimensional metadata for parallel processing of segmented media data
US11972672B1 (en) 2022-10-26 2024-04-30 Manufacturing Resources International, Inc. Display assemblies providing open and unlatched alerts, systems and methods for the same

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6141448A (en) * 1997-04-21 2000-10-31 Hewlett-Packard Low-complexity error-resilient coder using a block-based standard

Family Cites Families (106)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2637438B2 (en) * 1987-10-27 1997-08-06 キヤノン株式会社 Image processing device
DE68925003T2 (en) * 1988-07-14 1996-06-13 Casio Computer Co Ltd Document data processing system.
US5164828A (en) * 1990-02-26 1992-11-17 Sony Corporation Video signal transmission and method and apparatus for coding video signal used in this
US5455629A (en) * 1991-02-27 1995-10-03 Rca Thomson Licensing Corporation Apparatus for concealing errors in a digital video processing system
US5212742A (en) * 1991-05-24 1993-05-18 Apple Computer, Inc. Method and apparatus for encoding/decoding image data
JPH05115010A (en) 1991-10-22 1993-05-07 Canon Inc Picture decoder
US5141448A (en) * 1991-12-02 1992-08-25 Matrix Science Corporation Apparatus for retaining a coupling ring in non-self locking electrical connectors
GB2263373B (en) * 1992-01-09 1995-05-24 Sony Broadcast & Communication Data error concealment
FR2696026B1 (en) * 1992-09-18 1994-12-30 Sgs Thomson Microelectronics Method for masking errors in transmission of image compressed in MPEG.
JPH06111495A (en) * 1992-09-30 1994-04-22 Sony Corp Device for reproducing data
JP3255308B2 (en) * 1992-12-18 2002-02-12 ソニー株式会社 Data playback device
US5737022A (en) * 1993-02-26 1998-04-07 Kabushiki Kaisha Toshiba Motion picture error concealment using simplified motion compensation
JP3519441B2 (en) * 1993-02-26 2004-04-12 株式会社東芝 Video transmission equipment
US5442400A (en) * 1993-04-29 1995-08-15 Rca Thomson Licensing Corporation Error concealment apparatus for MPEG-like video data
JPH0775110A (en) * 1993-08-31 1995-03-17 Sony Corp Encoding method for image signal
US5771081A (en) * 1994-02-28 1998-06-23 Korea Telecommunication Authority Bit system for transmitting digital video data
JP3500724B2 (en) * 1994-09-05 2004-02-23 ソニー株式会社 Data reproducing method and data reproducing apparatus
CA2156463A1 (en) * 1994-09-05 1996-03-06 Nobuyuki Aoki Data reproducing method and data reproducing apparatus
US5550847A (en) * 1994-10-11 1996-08-27 Motorola, Inc. Device and method of signal loss recovery for realtime and/or interactive communications
US6222881B1 (en) * 1994-10-18 2001-04-24 Intel Corporation Using numbers of non-zero quantized transform signals and signal differences to determine when to encode video signals using inter-frame or intra-frame encoding
US5600663A (en) * 1994-11-16 1997-02-04 Lucent Technologies Inc. Adaptive forward error correction system
US5617149A (en) * 1994-11-22 1997-04-01 Electronics And Telecommunications Research Institute Apparatus and method for detecting scene changes using the difference of mad between image frames
JPH08214265A (en) * 1995-01-31 1996-08-20 Sony Corp Method and device for reproducing encoded data
US5621467A (en) * 1995-02-16 1997-04-15 Thomson Multimedia S.A. Temporal-spatial error concealment apparatus and method for video signal processors
US5731840A (en) * 1995-03-10 1998-03-24 Kabushiki Kaisha Toshiba Video coding/decoding apparatus which transmits different accuracy prediction levels
EP0735776B1 (en) * 1995-03-29 2004-01-28 Hitachi, Ltd. Decoder for compressed and multiplexed video and audio data
US5568200A (en) * 1995-06-07 1996-10-22 Hitachi America, Ltd. Method and apparatus for improved video display of progressively refreshed coded video
US6571361B1 (en) * 1995-09-29 2003-05-27 Kabushiki Kaisha Toshiba Encoder and decoder
US5737537A (en) * 1995-09-29 1998-04-07 Intel Corporation Two-measure block classification scheme for encoding video images
US5862153A (en) * 1995-09-29 1999-01-19 Kabushiki Kaisha Toshiba Coding apparatus and decoding apparatus for transmission/storage of information
US6415398B1 (en) * 1995-09-29 2002-07-02 Kabushiki Kaisha Toshiba Coding system and decoding system
US5778191A (en) * 1995-10-26 1998-07-07 Motorola, Inc. Method and device for error control of a macroblock-based video compression technique
US6192081B1 (en) * 1995-10-26 2001-02-20 Sarnoff Corporation Apparatus and method for selecting a coding mode in a block-based coding system
US5724369A (en) * 1995-10-26 1998-03-03 Motorola Inc. Method and device for concealment and containment of errors in a macroblock-based video codec
US6310922B1 (en) * 1995-12-12 2001-10-30 Thomson Consumer Electronics, Inc. Method and apparatus for generating variable rate synchronization signals
KR100197368B1 (en) 1995-12-23 1999-06-15 전주범 Apparatus for restoring error of image data
KR100196872B1 (en) 1995-12-23 1999-06-15 전주범 Apparatus for restoring error of image data in image decoder
US5801779A (en) * 1995-12-26 1998-09-01 C-Cube Microsystems, Inc. Rate control with panic mode
JPH09180273A (en) * 1995-12-28 1997-07-11 Toray Ind Inc Stamper for forming recording surface of optical recording medium and production of optical recording medium
KR100220678B1 (en) * 1995-12-29 1999-09-15 전주범 Method for rectifying channel errors in an image signal transmitted from a block-based encoder
JP3297293B2 (en) * 1996-03-07 2002-07-02 三菱電機株式会社 Video decoding method and video decoding device
JP3823275B2 (en) * 1996-06-10 2006-09-20 富士通株式会社 Video encoding device
USRE39115E1 (en) * 1996-07-05 2006-06-06 Matsushita Electric Industrial Co., Ltd. Method for display time stamping and synchronization of multiple video object planes
US5875199A (en) * 1996-08-22 1999-02-23 Lsi Logic Corporation Video device with reed-solomon erasure decoder and method thereof
JPH1174868A (en) * 1996-09-02 1999-03-16 Toshiba Corp Information transmission method, coder/decoder in information transmission system adopting the method, coding multiplexer/decoding inverse multiplexer
JP3011680B2 (en) * 1996-09-06 2000-02-21 株式会社東芝 Variable length coding apparatus and method
KR100501902B1 (en) * 1996-09-25 2005-10-10 주식회사 팬택앤큐리텔 Image information encoding / decoding apparatus and method
JPH10145789A (en) * 1996-11-15 1998-05-29 Oki Electric Ind Co Ltd Animation encoding and decoding method
CA2190785A1 (en) * 1996-11-20 1998-05-20 Nael Hirzalla Method of processing a video stream
KR100196840B1 (en) * 1996-12-27 1999-06-15 전주범 Apparatus for reconstucting bits error in the image decoder
US6148026A (en) * 1997-01-08 2000-11-14 At&T Corp. Mesh node coding to enable object based functionalities within a motion compensated transform video coder
EP0960532B1 (en) * 1997-02-12 2007-01-31 MediaTek Inc. Apparatus and method for optimizing the rate control in a coding system
JP3575215B2 (en) * 1997-03-05 2004-10-13 株式会社日立製作所 Packet communication method and communication terminal device
US6005980A (en) * 1997-03-07 1999-12-21 General Instrument Corporation Motion estimation and compensation of video object planes for interlaced digital video
US5991447A (en) * 1997-03-07 1999-11-23 General Instrument Corporation Prediction and coding of bi-directionally predicted video object planes for interlaced digital video
US6118817A (en) * 1997-03-14 2000-09-12 Microsoft Corporation Digital video signal encoder and encoding method having adjustable quantization
US6115420A (en) * 1997-03-14 2000-09-05 Microsoft Corporation Digital video signal encoder and encoding method
KR20050052484A (en) * 1997-03-17 2005-06-02 마츠시타 덴끼 산교 가부시키가이샤 Data processing method
US6304607B1 (en) * 1997-03-18 2001-10-16 Texas Instruments Incorporated Error resilient video coding using reversible variable length codes (RVLCS)
US6118823A (en) * 1997-04-01 2000-09-12 International Business Machines Corporation Control scheme for shared-use dual-port predicted error array
US6057884A (en) * 1997-06-05 2000-05-02 General Instrument Corporation Temporal and spatial scaleable coding for video object planes
US6181711B1 (en) * 1997-06-26 2001-01-30 Cisco Systems, Inc. System and method for transporting a compressed video and data bit stream over a communication channel
US6233356B1 (en) * 1997-07-08 2001-05-15 At&T Corp. Generalized scalability for video coder based on video objects
US6097725A (en) * 1997-10-01 2000-08-01 International Business Machines Corporation Low cost searching method and apparatus for asynchronous transfer mode systems
JP4558193B2 (en) * 1997-10-23 2010-10-06 ソニー エレクトロニクス インク Data receiving method and apparatus, received data recovery processing apparatus and recording medium
US6043838A (en) * 1997-11-07 2000-03-28 General Instrument Corporation View offset estimation for stereoscopic video coding
US6266375B1 (en) * 1997-11-13 2001-07-24 Sony Corporation Method and apparatus for selecting a quantization table for encoding a digital image
JP3622460B2 (en) * 1997-11-28 2005-02-23 松下電工株式会社 Semiconductor relay
KR100301825B1 (en) * 1997-12-29 2001-10-27 구자홍 Mpeg video decoding system and method of processing overflow of mpeg video decoding system
EP1051853B1 (en) * 1998-01-26 2003-06-18 STMicroelectronics Asia Pacific Pte Ltd. One-pass variable bit rate moving pictures encoding
JP3905969B2 (en) * 1998-01-30 2007-04-18 株式会社東芝 Moving picture coding apparatus and moving picture coding method
EP0935396A3 (en) * 1998-02-09 2004-08-11 Matsushita Electric Industrial Co., Ltd. Video coding method and apparatus
US6438165B2 (en) * 1998-03-09 2002-08-20 Lg Electronics Method and apparatus for advanced encoder system
US6289054B1 (en) * 1998-05-15 2001-09-11 North Carolina University Method and systems for dynamic hybrid packet loss recovery for video transmission over lossy packet-based network
US6804294B1 (en) * 1998-08-11 2004-10-12 Lucent Technologies Inc. Method and apparatus for video frame selection for improved coding quality at low bit-rates
US6137915A (en) * 1998-08-20 2000-10-24 Sarnoff Corporation Apparatus and method for error concealment for hierarchical subband coding and decoding
JP3604290B2 (en) * 1998-09-25 2004-12-22 沖電気工業株式会社 Moving image decoding method and apparatus
US6754277B1 (en) * 1998-10-06 2004-06-22 Texas Instruments Incorporated Error protection for compressed video
US6490705B1 (en) * 1998-10-22 2002-12-03 Lucent Technologies Inc. Method and apparatus for receiving MPEG video over the internet
US6192148B1 (en) * 1998-11-05 2001-02-20 Winbond Electronics Corp. Method for determining to skip macroblocks in encoding video
JP3166736B2 (en) * 1998-11-25 2001-05-14 日本電気株式会社 Moving picture coding apparatus and moving picture coding method
JP2000209580A (en) * 1999-01-13 2000-07-28 Canon Inc Picture processor and its method
GB2347038A (en) * 1999-02-18 2000-08-23 Nokia Mobile Phones Ltd A video codec using re-transmission
JP2000295626A (en) * 1999-04-08 2000-10-20 Mitsubishi Electric Corp Multi-stage image coder
KR100357093B1 (en) * 1999-06-02 2002-10-18 엘지전자 주식회사 apparatus and method for concealing error in moving picture decompression system
US6351491B1 (en) * 1999-06-23 2002-02-26 Sarnoff Corporation Apparatus and method for optimizing the rate control for multiscale entropy encoding
US6968008B1 (en) * 1999-07-27 2005-11-22 Sharp Laboratories Of America, Inc. Methods for motion estimation with adaptive motion accuracy
JP3630590B2 (en) * 1999-08-25 2005-03-16 沖電気工業株式会社 Decoding device and transmission system
US6999673B1 (en) * 1999-09-30 2006-02-14 Matsushita Electric Industrial Co., Ltd. Moving picture decoding method, moving picture decoding apparatus and program recording medium
EP1096804B1 (en) * 1999-10-25 2006-12-13 Matsushita Electric Industrial Co., Ltd. Video decoding method, video decoding apparatus, and program storage media
JP3840020B2 (en) * 1999-12-14 2006-11-01 株式会社東芝 Video encoding device
US6493392B1 (en) * 1999-12-27 2002-12-10 Hyundai Electronics Industries Co., Ltd. Method for coding digital interlaced moving video
US6421386B1 (en) * 1999-12-29 2002-07-16 Hyundai Electronics Industries Co., Ltd. Method for coding digital moving video including gray scale shape information
JP2001197501A (en) * 2000-01-07 2001-07-19 Fujitsu Ltd Motion vector searching device and motion vector searching method, and moving picture coder
US6601209B1 (en) * 2000-03-17 2003-07-29 Verizon Laboratories Inc. System and method for reliable data transmission over fading internet communication channels
US6724945B1 (en) * 2000-05-24 2004-04-20 Hewlett-Packard Development Company, L.P. Correcting defect pixels in a digital image
US6650705B1 (en) * 2000-05-26 2003-11-18 Mitsubishi Electric Research Laboratories Inc. Method for encoding and transcoding multiple video objects with variable temporal resolution
JP3662171B2 (en) * 2000-06-05 2005-06-22 三菱電機株式会社 Encoding apparatus and encoding method
US6738427B2 (en) * 2000-09-15 2004-05-18 International Business Machines Corporation System and method of processing MPEG streams for timecode packet insertion
US7133455B2 (en) * 2000-12-29 2006-11-07 Intel Corporation Providing error resilience and concealment for video data
WO2002063883A1 (en) * 2001-02-06 2002-08-15 Koninklijke Philips Electronics N.V. Preprocessing method applied to textures of arbitrarily shaped objects
EP1374429A4 (en) * 2001-03-05 2009-11-11 Intervideo Inc Systems and methods for encoding and decoding redundant motion vectors in compressed video bitstreams
US6700934B2 (en) * 2001-03-14 2004-03-02 Redrock Semiconductor, Ltd. Error detection using a maximum distance among four block-motion-vectors in a macroblock in a corrupted MPEG-4 bitstream
US6842484B2 (en) * 2001-07-10 2005-01-11 Motorola, Inc. Method and apparatus for random forced intra-refresh in digital image and video coding
US6810144B2 (en) * 2001-07-20 2004-10-26 Koninklijke Philips Electronics N.V. Methods of and system for detecting a cartoon in a video data stream
DE10139641C1 (en) * 2001-08-11 2003-04-10 Freudenberg Carl Kg cleaning tool

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6141448A (en) * 1997-04-21 2000-10-31 Hewlett-Packard Low-complexity error-resilient coder using a block-based standard

Cited By (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1251701A2 (en) * 2001-04-16 2002-10-23 Mitsubishi Denki Kabushiki Kaisha Encoding a video with a variable frame-rate while minimizing total average distortion
EP1251701A3 (en) * 2001-04-16 2004-09-22 Mitsubishi Denki Kabushiki Kaisha Encoding a video with a variable frame-rate while minimizing total average distortion
US7209519B2 (en) 2001-04-16 2007-04-24 Mitsubishi Electric Research Laboratories, Inc. Encoding a video with a variable frame-rate while minimizing total average distortion
US7020203B1 (en) 2001-12-21 2006-03-28 Polycom, Inc. Dynamic intra-coded macroblock refresh interval for video error concealment
WO2004043074A1 (en) * 2002-11-04 2004-05-21 Tandberg Telecom As Inter-network and inter-protocol video conference privacy method, apparatus and computer program product
US7509553B2 (en) 2002-11-04 2009-03-24 Tandberg Telecom As Inter-network and inter-protocol video conference privacy method, apparatus, and computer program product
SG106261A1 (en) * 2002-11-22 2005-06-29 Toshiba Kk Moving picture encoding/decoding method and device
US9084936B2 (en) 2002-12-10 2015-07-21 Sony Computer Entertainment America Llc System and method for protecting certain types of multimedia data transmitted over a communication channel
US9420283B2 (en) 2002-12-10 2016-08-16 Sony Interactive Entertainment America Llc System and method for selecting a video encoding format based on feedback data
US9077991B2 (en) 2002-12-10 2015-07-07 Sony Computer Entertainment America Llc System and method for utilizing forward error correction with video compression
EP1439714A3 (en) * 2002-12-23 2004-12-08 Polycom, Inc. Dynamic intra-coded macroblock refresh interval for video error concealment
EP1439714A2 (en) * 2002-12-23 2004-07-21 Polycom, Inc. Dynamic intra-coded macroblock refresh interval for video error concealment
JP2006521064A (en) * 2003-03-17 2006-09-14 クゥアルコム・インコーポレイテッド Partial frame encoding system and method for wireless multimedia communication
US8155182B2 (en) 2003-03-17 2012-04-10 Qualcomm Incorporated System and method for partial intraframe encoding for wireless multimedia transmission
US7949047B2 (en) 2003-03-17 2011-05-24 Qualcomm Incorporated System and method for partial intraframe encoding for wireless multimedia transmission
WO2006075070A1 (en) * 2005-01-07 2006-07-20 France Telecom Video encoding method and device
FR2881013A1 (en) * 2005-01-14 2006-07-21 Canon Kk Continuous video transmission method for e.g. Internet, involves interlacing transmission packets, created with elementary entities of images` spatial zones, so that two packets transport data of two different images and two different zones
WO2006126694A3 (en) * 2005-05-27 2007-08-16 Canon Kk Video coding using an alternative reference frame for motion compensated prediction
US8165217B2 (en) 2005-05-27 2012-04-24 Canon Kabushiki Kaisha Image decoding apparatus and method for decoding prediction encoded image data
CN101185339B (en) * 2005-05-27 2011-06-22 佳能株式会社 Image decoding apparatus and method for decoding image data, device and method for encoding image
WO2006126694A2 (en) * 2005-05-27 2006-11-30 Canon Kabushiki Kaisha Video coding using an alternative reference frame for motion compensated prediction
US9661376B2 (en) 2005-07-13 2017-05-23 Polycom, Inc. Video error concealment method
US11477253B2 (en) 2006-06-09 2022-10-18 Qualcomm Incorporated Enhanced block-request streaming system using signaling or block creation
US8787447B2 (en) 2008-10-30 2014-07-22 Vixs Systems, Inc Video transcoding system with drastic scene change detection and method for use therewith
EP2194719A2 (en) * 2008-10-30 2010-06-09 ViXS Systems Inc. Video transcoding system with quality readjustment based on scene change detection
EP2194719A3 (en) * 2008-10-30 2014-05-14 ViXS Systems Inc. Video transcoding system with quality readjustment based on scene change detection
US11743317B2 (en) 2009-09-22 2023-08-29 Qualcomm Incorporated Enhanced block-request streaming using block partitioning or request controls for improved client-side handling
US11770432B2 (en) 2009-09-22 2023-09-26 Qualcomm Incorporated Enhanced block-request streaming system for handling low-latency streaming
US10855736B2 (en) 2009-09-22 2020-12-01 Qualcomm Incorporated Enhanced block-request streaming using block partitioning or request controls for improved client-side handling
US9917874B2 (en) 2009-09-22 2018-03-13 Qualcomm Incorporated Enhanced block-request streaming using block partitioning or request controls for improved client-side handling
EP2654298A3 (en) * 2012-04-19 2014-10-01 ViXS Systems Inc. Detection of video feature based on variance metric
US9071842B2 (en) 2012-04-19 2015-06-30 Vixs Systems Inc. Detection of video feature based on variance metric
DE102015121148A1 (en) * 2015-12-04 2017-06-08 Technische Universität München Reduce the transmission time of pictures
CN105847796A (en) * 2016-03-31 2016-08-10 乐视控股（北京）有限公司 Bit distribution method and device used for video coding
JP2023517003A (en) * 2020-07-01 2023-04-21 ▲騰▼▲訊▼科技（深▲セン▼）有限公司 Image processing method, device, electronic device and computer program

Also Published As

Publication number Publication date
US20050201465A1 (en) 2005-09-15
US20050058199A1 (en) 2005-03-17
US7236520B2 (en) 2007-06-26
US20030053538A1 (en) 2003-03-20
US20050105625A1 (en) 2005-05-19
JP2009005357A (en) 2009-01-08
US7003033B2 (en) 2006-02-21
US7260150B2 (en) 2007-08-21
EP1374578A2 (en) 2004-01-02
US7221706B2 (en) 2007-05-22
US6993075B2 (en) 2006-01-31
US7215712B2 (en) 2007-05-08
WO2002071736A3 (en) 2003-04-03
JP2008236789A (en) 2008-10-02
US20030053454A1 (en) 2003-03-20
US20070121721A1 (en) 2007-05-31
US20050149831A1 (en) 2005-07-07
US6940903B2 (en) 2005-09-06
US7164717B2 (en) 2007-01-16
US20030031128A1 (en) 2003-02-13
US20030026343A1 (en) 2003-02-06
US7164716B2 (en) 2007-01-16
US20050105614A1 (en) 2005-05-19
US6876705B2 (en) 2005-04-05
WO2002071736A8 (en) 2003-10-23
JP2008259230A (en) 2008-10-23
US7042948B2 (en) 2006-05-09
US6990151B2 (en) 2006-01-24
US7242715B2 (en) 2007-07-10
JP2004528752A (en) 2004-09-16
US7133451B2 (en) 2006-11-07
US6970506B2 (en) 2005-11-29
EP1374430A1 (en) 2004-01-02
US20020176505A1 (en) 2002-11-28
US20050117648A1 (en) 2005-06-02
US20020181594A1 (en) 2002-12-05
US20020176025A1 (en) 2002-11-28
US20030053537A1 (en) 2003-03-20
US20030063806A1 (en) 2003-04-03
WO2002071639A8 (en) 2003-10-30
JP2004531925A (en) 2004-10-14
US20050254584A1 (en) 2005-11-17
US20030067981A1 (en) 2003-04-10
US20050089091A1 (en) 2005-04-28
WO2002071640A1 (en) 2002-09-12
US8135067B2 (en) 2012-03-13
AU2002245609A1 (en) 2002-09-19
US7224730B2 (en) 2007-05-29
EP1374430A4 (en) 2005-08-17
JP2008278505A (en) 2008-11-13
JP2008259229A (en) 2008-10-23
US20030012287A1 (en) 2003-01-16
EP1374429A4 (en) 2009-11-11
US20030012285A1 (en) 2003-01-16
JP2008306735A (en) 2008-12-18
US20050123044A1 (en) 2005-06-09
US20050201466A1 (en) 2005-09-15
US7110452B2 (en) 2006-09-19
JP2004532540A (en) 2004-10-21
JP2008306734A (en) 2008-12-18
EP1374578A4 (en) 2007-11-14
WO2002071736A2 (en) 2002-09-12
EP1374429A1 (en) 2004-01-02

Similar Documents

Publication Publication Date Title
US7133451B2 (en) Systems and methods for refreshing macroblocks
Wiegand et al. Multi-frame motion-compensated prediction for video transmission
US7738561B2 (en) MPEG-4 streaming system with adaptive error concealment
US20080025397A1 (en) Intra-Frame Flicker Reduction in Video Coding
JP2009524365A (en) Method and apparatus for error resilience algorithm in wireless video communication
Al-Mualla et al. Error concealment using motion field interpolation
Hinds et al. Joint block-based video source/channel coding for packet-switched networks
Shih et al. A new unequal error protection scheme based on FMO
Le Leannec et al. Error-resilient video transmission over the Internet
KR20010108077A (en) Method and device for robust decoding of header information in macroblock-based compressed video data
Zheng et al. Error-resilient coding of H. 264 based on periodic macroblock
Hsu et al. MPEG-2 spatial scalable coding and transport stream error concealment for satellite TV broadcasting using Ka-band
Jung et al. Error-resilient video coding using long-term memory prediction and feedback channel
Frossard et al. Adaptive MPEG-2 information structuring
Parameswaran et al. Adapting quantization offset in multiple description coding for error resilient video transmission
Vermeirsch et al. Low complexity multiple description coding for h. 264/avc
Kim et al. Integrated AIR/UEP scheme for robust video transmission with a corruption model
Duong et al. Frame error concealment using pixel correlation in overlapped motion compensation regions
Karlekar Content based robust video coding for videoconferencing
Tavares et al. Forward error protection of H. 263-DG bit streams
Behera FEC for efficient video transmission over CDMA
Cai et al. Joint mode selection and unequal error protection for bitplane coded video transmission over wireless channels
Tavares et al. A common channel adaptation approach for video coding
Navarro et al. Significance analysis of MPEG-4 video syntactical elements
Fang Error control for non-scalable and scalable video over error-prone networks

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ OM PH PL PT RO RU SD SE SG SI SK SL TJ TM TN TR TT TZ UA UG US UZ VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
WWE Wipo information: entry into national phase

Ref document number: 2002570429

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 2002721263

Country of ref document: EP

CFP Corrected version of a pamphlet front page
CR1 Correction of entry in section i

Free format text: IN PCT GAZETTE 37/2002 DUE TO A TECHNICAL PROBLEM AT THE TIME OF INTERNATIONAL PUBLICATION, SOME INFORMATION WAS MISSING (81). THE MISSING INFORMATION NOW APPEARS IN THE CORRECTED VERSION.

Free format text: IN PCT GAZETTE 37/2002 DUE TO A TECHNICAL PROBLEM AT THE TIME OF INTERNATIONAL PUBLICATION, SOME INFORMATION WAS MISSING (81). THE MISSING INFORMATION NOW APPEARS IN THE CORRECTED VERSION.

WWP Wipo information: published in national office

Ref document number: 2002721263

Country of ref document: EP

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642