ES2371188T3 - TRANSCODER DIRECTED BY THE CONTENT ORGANIZING THE MULTIMEDIA TRANSCODING USING CONTENT INFORMATION. - Google Patents

Abstract

A multimedia data processing method comprising: the reception of multimedia data and the coding of multimedia data (361) in a first layer of encoded data and a second layer of encoded data based on the content of the multimedia data, being configured the first coded data layer to be encodable independently of the second coded data layer and in which the first data layer is coded with a lower quality than the second data layer, characterized in that the method comprises: the coding of the first data layer through the steps of: transformation of the residual error coefficients of the first data layer, the residual error being the difference between the multimedia data being encoded (361) and the best fit macro block of a temporary reference chart; the calculation of a coefficient of the first data layer, C'base, from the transformed residual error coefficients, Cbase and the quantification of the coefficient of the first data layer, C'base, by a first quantification parameter , Qb; and the coding of the second data layer through the steps of: the transformation of the residual error coefficients in the second data layer, the residual error being the difference between the multimedia data being encoded (361) and the macro block best fit of a temporary reference chart; the calculation of a coefficient of the second data layer, C'mej, from the transformed residual error coefficients, Cmej and the quantification of the coefficient of the second data layer, C'mej, by a second quantification parameter , in which: - 0, otherwise C'mej = Cmej - Qb -1 (Qb (C'base)) in which Qb -1 is the quantification parameter of the first data layer and in which the First quantization parameter is set to be higher than the second quantification parameter, so that the first layer of encoded data can be decoded to form multimedia data that can be displayed if the second encoded data layer is not available and the first coded data layer and second coded data layer can be decoded in combination to form multimedia data that can be displayed if both the first coded data layer and the second coded data layer are arranged onibles.

Description

Transcoder driven by content that organizes multimedia transcoding using content information

Claim of priority under 35 U.S.C § 119

This patent application claims priority over (a) Provisional Patent Application No. 60 / 721,416 entitled “A VIDEO TRANSCODER FOR REAL-TIME STREAMING AND MOBILE BROADCAST APPLICATIONS” filed on September 27, 2005, (b) Patent Application Provisional No. 60 / 789,377 entitled “A VIDEO TRANSCODER FOR REAL-TIME STREAMING AND MOBILE BROADCAST APPLICATIONS” filed on April 4, 2006, (c) Provisional Application No. 60 / 727,643 entitled “METHOD AND APPARATUS FOR SPATIO-TEMPORAL DEINTERLACING AIDED BY MOTION COMPENSATION FOR FIELD-BASED VIDEO ”filed on October 17, 2005, (d) Provisional Application No. 60 / 727,644 entitled“ METHOD AND APPARATUS FOR SHOT DETECTION IN VIDEO STREAMING ”filed on October 17, 2005, (e) Provisional Application No. 60 / 727,640 entitled “A METHOD AND APPARATUS FOR USING AN ADAPTIVE GOP STRUCTURE IN VIDEO STREAMING” filed on October 17, 2005, (f) Provisional Application No. 60 / 730,145 entitled a “INVERSE TELECINE ALGORITHM BASED ON STATE MACHINE” filed on October 24, 2005, and (g) Provisional Application No. 60 / 789.048 entitled “SPATIO-TEMPORAL DEINTERLACING AIDED BY MOTION COMPENSATION FOR FIELD- BASED MULTIMEDIA DATA” filed on April 3 of 2006. All seven provisional patent applications are assigned to the assignee hereof.

Reference of applications being processed together with this one for the Patent

This Patent Application relates to United States Patent Application No. 11 / 373,577 entitled "CONTENT CLASSIFICATION FOR MULTIMEDIA PROCESSING" filed on March 10, 2006, transferred to the assignee hereof.

Background

Countryside

The present application is directed to devices and procedures for the transcoding of video of video data for its continuous transmission in real time and, more particularly, for the transcoding of video data for continuous transmission in real time in mobile broadcast applications .

Background

Efficient video compression is useful in many multimedia applications such as continuous wireless video transmission and video telephony, due to limited bandwidth resources and available bandwidth variability. Certain video coding standards, such as MPEG-4 (ISO / IEC), H.264 (ITU) or similar video encodings provide highly efficient encoding very suitable for applications such as wireless broadcasts. Some multimedia data, for example, digital television presentations, are generally coded according to other standards such as MPEG-2. Accordingly, transcoders are used to transcode or convert the encoded multimedia data in accordance with one standard (for example, MPEG-2) into another standard (for example, H.264) prior to wireless broadcast.

Improvements in speed-optimized codecs could offer advantages in resistance to errors, error recovery and scalability. Moreover, the use of information determined from multimedia data itself could also offer additional improvements for coding, including resistance to errors, error recovery and scalability. Mihaela van der Schaar et al. “A Hybrid Temporal SNR Fine-Granular Scalability for Internet Video,” IEEE Transactions on circuits and systems for video technology, IEEE Service Center, Piscataway, NJ, USA, vol. 11, nº 3, March 2001 reveals a scalable and innovative video coding framework and a corresponding compression procedure for the continuous transmission of video on the Internet. Consequently, there is a need for a transcoder that provides highly efficient multimedia data processing and compression that uses the information determined from the multimedia data itself, is scalable and is resistant to errors for use in many multimedia data applications. which include the broadcast of multimedia information in continuous transmission for mobile phones.

Summary

Each part of the inventive content based on transcoding apparatus and procedures described and illustrated has several aspects, none of which is responsible for its desirable attributes alone. Without limiting the scope of this disclosure, the most relevant features will now be briefly explained. After considering this explanation and particularly after reading the section entitled "Detailed description" it will be understood how the characteristics of this content-driven transcoding provides improvements for the multimedia data processing apparatus and procedures.

55 E06804241 04-11-2011

The inventive aspects described herein are set forth in the appended claims in relation to the use of content information for various multimedia data encoding procedures and in various modules or components of an encoder, for example, an encoder used in a transcoder. . A transcoder can organize the transcoding of multimedia data using content information. Content information can be received from another source, for example, metadata that is received with the video. The transcoder can be configured to generate content information through a variety of different processing operations. In some aspects, the transcoder generates a classification of the multimedia data content, which is then used in one or more encoding processes. In some aspects, a content driven transcoder can determine the spatial and temporal content information of the multimedia data and use the content information for a uniform quality coding related to the content through the channels and classification of the content into basis for compression / bit allocation.

In some aspects, the content information (e.g., metadata, content metric and / or a content classification) of the multimedia data is obtained or calculated and then provided to the transcoder components for use in the processing of Multimedia data for coding. For example, a preprocessor can use certain content information for scene change detection, reverse telecine realization ("IVTC"), deinterlacing, motion compensation and noise suppression (eg 2D waveform transform) and reduction of the temporary space noise, for example elimination of artificialities, de-plating, block suppression and / or noise suppression. In some aspects, a preprocessor can also use the content information for a reduction in sampling, for example by determining the appropriate "safe" and "action management" areas when sampling is reduced from the standard definition (SD) to the of a Video Graphic Adapter Room (QVGA).

In some aspects, an encoder includes a content classification module that is configured to calculate the content information. The encoder can use the content classification for a bit rate control (for example bit allocation) in the determination of the quantification parameters (QP) for each MB, for motion estimation, for example, by estimating movement (ME) in color, making a prediction of the motion vector (MV), scalability by providing a base layer and an improvement layer and for resistance to errors by using the content classification to affect the hierarchy of precision and resistance to error schemes that include, for example, intra-adaptive refreshment, boundary elimination processes and providing redundant frame I data in an improvement layer. In some aspects, the transcoder uses the classification of the content in coordination with a data multiplexer to maintain the optimal quality of the multimedia data through the channels. In some aspects, the encoder may use content classification information to force four Is to appear periodically in the encoded data to allow fast channel switching. Such implementations can also make use of I blocks that may be required in the data encoded for resistance against errors, such as random access switching and resistance against errors (based on, for example, content classification) can be Effectively combine through the prediction hierarchy to improve coding efficiency while increasing robustness against errors.

In one aspect a multimedia data processing procedure includes the reception of multimedia data and the encoding of multimedia data in a first data group and a second data group based on the content of the multimedia data, the first group of data being configured. data for decoding regardless of the second group of data and in which the first and second groups of data are encoded with different quality levels. In one aspect of the first aspect, the first group of data comprises frames I and frames P and the second group of data comprises frames I, frames P and frames B. In another aspect, the first group of data comprises a base layer and the second data group comprises an improvement layer. In a third aspect, the method further comprises the classification of the content of the multimedia data in which the coding is based on the classification of the content. In a fourth aspect, the coding comprises the determination of a first parameter of quantification of the multimedia data for the coding of the first group of data and the determination of a second parameter of quantification for the coding of the second group of data, in which the Determination of the first and second quantification parameters is based on the classification of the content. In a fifth aspect, the coding comprises the assignment of a bit rate for at least a part of the multimedia data based on the content classification. In a sixth aspect, the coding also includes the use of content classification to detect scene changes and in determining whether to include frames I in the first data group and in the second data group based on the scene changes. detected. In a seventh aspect, the coding includes the determination of a frame rate for the coding of the multimedia data based on the classification of the content. In an eighth aspect, the coding includes the realization of an estimation of movement of the multimedia data based on the classification of the content. In a ninth aspect, the procedure also includes the determination of a first frame rate for the coding of the first group of data and the determination of a second frame rate for the coding of the second group of data, in which the first rate of frames is less than the second frame rate. In a tenth aspect, the coding includes the performance of resistance processing against errors in the multimedia data based on the classification of the content. In an eleventh aspect, the encoding includes the encoding of the first data group and the second data group so that the first data group can be decoded to form multimedia data that can be displayed if the second data group is not available. and the first group of data and the

55 E06804241 04-11-2011

The second group of data can be decoded in combination to form multimedia data that can be displayed if both the first group of data and the second group of data are available. In a twelfth aspect, the first quantization parameter includes a first stage size for data coding and the second quantization parameter includes a second stage size for data coding, in which the first stage size is larger than The second stage size. In a thirteenth aspect, the method further comprises the classification of the content of the multimedia data and in which the coding is based on the classification of the content and in which the coding comprises the reduction of the noise in the multimedia data based on the classification Of content. In a fourteenth aspect, noise reduction includes the realization of an elimination of artifice. In a fifteenth aspect, noise reduction comprises the processing of at least part of the multimedia data with a scraping filter, in which the strength of the scraping filter is based on the content of the multimedia data. In a sixteenth aspect, noise reduction includes the processing of at least part of the multimedia data with a block suppression filter, in which the force of the block suppression filter is based on the content of the multimedia data. In a seventeenth aspect, noise reduction includes the filtering of selected frequencies of multimedia data. In a eighteenth aspect, the strength of the scraping filter is based on the classification of multimedia data content. In a nineteenth aspect, the strength of the block suppression filter is based on the classification of multimedia data content. In a twentieth aspect, coding comprises the reduction of sampling of multimedia data. Finally, in a twenty-first aspect, coding includes the association of a quality level with multimedia data and the use of the quality level and information of multimedia data content to determine a bit rate for data coding. multimedia.

In a second aspect, an apparatus for processing multimedia data includes an encoder configured to receive multimedia data and encode the multimedia data in a first data group and a second data group based on the content of the multimedia data, the first configured data group for decoding independently of the second data group and in which the first and second data groups are encoded with different quality levels. In one aspect of the first aspect, the encoder includes a content classification module configured to determine a multimedia data content classification and the coding module is further configured to encode multimedia data based on the content classification. In a second aspect, the encoder is further configured to determine a first multimedia data quantification parameter for the encoding of the first data group and a second quantification parameter for the encoding of the second data group in which the determination of the first and second quantification parameters is based on the classification of multimedia data content. In a third aspect, the encoder comprises a motion estimation module configured to perform an estimate of the movement of the multimedia data based on the classification of the content and generate compensation information of the motion data and in which the coding module It is also configured to use motion compensation information to encode multimedia data. In a fourth aspect, the encoder also includes a quantization module for the determination of a quantification parameter of the multimedia data based on the classification of the content and the encoder is further configured to use the quantization parameter to encode the multimedia data. In a fifth aspect, the encoder also includes a bit allocation module configured to provide a bit rate for at least a portion of the multimedia data based on the content classification. In a sixth aspect, the encoder also includes a scene change detection module configured to detect scene changes and the coding module is further configured to include I frames in the encoded multimedia data based on the scene changes detected. In a seventh aspect, the encoder also includes a frame rate module configured to determine a frame rate of the multimedia data based on the content classification and in which the coding module encodes the multimedia data based on the rate of picture. In an eighth aspect, the encoder is also configured to encode the first data group and the second data group based on the content classification. In a ninth aspect, the encoder is also configured to perform error processing on the multimedia data based on the content classification.

In a third aspect, an apparatus for processing multimedia data includes means for receiving multimedia data and means for encoding multimedia data in a first group of encoded data and a second group of encoded data based on the content of the data. multimedia data, the first data group is configured for decoding independently of the second data group and in which the first and second data groups are encoded with different quality levels. In one aspect of the first aspect, the receiving means comprises an encoder. In another aspect, the means for encoding comprise an encoder. In a third aspect, the coding means comprise means for determining a classification of the content of the multimedia data in which the coding means encodes the multimedia data on the classification of the content. In a fourth aspect, the coding means comprise a transcoder comprising an encoder.

In a fourth aspect, a medium that can read a machine comprises instructions that after its execution cause the machine to receive multimedia data and encode the multimedia data in a first group of encoded data and a second group of encoded data based on the content of the multimedia data, the first data group is configured for decoding independently of the second data group and in which the first and second data groups are encoded with different quality levels. In one aspect of the fourth aspect, the medium that can

50 E06804241 04-11-2011

reading a computer further comprises instructions for generating a classification of the content indicative of the content of the multimedia data, in which the coding of the multimedia data in a first group of encoded data and a second group of encoded data includes the encoding of the multimedia data based on the content classification. In another aspect, the coding comprises the determination of a first parameter of quantification of the multimedia data for the coding of the first group of data and the determination of a second parameter of quantification for the coding of the second group of data in which the determination of the First and second quantification parameters are based on content classification. In a third aspect, the instructions for encoding the multimedia data include instructions for assigning a bit rate for at least a part of the multimedia data based on the content of the multimedia data.

In a fifth aspect, a processor comprises a configuration for receiving multimedia data and encoding multimedia data in a first group of encoded data and a second group of encoded data based on the content of the multimedia data, the first group of data configured for decoding independently of the second group of data and in which the first and second group of data are encoded with different quality levels. In an aspect of the fifth aspect, the processor further comprises a configuration for generating a classification of the content indicative of the content of the multimedia data in which the coding comprises the coding of the multimedia data based on the classification of the content. In another aspect, the processor further comprises a configuration for determining a first parameter of quantification of the multimedia data for the encoding of the first group of data and a second parameter of quantification for the coding of the second group of data, in which the first and The second quantification parameters are based on the content classification.

Brief description of the drawings

FIGURE 1A is a block diagram in a media broadcast system that includes a transcoder for transcoding between different video formats.

FIGURE 1B is a block diagram of an encoder configured to encode multimedia data and provide a first group of encoded data and a second group of encoded data.

FIGURE 1C is a block diagram of a processor configured to encode multimedia data.

FIGURE 2 is a block diagram of an example of a transcoder of the system of FIGURE 1.

FIGURE 3 is a flow chart illustrating the operation of a syntactic analyzer used with the transcoder of FIGURE 2.

FIGURE 4 is a flow chart illustrating the operation of a decoder used with the transcoder of FIGURE 2.

FIGURE 5 is a system timing diagram illustrating a sequence of operations performed by the transcoder of FIGURE 2.

FIGURE 6 is a flow chart illustrating a sequence of operations and functions of a preprocessor that can be used in the transcoder of FIGURE 2.

FIGURE 7 is a block diagram of an example 2-step encoder that can be used in the transcoder of FIGURE 2.

FIGURE 8 illustrates an example of a classification chart that illustrates an aspect of how to associate texture and motion values with content classification.

FIGURE 9 is a flow chart illustrating an example operation for content classification, such as for use in the encoder of FIGURE 7.

FIGURE 10 is a flow chart illustrating the operation of a rate control such as for use with the encoder of FIGURE 7.

FIGURE 11 is a flow chart illustrating the operation of an example motion estimator, such as for use with the encoder of FIGURE 7.

FIGURE 12 is a flow chart illustrating the operation of an exemplary decision encoder function, such as for use with the encoder of FIGURE 7.

FIGURE 13 is a flow chart illustrating an example operation that performs scalability for use in the encoder of FIGURE 7.

FIGURE 14 is a flow chart illustrating an example operation that effects a rate of distortion data as occurs in the encoder of FIGURE 7, for example.

40 E06804241 04-11-2011

FIGURE 15 is a graph illustrating the relationship between coding complexity, assigned bits and human visual quality

FIGURE 16 is a graph illustrating a nonlinear scene detection formula. FIGURE 17 is a block diagram of the system illustrating means for receiving multimedia data and means for encoding multimedia data received.

FIGURE 18 is a diagram illustrating a deinterlacing process using estimation / compensation of movement.

FIGURE 19 is a block diagram of a multimedia communication system. FIGURE 20 is a diagram illustrating the organization of a video bit stream in an enhancement layer and a base layer

FIGURE 21 is a diagram illustrating the alignment of portions to the limits of video frames. FIGURE 22 is a block diagram illustrating the prediction hierarchy. FIGURE 23 is a process flow diagram illustrating a multimedia data coding procedure

based on content information.

FIGURE 24 is a process flow diagram illustrating a multimedia data coding procedure so that data limits are aligned based on the level of content information. FIGURE 25 is a graph illustrating a safe area of action and a safe title area of a data box. FIGURE 26 is a graph illustrating a safe area of action of a data frame. FIGURE 27 is a process flow diagram illustrating a multimedia data encoding process

using intra-adaptive soda based on multimedia content information.

FIGURE 28 is a process flow diagram illustrating a multimedia data encoding process using redundant I frames based on multimedia content information. FIGURE 29 illustrates motion compensation vectors between a current frame and a previous MVP frame and a

Current car and next MVN chart. FIGURE 30 is a process flow diagram illustrating a tap detection. FIGURE 31 is a process flow diagram illustrating the coding of the base and improvement layers. FIGURE 32 is a scheme illustrating the coding of a macro block. FIGURE 33 is a scheme illustrating modules for coding a base layer and an improvement layer. FIGURE 34 shows an example of a process for selecting the base layer and improvement layer coefficient. FIGURE 35 shows another example of a coefficient selector process of a base layer and a layer of

improvement.

FIGURE 36 shows another example of a coefficient selector process of a base layer and a layer of improvement. FIGURE 37 is a process flow diagram illustrating the encoding of multimedia data based on the

content information. FIGURE 38 is a diagram illustrating possible system decisions in a reverse telecine process. FIGURE 39 illustrates the limits to be filtered in a macro block by a block suppression process. FIGURE 40 is a diagram illustrating a process that deinterlaced space-time. FIGURE 41 illustrates an example for a new 1D polyphase sampling. FIGURE 42 is a flow chart illustrating an example of an adaptive transmission GOP structure

Video continues. It is indicated that, where appropriate, similar numbers refer to similar parts through the various views of the drawings. 6

55 E06804241 04-11-2011

Detailed description

The following detailed description addresses certain aspects explained in this disclosure. However, the invention can be carried out in a multitude of different ways. Reference in the present specification to "one aspect" means that a particular feature, structure or feature described in connection with the aspect in at least one aspect is included. The occurrences of the phrases "in one aspect", "according to one aspect" or "in some aspects", in several places in the specification do not necessarily all refer to the same aspect nor are they separate mutually exclusive alternative aspects of other aspects. Moreover, several characteristics that can be exhibited by some aspects and not by others are described. Similarly, several requirements are described that may be requirements for some aspects but not for other aspects.

The following description includes details to provide a comprehensive understanding of the examples. However, it will be understood by one skilled in the art that the examples can be put into practice even if every detail of a process or device in an example or aspect is not described or illustrated herein. For example, electrical components can be shown in block diagrams that do not illustrate each electrical connection or each electrical element of the component so as not to obscure the examples with unnecessary details. In other cases, such components, other structures and techniques can be shown in detail to further explain the examples.

The present disclosure relates to coding control and transcoding apparatus and procedures that use information from the content of the multimedia data that is encoded. “Content information” or “content” (of multimedia data) are broad terms that mean information in relation to multimedia data content and may include, for example, metadata, metrics calculated from multimedia data and related information with the content associated with one or more metrics, for example a classification of the content. Content information may be provided to an encoder or determined by an encoder, depending on the particular application. Content information can be used for many aspects of multimedia data coding, including scene change detection, temporal processing, spatio-temporal noise reduction, sampling reduction, determination of bit rates for quantification, scalability , resistance to errors, maintenance of optimal multimedia quality through broadcast channels and fast channel change. Using one or more of these aspects, a transcoder can organize the processing of multimedia data and produce encoded multimedia data in relation to the content. The descriptions and figures herein that describe the aspects of transcoding may also be applicable to aspects of coding and aspects of decoding.

The transcoder apparatus and procedures refer to transcoding from one format to another and is specifically described herein in relation to MPEG-2 video transcoding to an improved format, scalable H.264 for transmission through channels Wireless to mobile devices, illustrative of some aspects. However, the description of MPEG-2 video transcoding to the format

H.264 is not intended to limit the scope of the invention, but is merely an example of some aspects of the invention. The disclosed devices and procedures provide a highly efficient architecture that supports flexible coding against errors with random access and layer capability and which can also be applied to the transcoding and / or encoding of video formats other than MPEG-2 and H .264.

"Multimedia data" or simply "multimedia" as used herein, is a broad expression that includes video data (which may include audio data), audio data or both video and audio data. "Video data" or "video" as used herein is a broad expression that refers to frame-based or field-based data, which includes one or more images or sequences of related images, content text, information. of image and / or audio data and can be used to refer to multimedia data (for example, expressions can be used interchangeably) unless otherwise specified.

Examples of various components of a transcoder and examples of processes that can use content information for multimedia data encoding are described below.

Multimedia broadcasting system

FIGURE 1A is a block diagram illustrating a data flow of some aspects of a multimedia data broadcasting system 100. In system 100, a multimedia data provider 106 communicates encoded multimedia data 104 to a transcoder 200. The data encoded multimedia 104 are received by transcoder 200, which processes multimedia data 104 into raw multimedia data in block 110. Processing in block 110 decodes and analyzes encoded multimedia data 104 and further processes multimedia data to prepare them for processing. encoding in another format. Decoded multimedia data is provided to block 112 in which multimedia data is encoded to a predetermined multimedia format or standard. Once the multimedia data has been encoded, in block 114 they are prepared for transmission, for example, by means of a wireless broadcasting system (for example a cellular telephone broadcasting network

or through another communications network). In some aspects, the multimedia data received 104 has been encoded in accordance with the MPEG-2 standard. After the transcoded multimedia data 104 has been

60 E06804241 04-11-2011

decoded, transcoder 200 encodes multimedia data to an H.264 standard.

FIGURE 1B is a block diagram of a transcoder 130 that can be configured to process blocks 110 and 112 of FIGURE 1A. The transcoder 130 can be configured to receive multimedia data, decode and analyze the multimedia data in elementary continuous transmissions in packets (e.g., subtitles, audio, metadata, raw video, CC data and presentation time stamps), encode them into a desired format and provide encoded data for further processing or transmission. The transcoder 130 can be configured to provide coded data to two or more data groups, for example, a first group of coded data and a second group of coded data, which is referred to as layered coding. In some examples of aspects, the various data groups (or layers) in a layered coding scheme can be encoded with different quality levels and formatted so that the data encoded in a first data group is of the lowest quality (for example, provide a lower level of visual quality when viewed) than the data encoded in a second group of data.

FIGURE 1C is a block diagram of a processor 140 that can be configured to transcode multimedia data and can be configured to perform part or all of the processing represented in blocks 110 and 112 of FIGURE 1A. Processor 140 may include modules 124a..n, perform one or more of the transcoding processes described herein, including decoding, analysis, preprocessing and coding and use content information for processing. The processor 140 also includes an internal memory 122 and can be configured to communicate with an external memory 120, both directly and indirectly through another device. The processor 140 also includes a communications module 126 configured to communicate with one or more devices external to the processor 140, which include the reception of multimedia data and the provision of encoded data, such as data encoded in a first group of data and encoded data in a second group of data. In some examples of aspects, the various data groups (or layers) in a layered coding scheme can be encoded with different quality levels and formatted so that the data encoded in a first data group is of a lower quality (for example, they provide a lower level of visual quality when viewed) than the data encoded in a second group of data.

The transcoder 130 or the preprocessor 140 (configured for transcoding) the components thereof and the processes contained therein, can be implemented by hardware, software, firmware, middleware, microcode or any combination thereof. For example, an analyzer, decoder, preprocessor or encoder can be of independent components, incorporated as hardware, firmware, middleware in a component of another device or implemented in microcode or software running on a processor or a combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments that perform motion compensation, tap classification and coding processes can be stored in a medium that can read a machine such as a media storage. A code segment can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class or any combination of instructions, data structures or program statements. A code segment can be connected to another code segment or to a hardware circuit by passing and / or receiving information, data, arguments, parameters or memory content.

Illustrative example of transcoder architecture

FIGURE 2 illustrates a block diagram of an example of a transcoder that can be used by transcoder 200 illustrated in multimedia broadcasting system 100 of FIGURE 1. Transcoder 200 comprises an analyzer / decoder 202, a preprocessor 226, a encoder 228 and a synchronization layer 240, described further below. The transcoder 200 is configured to use information from the content of the multimedia data 104 for one or more aspects of the transcoding process as described herein. The content information can be obtained from an external source to transcoder 200, through multimedia metadata or be calculated by the transcoder, for example, by preprocessor 226 or encoder 228. The components shown in FIGURE 2 are illustrative of components that can be included in a transcoder that uses content information for one or more transcoding processes. In a particular implementation, one or more of the components of transcoder 200 may be excluded or additional components may be included. Additionally, parts of the transcoder and transcoding processes are described so as to allow someone skilled in the art to practice the invention although every detail of a process or device may not be described herein.

FIGURE 5 illustrates a timing diagram as a graphic illustration of the temporal relationships of the operation of various components and / or processes of transcoder 200. As shown in FIGURE 5, the continuous transmission of encoded video 104 (encoded multimedia data) , such as MPEG-2 video, are first received in an arbitrary zero (0) time by analyzer 205 (FIGURE 2). Next, the continuous video transmission is analyzed 501, demultiplexed 502 and decoded 503 such as by the analyzer 205 in combination with the decoder 214. As illustrated, these processes can occur in parallel, with a slight displacement of time, to provide a continuous transmission output of the data processing to the preprocessor 226

55 E06804241 04-11-2011

(FIGURE 2). At a time T1 504 once the preprocessor 226 has received sufficient data from the decoder 214 to begin the output of the processing results, the remaining processing steps become sequential in nature, with the first coding step 505, the second coding step 506 and recoding 507 in sequence after preprocessing until completion of recoding at time Tf 508.

The transcoder 200 described herein can be configured to transcode a variety of multimedia data and many of the processes apply to any type of multimedia data that is transcoded. Although some of the examples provided herein refer particularly to the transcoding of MPEG-2 data into H.264 data, those examples do not mean that they limit disclosure to such data. The coding aspects described below can be applied to the transcoding of any suitable multimedia data standard to another suitable multimedia data standard.

Analyzer / Decoder

Referring again to FIGURE 2, the analyzer / decoder 202 receives multimedia data 104. The analyzer / decoder 202 includes a transport continuous transmission analyzer ("analyzer") 205 that receives the multimedia data 104 and analyzes the data in a elementary continuous transmission (ES) of video 206, an audio ES 208, presentation time stamps (PTS) 210 and other data such as subtitles 212. An ES transports a type of data (video or audio) from a video encoder or simple audio. For example, a video ES comprises the video data for a sequence of data, including the sequence the header and all parts of the sequence. An elementary continuous transmission in packets, or PES, consists of a single ES that has been converted to packets, each typically beginning with an added packet header. A PES continuous transmission contains only one type of data from an origin, for example from a video or audio encoder. PES packages have variable length, which does not correspond to the fixed package length of transport packages and can be much longer than a transport package. When the transport packages are formed from a continuous PES transmission, the PES header can be placed at the beginning of the useful content of the transport package, immediately following the transport package header. The content of the remaining PES package fills the useful content of successive transport packages until the PES package has been fully used. The final transport package can be filled to a fixed length, for example, by filling it with bytes, for example bytes = 0xFF (all ones).

The analyzer 205 communicates the video ES 206 to a decoder 214 which is part of the analyzer / decoder 202 shown here. In other configurations the analyzer 205 and the decoder 214 are separate components. The PTS 210 are sent to a transcoder PTS generator 215, which can generate separate separate timestamps for transcoder 200 for use in the arrangement of the data to be sent from transcoder 200 to an emission system. The transcoder PTS generator 215 can be configured to provide data to a synchronization layer 240 of the transcoder 200 to coordinate the synchronization of the emitted data.

FIGURE 3 illustrates a flow chart of an example of a process 300 that analyzer 205 can follow when analyzing the various elementary continuous transmissions in packets described above. The process 300 begins in block 302 when multimedia data 104 is received from a content provider 106 (FIGURE 1). The process 300 continues in block 304 in which the initialization of the analyzer 205 is carried out. The initialization can be activated by an independently generated acquisition command 306. For example, a process that is independent of the analyzer 205 and is based on a schedule of externally received television and channel alignment information can generate the acquisition command 306. Additionally, you can enter the buffer descriptors 308 of the real-time transport transmission (TS) in real time to aid in both initialization and The main processing.

As illustrated in block 304, initialization may include the acquisition of a command syntax check, completion of a first step PSI / PSIP / SI (program specific information / program information protocol and system / system information) processing, the performance of a processing specifically related to both the acquisition command and the verification of the PSI / PSIP / SI consistency, the allocation of a PES buffer for each PES and the setting of times (for example, for the alignment with the desired moment of beginning of the acquisition). The PES buffer maintains the analyzed ES data and communicates each analyzed ES data to a corresponding audio decoder 216, test encoder 220, decoder 214 or PTS generator of transcoder 215.

After initialization, the process 300 continues in block 310 for the main processing of the multimedia data 104 received. The processing of block 310 may include a filtering of the target packet identifier (PID), continuous PSI / PSIP / SI monitoring and processing and a time process (for example, to achieve a desired acquisition duration) so that the multimedia data incoming are passed to the appropriate PES buffers. As a result of the processing, the multimedia data of block 310, a program descriptor and a 'read' indication of the PES buffer are generated, which will have an interface with the decoder 214 (FIGURE 2) as described herein. then.

55 E06804241 04-11-2011

After block 310, the process 300 continues in block 314, in which the completion of the analysis operations takes place, including the generation of a timer switch and the release of the PES buffers after consumption. It is indicated that the PES buffers will exist for all relevant elementary continuous transmissions of the program cited in its descriptor such as audio, video and subtitle transmissions.

With reference again to FIGURE 2, the analyzer 205 sends the audio ES 208 to an audio decoder 216 corresponding to the transcoder implementation and provides the encoded text 216 to the synchronization layer 240 and the decoding of the information Audio. The subtitle information 212 is supplied to a text encoder 220. The hidden subtitling (CC) data 218 from a decoder 214 is also provided to the text encoder 220, which encodes the subtitle information 212 and the CC data 218 in a format affected by transcoder 200.

The analyzer / decoder 202 also includes the decoder 214, which receives the video ES 206. The decoder 214 can generate metadata associated with the video data, decodes the elementary continuous transmission in video packets encoded in raw video 224 (for example , in a standard definition format) and processes hidden video captioning data in the continuous transmission of video ES.

FIGURE 4 shows a flowchart illustrating an example of a decoding process 400 that can be performed by decoder 214. Process 400 begins with the input of elementary video streaming data 206 in block 402. The process 400 continues in block 404 in which the decoder is initialized. Initialization can include a number of tasks, including the detection of a video sequence header (VSH), the completion of a first step of the VSH processing, the video sequence (VS) and the VS Display Extension (including the video format, primary colors and matrix coefficients) and the allocation of data buffers to store respectively the decoded image, the associated data and the hidden subtitling (CC) data. Additionally, the "read" information of the video PES buffer 406 provided by the analyzer 205 (for example, which can be generated by process 300 in block 310 of FIGURE 3) is entered.

After initialization in block 404, process 400 continues in block 408 in which the main processing of the video ES is performed by the decoder 214. The main processing includes extracting the 'read' information from memory. Intermediate of the video PES or “interface” on the availability of new data, decoding of the video ES, reconstruction and storage of pixel data in the synchronization of image boundaries, the generation of video metadata already / v and its storage in the image limits and the storage of the CC data in the image limits. The results of block 410 of main processing 408 include the generation of sequence descriptors, decoders of decoded image buffer, metadata buffer descriptors and CC data buffer descriptors.

After main processing 408, process 400 continues in block 412 in which it performs a completion process. The termination process may include the determination of the termination conditions, including no new data appearing for a particular duration above a predetermined threshold, the detection of a final code sequence and / or the detection of an explicit termination signal . The finalization process may also include the release of the buffers of the decoded image, the associated metadata and the CC data after consumption by a preprocessor which will be described below. The process 400 ends in block 414 in which it enters a waiting state until a video ES is received as input.

Preprocessor

FIGURE 2, and in more detail FIGURE 6, illustrate a sample aspect of a preprocessor 226 that can use content information for one or more peprocessing operations. Preprocessor 226 receives metadata 222 and "raw" video data 224 decoded from the analyzer / decoder 202. Preprocessor 226 is configured to perform certain types of processing on video data 224 and metadata 222 and provide processed multimedia (by example, reference tables of the base layer, reference tables of the improvement layer, bandwidth information, content information) and video to the decoder 228. Such preprocessing of the multimedia data can improve visual quality, scaling and efficiency of data compression. In general, the preprocessor 226 receives video sequences provided by the decoder 214 in the analyzer / decoder 202 and converts the video sequences into progressive video sequences for further processing (eg coding) by the encoder 228. In some aspects, the preprocessor 226 can be configured for numerous operations, including reverse telecine, deinterlacing, filtering (for example, elimination of artificialities, de-planing, block suppression and noise suppression), resizing (for example, reduction of spatial resolution from standard definition up to a Quarter of Graphic Video Matrix (QVGA)) and generation of a GOP structure (for example calculating the generation of a complexity map, scene change detection and flash / flash detection).

The preprocessor 226 may use the decoder metadata to affect one or more of the operations of

55 E06804241 04-11-2011

preprocessing Metadata may include information that relates to, describes or classifies the content of multimedia data ("content information"); In particular, metadata may include a classification of content. In some aspects, metadata does not include the desired content information for encoding operations. In such cases the preprocessor 226 can be configured to determine the content information and use the content information for preprocessing operations and / or provide the content information to other components of the transcoder 200, for example the decoder 228. In some aspects, preprocessor 226 may use said content information to influence the division of the GOP, determine the appropriate type of filtering and / or determine the encoding parameters that are communicated to an encoder.

FIGURE 6 shows an illustrative example of several in process blocks that can be included in preprocessor 226 and illustrate the processing that can be performed by preprocessor 226. In this example, preprocessor 226 receives metadata and video 222, 224 and provides 614 output data comprising metadata and video (processed) to encoder 228. Typically, there are three types of video that can be received. First, the received video can be progressive video, in which deinterlacing is not required. Second, the video data may be telecine video, interlaced video converted from film sequences of 24 images per second, in which case a reverse telecine operation may be required. Third, the video can be non-interlaced telecine video. Preprocessor 226 can process these types of video as described below.

In block 601, preprocessor 226 determines whether the received video data 222, 224 is progressive video. In some cases, this can be determined from metadata if the metadata contains such information or through the processing of the video data itself. For example, a reverse telecine process, described below, can determine whether the received video 222 is progressive video. If it is, the process continues in block 607 in which the filtering operations (for example noise suppression) are performed on the video to reduce noise, such as Gaussian white noise. If the video data 222, 224 is not progressive video, in block 600 the process proceeds to block 604 to a phase detector 604.

The phase detector 604 distinguishes between video that originated in a telecine and the one that begins in a standard broadcast format. If the decision is made that the video was from telecine (the decision path YES coming out of the phase detector 604), the telecine video is returned to its original format in reverse telecine 606. The redundant frames are identified and they delete and the fields derived from the same video frame are recomposed into a complete image. Since the sequence of the reconstructed film images were photographed at regular intervals 1/24 of a second, the process of estimating the movement performed on a GOP 612 splitter or decoder 228 is more accurate using the inverted telecine images instead of Telecine data, which has an irregular time base.

In one aspect, the phase detector 604 makes certain decisions after the reception of a video frame. These decisions include: (i) if the present video is of a telecine output and the 3: 2 extraction phase is one of the 5 phases P0, P1, P2, P3 and P4 shown in FIGURE 38 and (ii) the Video was generated as conventional NTSC. That decision is indicated as phase P5. These decisions appear as outputs of the phase detector 604 shown in FIGURE 2. The route from the phase detector 604 labeled "YES" acts on the reverse telecine 606, indicating that the correct adaptation phase has been supplied so that you can classify the fields that were formed from the same photographic image and combine them. The route from the phase detector 604 labeled "NO" acts similarly on deinterlacer 605 to separate an apparent NTSC frame into fields for optimal processing. The phase detector 604 can continuously analyze the video frames because different types of video can be received at any time. As an example, video can be inserted according to the NTSC standard in the video as an advertisement. After reverse telecine, the resulting progressive video is sent to a noise suppressor (filter) 607 that can be used to reduce white Gaussian noise.

When conventional NTSC video is recognized (the NO path from phase detector 601), it is transmitted to deinterlacer 605 for compression. Deinterlacer 605 transforms the interlaced fields into progressive video and noise suppression operations can then be performed on the progressive video. An illustrative example of the deinterlacing process is described below.

Traditional analog video devices such as televisions reproduce the video in an interlaced manner, that is, such devices transmit even numbered scan lines (even fields) and odd numbered scan lines (odd fields). From the point of view of signal sampling, this is equivalent to a space-time subsample in a pattern described by:

 Θ (x, y, n), if ymod2 = 1 for even fields,

F (x, y, n) = Θ (x, y, n), if ymod2 = 1 for odd fields, [1]

Otherwise, deleted,

in which Θ refers to the original frame image, F indicates the interlacing field and (x, y, n) represent the horizontal, vertical and temporal positions of a pixel respectively.

55 E06804241 04-11-2011

Without loss of generality, it can be assumed that n = 0 is an even field throughout this disclosure so that Equation 1 above is simplified as

Θ (x, y, n), if ymod2 = n mod2,

F (x, y, n) = [2]

Otherwise, deleted,

Since the reduction is not performed in the horizontal dimension, the subsample pattern can be represented in the following coordinate n ~ y.

The goal of a deinterlacer is to transform the interlaced video (a sequence of fields) into progressive non-interlaced frames (a sequence of frames). In other words, interpolate the odd and even fields to "recover" or generate the complete picture frames. This can be represented by Equation 3:

F (x, y, n), ymod2 = n mod2,

F (x, y, n) = [3]

or Fi (x, y, n), otherwise,

in which Fi represents deinterlacing results for lost pixels.

FIGURE 40 is a block diagram illustrating certain aspects of an aspect of a deinterlacer 605 that uses Wmed filtering and motion estimation to generate a progressive frame from interlaced multimedia data. The upper part of FIGURE 40 shows a motion intensity map 4002 that can be generated using information from the current field, two previous fields (PP Field and P Field) and two subsequent fields (Next Field and Next to Next Field) . The motion intensity map 4002 classifies or divides, the current frame into two or more different levels of motion and can be generated by a temporal space filtering, described in more detail in this document below. In some aspects, the motion intensity map 4002 is generated to identify static areas, slow moving areas and fast moving areas as described with reference to Equations 4-8 below. A temporary space filter, for example the Wmed 4004 filter, filters interlaced multimedia data using criteria based on the motion intensity map and produces a temporary temporary space deinterlacing box. In some aspects, the Wmed filtering process involves a horizontal proximity of [-1, 1], a vertical proximity of [-3, 3] of five adjacent fields, which are represented by the five fields (Field PP, Field P, Current Field, Next Field, Next to Next Field) illustrated in FIGURE 40, in which Z-1 represents a delay of a field. In relation to the Current Field, the Next Field and the P Field are non-parity fields and the PP Field and the Next Next Field are parity fields. The proximity used for space-time filtering refers to the spatial and temporal location of fields and pixels actually used during the filtering operation and can be illustrated as an opening, as shown, for example, in Figures 6 and 7.

The deinterlacer 605 may also include a noise suppressor (noise suppressor filter) 4006 configured to filter the temporary space deinterlacing frame generated by the Wmed filter 4004. The noise suppression of the temporary space deinterlacing frame makes the search process of more precise subsequent movement especially if the sequence of interlaced multimedia data of origin is contaminated with white noise. This can also at least partially remove the step between even and odd rows in a Wmed image. The noise suppressor 4006 can be implemented as a variety of filters that include a noise reducer based on a reduction of the wavelets and a Wiener wavelet filter. A noise reducer can be used to eliminate noise from the candidate Wmed frame before it is further processed using motion compensation information and can eliminate the noise that is present in the Wmed frame and retain the present signal regardless of the frequency content of the signal. Various types of noise suppression filters can be used, including wavelet filters. Ripples are a class of functions used to locate a given signal in both space and scaling domains. The fundamental idea behind the wavelets is to analyze the signal at different scales or resolutions so that small changes in the representation of the wavelet produce a correspondingly small change in the original signal.

A reduction of the wavelets or a Wiener wavelet filter can also be applied as the noise suppressor. The reduction of the wavelets consists of a transformation of the wavelength of the signal with noise, followed by a reduction of the small wavelet coefficients to zero (or the smallest value), while the large coefficients are left unaffected. Finally, an inverse transformation is performed to obtain the estimated signal.

Noise suppression filtering reinforces the precision of motion compensation in noisy environments. The suppression of noise by reduction of the wavelet can involve the reduction in the domain of transforms of the wavelet, and typically comprises three stages: a linear direct waveform transform, a noise suppression by nonlinear reduction and a linear reverse waveform transform . The Wiener filter is an optimal MSE filter that can be used to improve images degraded by the addition of noise and blur. Such filters are generally known in the art and are described, for example, in "Ideal spatial adaptation by wavelet shrinkage", referenced

50 E06804241 04-11-2011

previously and by the document of S. P. Ghael, A. M. Sayeed and R. G. Baraniuk, “Improvement Wavelet denoising via empirical Wiener filtering” Proceedings of SPIE, vol. 3169, p. 389-399, San Diego, July 1997.

In some aspects, a noise suppression filter is based on an aspect of a biortogonal cubic B-spline wavelet filter (4, 2). Such a filter can be defined by the following direct and reverse transforms.

31 −11 −2

h (z) = + (z + z) + (z + z) (direct transform) [4]

42 8

and 5 −15 −23 −33 2 -

g (z) = z - (1+ z) - (z + z) - (z + z 4) (inverse transform) [5]

4328 32

The application of a noise suppression filter can increase the accuracy of motion compensation in a noisy environment. The implementations of such filters are further described in "Ideal spatial adaptation by wavelet shrinkage", D.L. Donoho and I.M. Johnstone, Biometrika, vol. 8, p. 425-455, 1994.

The lower part of FIGURE 40 illustrates an aspect for the determination of motion information (eg candidates for motion vector, motion estimation, motion compensation) of interlaced multimedia data. In particular, FIGURE 40 illustrates a motion estimate and a motion compensation scheme that is used to generate a provisional motion compensated progressive frame of the selected frame and then combined with the Wmed provisional frame to form a "final" progressive frame resulting, shown as the framework to the current deinterlaced 4014. In some aspects, the candidates (or estimates) of the motion vector ("MV") of the interlaced multimedia data are provided to the deinterlacer from the external estimators of the movement and are used to provide a starting point for the bidirectional motion estimator and compensator ("ME / MC") 4018. In some aspects, a MV 4022 candidate selector uses the previously determined MVs for adjacent blocks for MV candidates of the blocks that are they are processing, just like the MV of the blocks previously processed, for example the blocks in a previous table io deinterlacing 4020. Motion compensation can be performed bidirectionally, based on the previous deinterlacing frame 70 and a following Wmed frame (for example future) 4008. A current Wmed frame 4010 and a current offset moving frame are mixed, or combined. "MC") 4016, by means of a combiner 4012. A resulting deinterlaced current frame 4014, now a progressive frame, is provided to the ME / MC 4018 for use as a previous deinterlaced frame 4020 and is also communicated externally to the deinterlacer 605 for processing later.

It is possible to decouple deinterlacing prediction schemes comprising interpolation between fields from interpolation between fields with a deinterlacing scheme Wmed + MC. In other words, Wmed temporal space filtering can be used primarily for interpolation purposes between fields, while interpolation between fields can be performed during motion compensation. This reduces the signal-to-noise ratio of the resulting Wmed, although the visual quality after motion compensation is applied is more satisfactory, because bad pixels of the inaccurate prediction mode decisions between fields will be removed from the process. Wmed filtering.

After the appropriate reverse telecine or deinterlacing process, in block 608, the progressive video is processed for suppression of scaling and resampling (eg resizing). In some aspects of resampling, a polyphase resampler for resizing image size is implemented. In an example of the sampling reduction, the relationship between the original and the resized image can be p / q, in which p and q are prime integers to each other. The total number of phases that is p. The cutoff frequency of the polyphase filter in some aspects is 0.6 for resizing factors around 0.5. The cutoff frequency does not exactly match the resizing ratio to reinforce the high frequency response of the resized sequence. This inevitably allows some stepping. However, it is well known that the human eye prefers definite but slightly staggered images to staging and blurry free images.

FIGURE 41 illustrates an example of a multi-phase resampling, showing that the phases in the resizing ratio is ¾. The cutoff frequency illustrated in FIGURE 41 is also ¾. The original pixels are illustrated in the previous figure with vertical axes. A synchronism function centered around the axes is also drawn to represent the filtering waveform. Because the cutoff frequency was chosen to be exactly the same as the resampling ratio, the zeros of the synchronization function only overlap the pixel positions after resizing, FIGURE 41 illustrated with crosses. To find a pixel value after resizing, the contribution can be accumulated from the original pixels as shown in the following equation:

∞

v (x) = Lu (i) × senc (πf (i - x)) [6]

C i = −∞

55 E06804241 04-11-2011

in which fc is the cutoff frequency. The above 1-D multi-phase filter can be applied to both the horizontal dimension, the vertical dimension.

Another aspect of resampling (resizing) is to consider overbarrido. In an NTSC television signal, an image has 486 scan lines and in digital video it could have 720 pixels in each scan line. However, not the entire image is visible on television due to mismatches between size and screen format. The part of the image that is not visible is called an overdraft.

To help broadcasters put useful information in the visible area by as many televisions as possible, the Society of Motion Picture and Television Engineering (SMPTE) defined specific sizes of the action box called the safe area of action and the secure area of titles . See the best practices of SMPTE RP 27.3-1989 on Specifications for Safe Action and Safe Title Areas Test Pattern for Television Systems. The safe area of action is defined by the SMPTE as the area in which "all significant action must take place". The secure title area is defined as the area in which "all useful information can be confined to ensure visibility on most domestic television receivers."

For example, with reference to FIGURE 25, the secure data area 2510 occupies the center of 90% of the screen, giving a 5% edge all around. The secure title area 2505 occupies the center of 80% of the screen, giving a 10% edge. With reference now to FIGURE 26, because the secure title area is so small, to add more content in the image, some stations will put text in the safe area of action, which is inside the white rectangular box 2615.

Normally you can see black borders on the overbarrido. For example, in FIGURE 26 black borders appear on the upper side 2620 on the lower side 2625 of the image. These black borders can be eliminated in the overbarrido, because the H.264 video uses extension of the limits in the estimation of the movement. Extended black edges can increase the residual part. Conservatively, the limits can be cut by 2% and then resize. The filters for resizing can be generated accordingly. Truncation is done to eliminate overbarrido before a reduction of multi-phase sampling.

With reference again to FIGURE 6, the progressive video then proceeds to block 610 in which the block suppression and unscrewing operations are performed. Two types of artifice, "blocks" and "banding," commonly occur in video compression applications. Block artificialities occur because the compression algorithms divide each frame into blocks (for example 8x8 blocks). Each block is reconstructed with some small errors and the errors on the edges of a block often contrast with the errors on the edges of the adjacent blocks, making the boundaries between blocks visible. On the contrary, the artificialities of rings appear as distortions around the edges of the characteristics of the image. Ring artificialities occur because the encoder discards too much information in the quantification of high frequency DCT coefficients. In some illustrative examples, both block suppression and re-stripping may use low-pass FIR (finite pulse response) filters to hide these visible artifice.

In one example block suppression processing, a block suppression filter can be applied to all the edges of the 4x4 blocks of a frame, except the edges in the frame boundaries and any edge for which the process of disabling is disabled. block suppression filter. The filtering process will be based on macro blocks after the completion of the frame construction process with all the macro blocks processed in a box to increase the macro block addresses. For each macro block, the vertical edges are filtered first, from left to right and then the horizontal edges are filtered from top to bottom. The luma block suppression filter is performed on four edges of 16 samples and the block suppression filter process for each chroma component is performed on two edges of 8 samples, for the horizontal direction and for the vertical direction, such as shown in FIGURE 39. The sample values above and to the left of the current macro block that may already have been modified by the operation of the block suppression process over the previous macro blocks should be used as input for the filter process block suppression in the current block macro and can be modified further during the filtering of the current block macro. Modified sample values during vertical edge filtering can be used as input for horizontal edge filtering for the same macro block. A block suppression process can be invoked for luma and chroma components separately.

In an example of the stripping process, a 2-D filter can be applied adaptively to soften the areas near the edges. Border pixels undergo little or no filtering to avoid blurring.

GOP Divider

After block suppression and unraveling, the progressive video is processed by a GOP 612 splitter. GOP positioning can include detection of tap changes, generation of complexity maps (e.g., time, spatial bandwidth maps) and adaptive GOP division. Each of these is described below.

55 E06804241 04-11-2011

A. Scene change detection

Shooting detection refers to the determination of when a frame in a group of images (GOP) displays data indicating that a scene change has taken place. In general, within a GOP. The frames may not have significant changes in any of two or three (or more) adjacent frames or there may be slow changes

or quick changes. Naturally, these scene change classifications can be further divided into a higher level of changes depending on a specific application or need.

The detection of changes of shot or scene is important for efficient video coding. Typically, when a GOP is not changing significantly, a frame I at the beginning of a GOP is followed by a number of predictive frames that can sufficiently encode the video so that subsequent decoding and viewing of the video is visually acceptable. However, when a scene is changing, both abruptly and slowly, additional frames I and less predictive coding (frames P and frames B) may be needed to subsequently produce visually acceptable decoded results.

The systems and procedures for detection and coding of sockets that improve the performance of existing coding systems are described below. Such aspects can be implemented in the GOP splitter 612 of the preprocessor 226 (FIGURE 7) or included in an encoder device that can operate with or without a preprocessor. Such aspects use statistics (or metrics) that include statistical comparisons between adjacent frames of video data to determine if an abrupt scene change has taken place, a scene is slowly changing or if there are camera flash lights in the scene that can make Video coding especially complex. Statistics can be obtained from a preprocessor and then sent to an encoding device or can be generated in an encoding device (for example, by a processor configured to perform motion compensation). The resulting statistics help in the decision of the scene change detection. In a system that does not transcode, there is often a suitable preprocessor or a configurable processor. If the preprocessor performs a deinterlacing aided by motion compensation, motion compensation statistics and ready for use are available. In such systems, a shot detection algorithm may slightly increase the complexity of the system.

The illustrative example of a shot detector described herein only needs to use statistics from a previous frame, a current frame and a following frame and consequently has a very low latency. The shooting detector differentiates several different types of shooting events, including abrupt scene switching, cross fading and other slow scene changes and camera flash lights. By determining the different types of shooting events with different encoder strategies, the coding efficiency and visual quality are improved.

Scene change detection can be used for any video encoding system to conserve bits intelligently by inserting a frame I at fixed intervals. In some aspects, the content information obtained by the preprocessor (for example, either incorporated into metadata or calculated by preprocessor 226) can be used for scene change detection. For example, depending on the content information, threshold values or other criteria described below can be dynamically adjusted for different types of video content.

Video encoding normally works in a structured group of images (GOP). A GOP normally begins with an intracoded frame (Table I), followed by a series of P (predictive) or B (bidirectional) frames. Typically, a table I can store all the data required to display the table, a table B relies on the data in the preceding and following tables (for example, containing only the data that changes from the preceding table or is different from the data in the table next) and a P chart contains data that has changed since the preceding chart. In common use, frames I are interspersed with frames P and frames B in the encoded video. In terms of size (for example number of bits used to encode the frame), frames I are typically much larger than frames P which in turn are larger than frames

B.

For an efficient coding, transmission and decoding process, the length of a GOP should be long enough to reduce the loss of efficiency of the large frames I and short enough to combat the mismatch between encoder and decoder or channel impediments. In addition, macro blocks (MB) can be internally encoded in P frames for the same reason.

Scene change detection can be used for a video encoder to determine an appropriate GOP length and insert frames I based on the length of the GOP, instead of inserting a frame I often not necessary at fixed intervals. In a practical continuous streaming video system, the communication channel is affected by bit errors or packet loss. Where to place the I or MB I frames can significantly impact the quality of the decoded video and the viewing experience. An encoding scheme is to use intra-coding frames for images or parts of images that have a significant change against the images or parts of previous images located together. Normally these regions cannot be predicted effectively and efficiently with an estimation of movement and coding can be performed more efficiently if such regions are exempted from intra frame coding techniques (for example using

55 E06804241 04-11-2011

coding of frames B and frames P). In the context of channel impediments, these regions are likely to suffer from propagation of errors, which can be reduced or eliminated (or being close to doing so) by coding intra frames.

Parts of the video GOP can be classified into two or more categories, in which each region may have a different intra frame coding criteria that may depend on the particular implementation. As an example, the video can be classified into three categories: abrupt scene changes, cross fading and other slow scene changes and camera flash lights.

Abrupt scene changes include frames that are significantly different from the previous frame, normally produced by a camera operation. Since the content of these frames is different from the previous frame, the abrupt scene change frames should be encoded as frames I.

Cross fade and other slow scene changes include the slow change of scenes, usually caused by computer processing of the camera shots. Gradually mixing two different scenes may be more pleasing to human eyes, but it is a challenge to video coding. Motion compensation cannot reduce the bit rate of those frames effectively, and more intra MB can be updated for these frames.

Camera flash lights or camera flash events take place when the contents of a frame include camera flashes. Such flashes are of relatively short duration (for example a frame) and extremely bright so that the pixels in a frame representing the flashes exhibit an unusually high luminance relative to the corresponding area in an adjacent frame. Camera flash lights shift the luminance of an image sharply and quickly. Normally the duration of a camera flash is shorter than the duration of temporary masking of the human vision system (HVS), which is typically defined as 44 ms. Human eyes are not sensitive to the quality of these short bursts of brightness and therefore can be coded roughly. Because flash light frames cannot be handled effectively with motion compensation and are poor candidates for prediction of future frames, the rough coding of these frames does not reduce the coding efficiency of future frames. Scenes classified as flash lights should not be used to predict other frames due to artificially high luminance and other frames cannot be used effectively to predict these frames for the same reason. Once identified, these frames can be removed because they may require a relatively high amount of processing. One option is to remove the camera flash light boxes and encode a DC coefficient instead; Such a solution is simple, computationally fast and saves many bits.

When any of the previous categories of frames is detected, a shooting event is declared. Shooting detection not only useful for improving the quality of coding, it can also help in the identification of the indexed search of the video content. An illustrative aspect of a scene detection process is described herein below. In this example, a shot detection process first calculates information, or metric, for a selected frame that is being processed for shot detection. Metrics can include bidirectional motion estimation information and video processing compensation and other luminance based metrics.

To perform a bidirectional motion estimation / compensation, a video sequence can be reprocessed with a bidirectional motion compensator that adapts each 8x8 block of the current frame with blocks in two of the most adjacent frames of the adjacent frames, one in the past and one in the future. The motion compensator produces metric motion and difference vectors for each block. Figure 29 is an illustration showing an example of the adjustment of the pixels of a current frame C to a past frame P and a future frame (or following) N, and represents the motion vectors of the adjusted pixels (past motion vector MVP, MVN future motion vector). A description of a generation of the bidirectional motion vector and related coding with reference to FIGURE 32 is described herein in general.

After determining the bidirectional movement information (for example movement information that identifies the MB (better adjusted) in the corresponding adjacent frames), additional metrics can be generated (for example by a motion compensator in the GOP 612 splitter or other suitable component) through several comparisons of the current table with the following table and the previous table. The motion compensator can produce a difference metric for each block. The difference metric can be a sum of difference of squares (SSD) or a sum of absolute difference (SAD). Without loss of generality, SAD is used here as an example.

For each frame, an SAD relationship is calculated, also referred to as a "contrast ratio", as follows:

ε + SADP

γ = [6]

ε + SADN

in which SADP and SADN are the sum of the absolute differences of the direct and inverse difference metric,

40 E06804241 04-11-2011

respectively. It should be noted that the denominator contains a small positive number ε to balance the effect of the unit on the denominator. For example, if the previous table, the current table and the following table are identical, the motion search would lead to SADP = SADN = 0. In this case, the previous calculation generates γ = 1 instead of 0 or infinity.

A luminance histogram can be calculated for each frame. Typically multimedia images have a luminance depth of 8 bits (for example the number of "bins"). The luminance depth used for the calculation of the luminance histogram according to some aspects can be set to 16 to obtain the histogram. In other aspects, the luminance depth can be set up to an appropriate number that may depend on the type of data being processed, the available computing power or other predetermined criteria. In some aspects, the luminance depth can be set dynamically based on a calculated or received metric, such as the content of the data.

The equation below illustrates an example of the calculation of a luminance histogram difference (lambda):

N - N

L

Pi Ci i = 1

λ =

 [7]

N

where NPi is the number of blocks in the 1st bin for the previous frame and NCi is the number of blocks in the 1st bin for the current frame and N is the total number of blocks in a frame. If the luminance histogram difference of the previous and current table are completely different (or disjoint), then λ = 2.

Using this information, a table difference metric (D) is calculated as follows:

γ

D = C + Aλ (2λ + 1) [8]

γ P

ε + SAD ε + SAD

where A is a constant chosen by application, γ = P and γ = PP.

C ε + SADN P ε + SADC

The selected (current) frame is classified as an abrupt scene change frame if the difference metric in the frame satisfies the criteria shown in Equation 9:

γ

D = γ CP + Aλ (2λ + 1) ≥ T1 [9]

in which A is a constant chosen by application, and T1 is a threshold.

In an example simulation it shows that, by setting A = 1 and T1 = 5, a good detection performance is achieved. If the current frame is an abrupt scene change frame, then γC should be large and γP should be

γ

little. The C ratio can be used instead of γC only so that the metric is normalized to the level of

γ P context activity.

It should be noted that the above criterion uses the lambda luminance histogram difference (λ) in a non-linear manner. FIGURE 16 illustrates that λ * (2λ + 1) is an inverse function. When λ is small (for example near zero) it is hardly pre-emphasized. The larger λ is made, the greater emphasis is placed on the function. With this pre-emphasis, for any λ greater than 1.4, an abrupt scene change is detected if the threshold T1 is set to 5.

The current table is determined to be a cross fade or a slow scene change if the scene intensity metric D satisfies the criteria shown in Equation 5:

T2 ≤ D <T1 [10]

for a certain number of continuous frames, in which T1 is the same threshold used previously and T2 is another threshold value.

A flash light event usually causes the luminance histogram to jump to a brighter side. In this illustrative camera aspect, the statistics of the luminance histogram is used to determine if the current frame comprises camera flash lights. A shot detection process can determine whether the luminance of the current frame less is greater than the luminance of the previous frame at a certain threshold T3 and the luminance of the current frame is greater than the luminance of the next frame at the threshold T3, as shown Equations 11 and 12.

YC –YP ≥T3 [11]

17 E06804241 04-11-2011

YC –YN ≥T3 [12]

If the above criteria is not met, the current frame is not classified as comprising camera flash lights. If the criterion is met, the shot detection process determines whether a SADP backward difference metric and SADN forward difference metric are greater than a certain T4 threshold, as illustrated in the Equations

5 below:

SADP ≥ T4 [13]

SADN ≥ T4 [14]

where YC is the average luminance of the current frame, YP is the luminance using the previous frame, YN is the average luminance of the next frame, and SADP and SADN are the forward and backward difference metrics

10 associated with the current table.

The shooting detection process determines camera flash events by first determining whether the luminance of a current frame is greater than the luminance of the previous frame and the luminance of the next frame. If not, the frame is not a camera flash event; but if yes, it could be. The shot detection process can then assess whether the backward difference metric is greater than a T3 threshold and if the forward difference metric

15 is greater than a threshold T4; If both conditions are satisfied, the shot detection process classifies the current frame as having camera flash lights. If the criterion is not satisfied, the table is not classified as any type of taking events or it can be given a default classification that identifies the coding to be performed in the table (for example, table jump encoded as a table I) .

Some example values for T1, T2, T3 and T4 have been shown above. Typically, these values of

20 thresholds are selected through tests of a particular shot detection implementation. In some aspects, one more of the threshold values T1, T2, T3 and T4 are predetermined and such values are incorporated into the tap sorter in the coding device. In some aspects, one more of the threshold values T1, T2, T3 and T4 can be set during processing (for example dynamically) based on the use of information (eg metadata) supplied to the tap sorter or based on the information calculated by the classifier of

25 shots for himself.

Video coding using the shot detection information is typically performed in the encoder, but is described here to complete the disclosure of the shot detection. With reference to FIGURE 30, an encoding process 301 can use the shot detection information to encode the video based on the shot detected in the frame sequence. Process 301 continues in block 303 and checks 30 to see if the current frame is classified as an abrupt scene change. If so, in block 305 of the current frame it can be encoded as a frame I and a GOP limit can be determined. If not, process 301 continues in block 307; If the current frame is classified as a part of a scene that slowly changes in block 309 the current frame and other frames in the slowly changing scene can be encoded as a predictive frame (for example a P frame or a B frame). The process 301 then continues towards block 311 in which

35 checks if the current frame will be classified as a flash light scene that includes camera flashes. If so, in block 313 the frame can be identified for special processing, for example, its elimination or coding with a DC coefficient for the frame; if not, no classification of the current table is made and the current table can be coded according to other criteria, coded as a table I or removed.

In the aspect described above, the amount of difference between the frame to be compressed and its two frames

Adjacent 40 is indicated by the table D difference metric. If a significant amount of unidirectional change in luminance is detected, it means a cross-fade effect in the frame. The more prominent the cross fade, the greater the gain will be obtained through the use of B squares. In some aspects, a modified table difference metric is used as shown in the equation below.

dP - dN

−α + 2α

×

D, yes Y

−Δ≥ C ≥ + Δ + Δ≤ C ≤ −Δ,

AND AND OR AND AND AND

P N P N

D1

=

[fifteen]

d + d

P N

D, otherwise

45 in which dP = | YC - YP | and dN = | YC - YN | are the difference in luma between the current and the previous table and the difference in luma between the current and the next table, respectively, Δ represents a constant that can be determined in a normal experimentation since it can depend on the implementation and α It is a weighting variable that has a value between 0 and 1.

B. Generation of the bandwidth map 50 The preprocessor 226 (FIGURE 6) can also be configured to generate a bandwidth map that can be

You can use for multimedia data encoding. In some aspects, a content classification module 18

E06804241 04-11-2011

712 in encoder 228 (FIGURE 7) generates the bandwidth map instead.

The human visual quality V can be a function of both the coding complexity C and the assigned bits B (also referred to as bandwidth). FIGURE 5 is a graph illustrating this relationship. It should be noted that the coding complexity metric C considers spatial and temporal frequencies from the point of view of human vision. For distortions more sensitive to human eyes, the complexity value is correspondingly higher. It can typically be assumed that V is monotonously decreasing in C and monotonously increasing in B.

To achieve a constant visual quality, a bandwidth (Bi) is assigned to the object i (frame or MB) to be

coded that satisfies the criteria expressed in the two equations immediately below:

Bi = B (Ci, V)

 [16]

B = L and Bi

 [17]

In the two immediately preceding equations, Ci is the coding complexity of the object i, B is the total available bandwidth and V is the visual quality achieved for an object. Human visual quality is difficult to formulate as an equation. Therefore, the equation set out above is not defined precisely. However, if the 3-D model is assumed to be continuous in all variables, the bandwidth ratio (Bi / B) can be treated as unchanged within the proximity of a pair (C, V). The bandwidth ratio βi is defined in the equation shown below:

Bi

β =

 [18]

iB

The bit allocation can then be defined as expressed in the equations below:

βi = β (Ci)

 1 = LBi for (Ci, V) ∈δ (C0, V0) [19]

i

in which δ indicates the "neighborhood".

The complexity of coding is affected by human visual sensitivity, both spatial and temporal. Girod's model of human vision is an example of a model that can be used to define spatial complexity. This model considers the local spatial frequency and ambient lighting. The resulting metric is called Dcsat. At a processing point in the process, if an image is to be intra-encoded or inter-encoded it is not known and bandwidth relationships are generated for both. The bits are assigned according to the relationship between βINTRA of different video objects. For intra-encoded images, the bandwidth ratio is expressed in the equation below:

βINTRA = β0INTRA log10 (1 + αINTRA Y2 Dcsat) [20]

In the previous equation, Y is the average luminance component of a macro block, αINTRA is a weighting factor for the square of the luminance and the Dcsat term that follows it, β0INTRA is a normalization factor to guarantee 1 = LBi. For example, a value for αINTRA = 4 achieves a good visual quality. Information

i

of content (for example a classification of content) can be used to set αINTRA to a value that corresponds to a good visual quality level desired for the particular content of the video. In one example, if the video content includes a news broadcast with a "talking head", the visual quality level can be set lower because the information image or part that can be viewed from the video can be considered as less important than the audio part and less bits can be assigned to encode the data. In another example, if the video content comprises a sporting event, the content information can be used to set αINTRA to a value that corresponds to a higher level of visual quality because the images displayed may be more important to an observer. and consequently more bits can be assigned to encode the data.

To understand this relationship, it should be noted that bandwidth is logarithmically allocated with the complexity of the coding. The square term of luminance Y2 reflects the fact that coefficients with larger magnitudes use more bits to encode. To prevent the logarithm from entering negative values, the unit is added to the term in the parentheses. Logarithms can also be used with other bases.

Temporal complexity is determined by measuring a frame difference metric, which measures the difference between two consecutive frames taking into account the amount of movement (motion vectors). With a table difference metric such as the sum of the absolute differences (SAD).

55 E06804241 04-11-2011

Bit allocation for inter-encoded images may consider spatial as well as temporal complexity. This is expressed below:

2)) [21]

βINTER = β0INTER log10 (1 + α⋅SSD ⋅ D exp (- γ

MVP + MVN

INTER csat

In the previous equation MVP and MVN are the forward and backward motion vectors of the current MB (see FIGURE 29). It can be indicated that Y2 in the intra-coding bandwidth formula is replaced

by the sum of the square differences (SSD). To understand the role of

MVP + MVN

2 in the equation

above, note the following characteristics of the human visual system: the areas that experience a

smooth, predictable movement (a small

MVP + MVN

2) attract attention and can be followed by the eye and

They typically cannot tolerate more distortion than in fixed regions. However, the areas that experience a

fast or unpredictable movement (a great

MVP + MVN

2) cannot be followed and a

significant quantification. Experiments show that αINTRA = 1, γ = 0.001 achieves a good visual quality.

C. Adaptive GOP Division

In another illustrative example of the processing that can be performed by the preprocessor 226, the GOP Divider 612 of FIGURE 6 can also adaptively change the composition of a group of images encoded together and an example using MPEG2 is explained with reference. Some old video compression standards (for example MPEG2) do not require a GOP to have a regular structure, although one can be imposed. The MPEG2 sequence always begins with a frame I, that is one that has been encoded without reference to previous images. The MPEG2 GOP format is normally predisposed in the encoder by fixing the GOP spacing of the P or predictive images that follow Table I. P frames are images that have been partly predicted from previous I or P images. . The frames between the beginning frame I and the successive frames P are encoded as frames B. A frame "B" (B refers to bidirectional) can use the previous and next images I or P both individually and simultaneously as a reference. The number of bits needed to encode a frame I on average exceeds the number of bits needed to encode a frame P; in the same way the number of bits needed to encode a frame P exceeds the average necessary for a frame B. A frame that is skipped, if used, will not require any bit for its representation.

The concept underlying the use of P and B frames and, in the most recent compression algorithms, the jump of frames to reduce the rate of data needed to represent the video is the elimination of temporal redundancy. When the temporal redundancy is high - that is, there is a small change from image to image - the use of P, B or skipped images efficiently represents the continuous transmission of video, because previously decoded I or P images are subsequently used as references to decode other P or B images.

The division of the adaptive GOP is based on the use of this concept adaptively. The differences between frames are quantified and a decision is made automatically to represent the image using frames I, P, B or skipped after the appropriate tests on the quantified differences have been performed. An adaptive structure has advantages not available in a fixed GOP structure. A fixed structure will ignore the possibility that a small change in content has taken place; An adaptive procedure would allow many more B frames to be inserted between each I and P or two P frames, thereby producing the number of bits necessary to adequately represent the sequence of frames. Conversely when the change in video content is significant, the efficiency of the P frames is greatly reduced because the difference between the predicted and reference frames is too large. Under these conditions, the adjustment of the objects may fall outside the motion search regions or the similarity between the adjustment objects is reduced due to the dispersion produced by changes in the camera angle. At that point the P or I frames that are adjacent to the P frame should be chosen as closer to each other and fewer B frames should be inserted. A GOP said it cannot make that adjustment.

In the system disclosed in this document, these conditions are automatically detected. The structure of the GOP is flexible and is done to adapt to these changes in the content. The system evaluates a frame difference metric, which can be considered as measuring the distance between frames, with the same additive distance properties. Conceptually, given the frames F1, F2 and F3 that have distances between frames d12 and d23, the distance between F1 and F3 is taken as being at least d12 + d23. Table assignments are made based on the similar distance metric.

The GOP splitter operates by assigning image types to frames as received. The type of image indicates the prediction procedure that may be required in the coding of each block:

Images I are encoded without reference to other images. Since they remain solitary they provide access points in the continuous transmission of data when decoding can begin. A coding type I is assigned to a frame if the "distance" to its predecessor frame exceeds a scene change threshold.

P images can use the previous I or P images for compensated motion prediction. Use

55 E06804241 04-11-2011

blocks in the previous fields or frames that can be moved from the block that is predicted as a basis for coding. After the reference block is subtracted from the block under consideration, the residual block is coded, typically using the discrete cosine transform for the elimination of spatial redundancy. A type of modification P is assigned to a frame if the "distance" between it and the last frame assigned to be a frame P exceeds a second threshold, which is typically less than the first.

B-frame images can use the previous and next images, P or I for motion compensation as described above. A block in an image B can be predicted forward, backward or bidirectionally or it could be internally encoded without reference to other frames. In H.264 a reference block can be a linear combination of up to 32 blocks of as many frames. If the table cannot be assigned to be a type I or P, it is assigned as a type B, if the "distance" from it to its immediate predecessor is greater than a third threshold, which is typically less than the second threshold.

If the frame cannot be assigned to become an encoded B frame, a “skipped frame” status is assigned. This frame can be skipped because it is virtually a copy of a previous frame.

The evaluation of a metric that quantifies the difference between adjacent frames in the order of display is the first part that takes place in this processing. This metric is the distance referred to above; With it, each frame is evaluated for its own type. Therefore, the spacing between adjacent frames I and P or two consecutive frames P can be variable. The calculation of the metric begins with the processing of the video frames with a block-based motion compensator, the basic video compression unit, usually consisting of 16x16 pixels, being a block, although other block sizes such as 8x8 are possible , 4x4 and 8x16. For frames that consist of two deinterlaced fields, movement compensation can be performed based on one field, the reference blocks being searched in the fields rather than in the frames. For a block in the first field of the current frame there is a forward reference block in the fields of the frame that follow it; in the same way there is a backward reference block in fields of the table that immediately precedes the current field. The current blocks are mounted in a compensated field. The process continues with the second field of the table. The two offset fields combine to form a compensated frame back and forth.

For frames created in reverse telecine 606, the reference block search is based on frames only, since there are only reconstructed film frames. There are two reference blocks and two differences, forward and backward, also leading to a compensated frame forward and backward. In summary, the motion compensator produces motion vectors and difference metrics for each block; but a block is part of an NTSC field in the event that the output of deinterlacer 605 is being processed and is part of a film frame if the output of the reverse telecine is processed. Note that the differences in the metric are evaluated between a block in the field or frame being considered and a block that best fits it, both in a preceding field or frame and in a field or frame that follows immediately, depending of whether the difference is being evaluated forward or backward. Only luminance values enter this calculation.

The motion compensation stage thus guarantees two sets of differences. These are between blocks of current luminance values and luminance values in the reference blocks taken from frames that are immediately ahead and immediately behind frame one in time. The absolute value of each difference forward and each difference backward is determined for each pixel and each one is added separately across the entire frame. Both fields include in the two sums when deinterlaced NTSC fields that comprise a frame are processed. In this form are SADP and SADN, the absolute values added of the differences forward and backward.

For each table an SAD relationship is calculated using the relationship,

ε + SADP

γ = [22]

ε + SADN

in which SADP and SADN are the aggregate absolute values of the forward and backward differences, respectively. A small positive number ε is added to the numerator to prevent the "zero division" error. A term ε similar to the denominator is added, further reducing the sensitivity of γ when both SADP and SADN are close to zero.

In an alternative aspect, the difference can be the SSD, the sum of the different squares and SAD, the sum of the absolute differences, or the SATD in which the blocks of the pixel values are transformed by applying the Transform of Discrete cosine two-dimensional to them before the differences in the elements of the block are taken. The sums are evaluated through the video area, although a smaller area can be used in other aspects.

The luminance histogram of each frame is also calculated as received (without motion compensation). The histogram operates in the DC coefficient, that is, the coefficient (0,0), in the 16x16 coefficient matrix which is

50 E06804241 04-11-2011

the result of the application of the two-dimensional Discrete Cosine Transform to the luminance value block if available. Equivalently, the average value of the 256 luminance values in the 16x16 block in the histogram can be used. For images whose luminance depth is eight bits, the number of bins is set to 16. The following metric evaluates the histogram difference

1 16

λ = L

N - N

 [2. 3]

Pi Ci 4 i = 1

In the previous one, NPi is the number of blocks in the previous frame in the 1st bin and NCi is the number of blocks in the current frame that belong in the 1st bin, N is the total number of blocks in a frame.

These intermediate results meet to form the difference metric of the current table as

γ

D = C + λ (2λ + 1) [24]

γ P

where γC is the SAD relationship based on the current table and γP is the SAD relationship based on the previous table. If a scene has a smooth movement and its luma histogram barely changes, then D ≈ 1. If the current frame

γ

visualize an abrupt scene change so γC will be large and γP should be small. The relation C is used in

γ P instead of γC alone so that the metric is normalized to the level of context activity.

FIGURE 42 illustrates the process of assigning compression types to frames. D, the current table difference defined in Equation 19, is the basis for decisions made regarding table assignments. As indicated by decision block 4202, if the table under consideration is the first in a sequence, the decision path marked YES is followed to block 4206, thereby declaring the table as a table I. The cumulative table differences are set to zero in block 4208 and the process returns (in block 4210) to the starting block. If the frame being considered is not the first frame in a sequence, the path marked NO is followed from block 4202 in which the decision was made and in test block 4204 the current frame difference is tested against the threshold of scene change. If the current frame difference is greater than that threshold, the decision path marked YES is followed to block 4206, again leading to the assignment of a table I.

If the current frame difference is less than the scene change threshold, the NO path is followed to block 4212 in which the current frame difference is added to the cumulative frame difference. Continuing through the flowchart, in decision block 4214 the accumulated frame difference is compared with the threshold t, which is generally less than the scene change threshold. If the cumulative frame difference is greater than t, the control is transferred to block 4216 and the frame is assigned as a P frame; the cumulative frame difference is then reset to zero in step 4218. If the cumulative frame difference is less than t, the control is transferred from block 4214 to block 4220. There the current frame difference is compared with τ, which is less than t. If the current frame difference is smaller than τ, the frame is assigned to be skipped in block 4222 and then the process returns; if the current frame difference is greater than τ, the frame is assigned to be a frame B in block 4226.

Encoder

Referring back to FIGURE 2, transcoder 200 includes an encoder 228 that receives processed metadata and raw video from preprocessor 226. Metadata can include any information originally received in source video 104 and any information calculated by preprocessor 226. The encoder 228 includes a first-pass encoder 230, a second-pass encoder 232 and an encoder 234. The encoder 228 also receives an input from the control of transcoder 231 which can provide information (e.g. metadata, resistance information versus errors, content information, encoded bit rate information, balance information between the base layer and the enhancement layer and quantization information) from the encoder of the second step 232 to the encoder of the first step 230, the recoder 234 as well as preprocessor 226. Encoder 228 encodes the received video use Do the content information received from the preprocessor 226 and / or the content information generated by the encoder 228 itself, for example, by the content classification module 712 (FIGURE 7).

FIGURE 7 illustrates a block diagram of functional modules that can be included in an example two-step encoder that can be used for the encoder 228 illustrated in FIGURE 2. Various aspects of the modules are shown in FIGURE 7 functional, although FIGURE 7 and the description herein do not necessarily address all the functionality that can be incorporated into an encoder. Accordingly, certain aspects of the functional modules are described below following the explanation of the coding of the base and improvement layers below.

Coding of the base layer and the improvement layer 22

60 E06804241 04-11-2011

The encoder 228 may be an SNR scalable encoder, which can encode the raw video and metadata from the preprocessor 226 into a first set of encoded data, also referred to herein as a base layer and one or more additional groups of encoded data, also referred to herein as improvement layers. An encoding algorithm generates the coefficients of the base layer and the improvement layer which, when decoded, can be combined in the decoder when both layers are available for decoding. When both layers are not available, the coding of the base layer allows it to be decoded as a single layer.

An aspect of said multi-layer coding process is described with reference to FIGURE 31. In block 321, a frame I is coded with fully intracoded macro blocks (intracoded MB). In H.264, intra-encoded MBs in I-frames are decoded with a fully exploited spatial prediction, which provides a significant amount of coding gain. There are two submodes: intra 4x4 and intra 16x16. If the base layer is to take advantage of the coding gain provided by the spatial forecast, then the base layer needs to be encoded and decoded before the coding and decoding of the improvement layer. A coding and decoding of the tables I is used in two steps. In the base layer, a quantization parameter of the base layer QPb allows the transformation coefficients a rough quantization jump size. The difference in pixels between the original frame and the reconstructed base layer frame is encoded in the enhancement layer. The enhancement layer uses a quantization parameter QPe that allows a finer quantization jump size. The coding means, such as encoder 228 of FIGURE 2, can perform the coding in block 321.

In block 323, an encoder encodes the base layer data and the improvement layer data for the P and / or B frames in the GOP being processed. The coding means, such as the encoder 228 can perform the coding in block 323. In block 325, the coding process checks whether there are more P or B frames to encode. The coding means, such as the scalable encoder in SNR 228 can perform act 325. If more P or B frames remain, step 323 is repeated until all the frames in the GOP have finished being encoded. Tables P and B are composed of intra-encoded macro blocks (intracoded MB), although there may be intra-encoded MB in tables P and B as will be explained below.

In order for the decoder to distinguish between the data of the base layer and the improvement layer, the encoder 228 encodes an additional header information, in block 327. The types of additional header information include, for example, data identifying the number of layers, data that identifies a layer as a base layer, data that identifies a layer as an improvement layer, data that identifies the interrelationships between layers (such as, layer 2 is an improvement layer for base layer 1, or layer 3 is an improvement layer for base layer 2) or data identifying a layer as a final improvement layer in a chain of improvement layers. Additional header information may be contained in headers connected to the data of the base and / or enhancement layer to which they belong, or be contained in separate data messages. The coding means, such as the encoder 228 of FIGURE 2, can perform the process in block 327.

To have a single layer decoding, the coefficients of the two layers are combined before inverse quantization. Therefore the coefficients of the two layers have to be generated interactively; otherwise this could introduce a significant amount of additional header. One reason for the increase in the additional header is that the coding of the base layer and the coding of the improvement layer could use different time references. An algorithm is needed to generate the coefficients of the base layer and the improvement layer, which can be combined in the decoder before quantification when both layers are available. At the same time, the algorithm should provide an acceptable base layer video when the enhancement layer is not available or the decoder decides not to decode the enhancement layer for reasons such as, for example, energy saving. The details of an illustrative example of such a process will be further explained below in the context of a brief explanation of the standard predictive coding immediately below.

P frames (or any intra-coded section) can take advantage of the temporal redundancy between a region in a current image and a region of better fit forecasting in a reference image. The location of the best fit forecast region in the reference frame can be encoded in a motion vector. The difference between the current region and the reference forecast region of the best fit is known as residual error (or prediction error).

FIGURE 32 is an illustration of an example of a process of building a P-frame in, for example, MPEG-4. Process 331 is a more detailed illustration of an example process that could take place in block 323 of FIGURE 31. Process 331 includes the current image 333 formed by 5 x 5 macro blocks, in which the number of macro blocks In this example it is arbitrary. A macro block consists of 16 x16 pixels. Pixels can be defined by an 8-bit luminance value (Y) and two 8-bit chrominance values (Cr and Cb). In MPEG the Y, Cr and Cb components can be stored in a 4: 2: 0 format, in which the Cr and Cb components are sampled in 2 in the X and Y directions. Hence, each macro block could consist of 256 Y components, 64 Cr components and 64 Cb components. The macro block 335 of the current image 333 is predicted from the reference image 337 at a point in time different from that of the current image 333. A search is made in the reference image 337 to locate the macro block of the adjustment better 339 that is closer,

55 E06804241 04-11-2011

in terms of the values of Y, Cr and Cb to the current macro block 335 that is being encoded. The location of the macro block of the best fit 339 in the reference image 337 is encoded in the motion vector 341. The reference image 337 may be a frame I or a frame P that a decoder will have reconstructed prior to the construction of the image current 333. The best-fit macro block 339 is subtracted from the current macro block 335 (a difference is calculated for each of the components Y, Cr and Cb) resulting in a residual error 343. The residual error 323 is coded with a Transformed Discrete Cosine (DCT) 2D 345 and quantified 347 below. Quantification 347 can be performed to provide spatial compression by, for example, assigning fewer bits to high frequency coefficients while more bits are assigned to low frequency coefficients. The quantification coefficients of the residual error 343, together with the motion vector 341 and the reference image 333 that identify the information, are encoded in the information representing the current macro block 335. The encoded information can be stored in memory for a future use or used for purposes of, for example, error correction or image enhancement, or transmitted over the 349 network.

The encoded quantified coefficients of the residual error 343, together with the encoded motion vector 341 can be used to reconstruct the current macro block 335 in the encoder for use as part of a reference chart for an estimation and compensation of the subsequent movement. The encoder can emulate the process of a decoder for this reconstruction of the P-frame. The emulation of the decoder will result in both the encoder and the decoder working with the same reference image. The reconstruction process, whether performed in an encoder, for additional inter-coding or in a decoder, is presented herein. The reconstruction of a P frame can be started after the reference frame (or a part of an image or frame being referenced) is reconstructed. The encoded quantified coefficients are quantified 351 and then a 2D inverse DCT, or IDCT, 353 is performed resulting in the decoded or reconstructed residual error 355. The encoded motion vector 341 is decoded and used to locate the macro block of the fit better and reconstructed 357 in the reference image already reconstructed 337. The reconstructed residual error 355 is then added to the macro block of the best reconstructed setting 357 to form the reconstructed macro block 359. The reconstructed macro block 359 can be stored in memory, independently displayed or in an image with other reconstructed macro blocks, or be further processed for image enhancement.

The B frames (or any section encoded with bidirectional prediction) can take advantage of the temporal redundancy between a region in a current image and a region of better fit prediction in a previous image and a region of better fit prediction in a subsequent image. The prediction region of the best posterior adjustment and the prediction region of the best prior adjustment combine to form a combined bidirectional prediction region. The difference between the region of the current image and the combined bidirectional prediction region of the best fit is a residual error (or prediction error). The locations of the best fit prediction region in the subsequent reference image and the best fit prediction region in the previous reference image can be encoded into two motion vectors.

FIGURE 33 illustrates an example of an encoder process for coding the coefficients of the base layer and the improvement layer that can be performed by the encoder 228. The base and improvement layers are encoded to provide scalable bit streams in SNR. FIGURE 33 represents an example for the coding of residual inter-error coefficients inter MB as would be done in step 323 of FIGURE 31. However, similar procedures could be used to encode intra-MB coefficients as well. The coding means such as the coding component 228 of FIGURE 2 can perform the process illustrated in FIGURE 33 and step 323 of FIGURE 32. The original video data (to be encoded) 361 (the video data comprises information of luma and chroma in this example) are introduced in a macro block loop of better adjustment of the base layer 363 and a macro block loop of the best adjustment of the improvement layer 365. The object of both loops 363 and 365 is to minimize the residual error that is calculated in adders 367 and 369 respectively. Loops 363 and 365 can be performed in parallel, as shown, or sequentially. Loops 363 and 365 include logic for searching in buffers 371 and 373, respectively, which contain reference tables, to identify the best fit macro block that minimizes the residual error between the best fit macro block and the original data. 361 (buffers 371 and 373 may be the same buffer). The residual errors of loop 363 and loop 365 will be different since the loop of the base layer 363 will generally use a rougher quantization stage size (a higher QP value) than the loop of the improvement layer 365. The blocks Transformation 375 and 377 transform the residual errors for each loop.

The transformed coefficients are then separated into coefficients of the base layer and the improvement layer in the selector 379. The separation of the selector 379 can take several forms, as explained below. A common feature of separation techniques is that the coefficient of the improvement layer, C’mej, is calculated to be a differential refinement to the coefficient of the base layer C’base. The calculation of the improvement layer to be a refinement of the base layer allows a decoder to decode the coefficient of the base layer by itself and have a reasonable representation of the image or combine the coefficients of the base and improvement layer and have A refined representation of the image. The coefficients selected by selector 379 are quantified

~~

′ ′

by quantifiers 381 and 383. The quantified coefficients Cbase and Cmej (calculated with the quantifiers

381 and 383 respectively) can be stored in memory or transmitted through a network to a decoder.

50 E06804241 04-11-2011

To adjust the reconstruction of the macro block in a decoder, the quantifier 385 quantifies the residual error coefficients of the base layer. The quantified residual error coefficients are reversed 387 added 389 to the macro block of the best fit found in buffer 371, resulting in a reconstructed macro block that conforms to what will be reconstructed in the decoder. Quantifier 383, quantifier 391, reverse transformer 393, adder 397 and buffer 373 perform similar calculations in the improvement loop 365 to those performed in the base layer loop 363. In addition, adder 393 is used to combine the coefficients of the improvement layer and the base layer quantified in the reconstruction of the improvement layer. The quantifier and quantifier of the enhancement layer will generally use a finer quantizer stage size (a lower QP) than in the base layer.

FIGURES 34, 35 and 36 show examples of the coefficient selector processes of the base layer and the improvement layer that can be used in the selector 379 of FIGURE 33. The selection means such as the encoder 228 of the FIGURE 2 can perform the processes depicted in FIGURES 34, 35 and 36. Using FIGURE 34 as an example, the transformed coefficients are divided into coefficients of the base and improvement layer as shown in the following equations:

10, if Base and Cmej have opposite signs C 'base = [25] 1min (Cbase, Cmej), otherwise

C’mej = Cmej - Qb -1 (Qb (C’base)) [26]

in which the “min” function can be both a mathematical minimum and a minimum magnitude of the 2 arguments. Equation 25 is represented as block 401 and Equation 26 is represented as adder 510 in FIGURE 34. In Equation 26, Qb refers to the base layer quantifier 381 and Qb-1 refers to the layer quantifier base 385. Equation 26 converts the coefficient of the improvement layer into a differential refinement of the coefficient of the base layer calculated with Equation 25.

FIGURE 35 is an illustration of another example of a selector of the coefficient of the base layer and the improvement layer 379. In this example, Equation (.) Contained in block 405 represents the following.

−1

1C, if QQ (C) - C

<C

base b b base mej mej C 'base =

 [27] 10, otherwise

Adder 407 calculates the coefficient of the improvement layer as shown in the following two equations:

C’mej = Cmej - Qb -1 (Qb (C’base)) [28]

in which C’base is given by Equation 27.

FIGURE 36 is an illustration of another example of a selector of a base layer and an improvement layer 379. In this example, the coefficient of the base layer is not changed and the improvement layer is equal to the difference between the coefficient of the quantized / quantified base layer and the coefficient of the original improvement layer.

In addition to the residual error coefficients of the base and improvement layer, the decoder needs information that identifies how MBs are encoded. The coding means such as the coding component 228 of FIGURE 2 may encode the additional header information that may include a map of intracoded and intercoded parts, such as, for example, an MB map in which the macro blocks (or submacro blocks) are identified as being intracoded or intercoded (also identifying what type of intercoding, including, for example, direct, inverse or bidirectional) and which parts of the intercoded frames are referenced. In an example aspect, the MB map and the coefficients of the base layer are encoded in the base layer and the coefficients of the improvement layer are encoded in the improvement layer.

P frames and B frames can contain intra-encoded MB as well as inter MB. It is common for hybrid video encoders to use a rate distortion (RD) optimization to decide to encode certain macro blocks in P or B frames as intra-encoded MB. To have a single layer decoding where the intra-encoded MBs do not depend on the inter MB of the enhancement layer, no contiguous inter MB is used for spatial prediction of the intra-encoded MBs of the base layer. To keep the calculation complexity unchanged for decoding the enhancement layer, for intra-encoded MBs in the P or B frames of the base layer, refinement could be skipped in the enhancement layer.

Intracoded MBs in P or B frames may require many more bits than inter MBs. For this reason, intra-encoded MBs in P or B frames could only be encoded in the quality of the base layer with a higher QP. This will introduce some deterioration in video quality, but this deterioration should go unnoticed if it is refined in a final frame with the inter MB coefficients of the base and improvement layer as explained above. Two reasons make this deterioration unnoticed. The first is a characteristic of the human visual system (HVS) and the other is that the inter MB refine the intra MB. With objects that change position from the first

60 E06804241 04-11-2011

frame to the second frame, some pixels in the first frame are not visible in the second frame (information to be covered) and some pixels in the second frame are visible for the first time (information discovered). Human eyes are not sensitive to the visual information discovered and to be covered. So for the information discovered, even if it is coded with a lower quality, the eyes may not notice the difference. If the same information remains in the following table P, there will be a high probability that the next table P in the improvement layer can refine it because the improvement layer has a lower QP.

Another common technique introduced by intra-encoded MBs in P or B frames is known as Intra Refreshment. In this case, some MB are encoded as intra-encoded MB, even though the standard R-D optimization would dictate that they should be inter-encoded MB. These intra-encoded MBs, contained in the base layer, can be encoded either with a QPb or with a QPe. If the QPe is used for the base layer, then no refinement is needed in the improvement layer. If QPb is used for the base layer, then a refinement may be necessary, otherwise in the improvement layer, the quality drop will be noticeable. Since intercodes are more efficient than intracodes in the sense of coding efficiency, these refinements in the enhancement layer will be intercoded. In this way, the coefficients of the base layer will not be used for the improvement layer. Therefore, a quality improvement is obtained in the improvement layer without introducing new operations.

B frames are commonly used in the improvement layers due to the high compression qualities they offer. However, frames B may have to reference intra-encoded MB of a frame P. If the pixels in frame B were to be encoded in an enhancement layer quality, they may require too many bits due to the lower quality of the MB intra-encoded table P, as explained above. Taking advantage of the qualities of HVS, as explained above, the MB of frame B could be coded with a lower quality when they reference intra-encoded MB of lower quality P frames.

An extreme case of intra-encoded MB of P or B frames is when all MBs in a P or B frame are encoded in intra mode due to the presence of a scene change in the video being encoded. In this case, the entire frame can be encoded in the base layer quality without any refinement in the improvement layer. If a scene change occurs in a frame B and it is assumed that frames B are only encoded in the enhancement layer, then frame B could be encoded in the quality of the base layer or simply deleted. If a scene change occurs in a P frame, no change would be necessary, but the P frame could be deleted or encoded in a base layer quality. The scalable layered coding is further described in the Pending US Patent Application together with this No. [representative file reference number 050078] entitled "SCALABLE VIDEO CODING WITH TWO LAYER ENCODING AND SINGLE LAYER DECODING" and property of the assignee of the present.

Part of the first step encoder

FIGURE 7 shows an illustrative example of encoder 228 of FIGURE 2. The blocks shown illustrate various encoder processes that can be included in encoder 228. In this example, encoder 228 includes a first-pass portion 706 above a demarcation line 704, and a second pass portion 706 (which includes the functionality of a second pass encoder 232 and recoder 234 of FIGURE 2) below line 704.

The encoder 228 receives metadata and raw video from the preprocessor 226. The metadata may include any metadata received or calculated by the preprocessor 226, including metadata related to the video content information. The part of the first step 702 of the encoder 228 illustrates example processes that can be included in an encoding of the first step 702, which is described below in terms of its functionality. As one skilled in the art will know, such functionality can be performed in several ways (for example by hardware, software, firmware or a combination thereof).

Figure 7 illustrates an adaptive intra refreshment module (AIR). The AIR 710 module provides an input to an instantiation module in Table I 708 that instantiates a Table I based on metadata. The part of the first step 702 may also include a content classification module 712 configured to receive the metadata and the video and determine the content information in relation to the video. Content information can be provided to a bit allocation and rate control module 714, which will also receive metadata and video. The control bit allocation module 714 determines the bit rate control information and provides it to the mode decision module 715. The video content information can be provided to an intra-model (distortion) module 716 , which provides intra-coding distortion information to the 715 mode decision module and a rate-distortion scalability for the base and improvement layer module 718. The video metadata is provided to a motion estimation module. (distortion) 720 that provides inter-coding distortion information to the rate-distortion scalability module for the base and enhancement layers 718. The rate-distortion scalability module for the base and enhancement layers 718 determines the information of rate of a scalability distortion using distortion estimates from the motion estimation module 720 and the int distortion module ra model 716 that are provided to the mode decision module 715. The mode decision module 715 also receives inputs from the portion ordering module / MB 722. The portion ordering module / MB 722 receives inputs from a front resistance module. to errors 740 (shown in the part of the second step 706) and provides information on the alignment of

60 E06804241 04-11-2011

the video parts (portions) independently encodable with access unit limits for error resistance to the mode decision module 715. The mode decision module 715 determines the information of the coding mode based on its inputs and provides the "Better" coding mode to the part of the second step 706. A further illustrative explanation of some examples of said coding of the part of the first step 702 is described below.

As stated above, the content classification module 712 receives the metadata and raw video provided by the preprocessor 226. In some examples, the preprocessor 226 calculates the content information from the multimedia data and provides the content information. to the content classification module 712 (eg metadata), which can use the content information to determine a content classification for multimedia data. In some aspects, the content classification module 712 is configured to determine various content information from the multimedia data and can also be configured to determine a content classification.

The content rating module 712 can be configured to determine a different content rating for the video that has different types of content. The classification of different content may result in different parameters used in aspects of the coding of the multimedia data, for example, the determination of the bit rate (for example the bit allocation) for the determination of the quantification parameters, estimation of movement, scalability, resistance to errors, maintenance of optimal multimedia data quality through the channels and for fast channel change schemes (for example, forcing I frames periodically to allow a fast channel change). According to an example, encoder 228 is configured to determine a rate-distortion (R-D) utilization and bit rate assignments based on the content classification. The determination of the content classification allows multimedia data to be compressed to a quality level since it corresponds to a desired bit rate based on a content classification. Also, and by classifying the content of the multimedia data (for example, determining the classification of the content based on the human visual system), the perceptual quality resulting from the multimedia data communicated on a screen of a receiving device is made depending of video content.

As an example of a procedure that the content classification module 712 performs to classify the content, FIGURE 9 shows a process 900 illustrating an example process by which the content classification module 712 can operate. As shown, process 900 begins at input block 902 in which the content classification module 712 receives raw multimedia data and metadata. The process 900 then proceeds to block 904 in which the content classification module 712 determines the spatial information and the temporal information of the multimedia data. In some aspects, spatial and temporal information is determined by spatial and temporal masking (for example filtering). Spatial and temporal information can be determined based on metadata that includes scene change data and smoothing of motion vectors (MV). Process 900 then proceeds to block 912, which estimates spatial complexity, temporal complexity and sensitivity. The process 900 then proceeds to block 916 in which the content of the multimedia data is classified based on the results of the spatial, temporal and sensitivity data determined in blocks 904 and 912. Also in block 916, one can select a particular rate-distortion curve (RD) and / or the data of the RD curve can be updated. The process 900 then proceeds to the output block 918, in which the output may include a distortion complexity map or a value indicating the spatial and temporal activity (for example a content classification) and / or the selected R-D curves. Referring back to FIGURE 7, the content classification module 712 provides an output to a rate control bit allocation module 714, an intra model (distortion) module 716 and also for the frame instantiation module I 708, explained above.

Content Information

The content classification module 712 can be configured to calculate a variety of content information from the multimedia data, including a variety of content related metrics, including spatial complexity, temporal complexity, contrast ratio values, standard deviations and Table difference metrics, described further below.

The content classification module 712 can be configured to determine the spatial complexity and temporal complexity of multimedia data and also to associate a texture value with spatial complexity and a movement value with temporal complexity. The content classification module 712 receives preprocessed content information in relation to the content of the multimedia data that is being encoded from the preprocessor 226 or, alternatively, the preprocessor 226 can be configured to calculate the content information. As described above, the content information may include, for example, one or more Dcsat values, contrast ratio values, motion vectors (MV) and sum of absolute differences (SAD).

In general, multimedia data includes one or more sequences of images or frames. Each frame can be divided into blocks of pixels for processing. Spatial complexity is a broad term that generally describes a measurement of the level of spatial details within a picture. The scenes with areas

50 E06804241 04-11-2011

mainly flat or unchanged or few changes in lighting and chrominance will have low spatial complexity. Spatial complexity is associated with the texture of video data. Spatial complexity is based on, in this aspect, a human visual sensitivity metric called Dcsat, which is calculated for each block as a function of local spatial frequency and ambient lighting. Those skilled in the art will be aware of the techniques for using spatial frequency and lighting patterns and contrast characteristics of visual images to take advantage of the human visual system. A certain number of sensitivity metrics are known to take advantage of the perspective limitations of the human visual system and could be used with the procedure described herein.

Temporal complexity is a broad term used to describe in general a measurement of the level of movement in a multimedia data as reference between frames in a sequence of frames. Scenes (for example sequences of video data frames) with little or no movement have a low temporal complexity. The temporal complexity can be calculated for each macro block and can be based on the Dcsat value, motion vectors and the sum of the absolute pixel differences between a frame and another frame (for example a reference frame).

The frame difference metric gives a measurement of the difference between two consecutive frames taking into account the amount of movement (for example, the motion vector or MV) together with the residual energy represented as the sum of the absolute difference (SAD) between a forecaster and the current block framework. The difference in tables also provides a measure of bidirectional or unidirectional prediction efficiency.

An example of a frame difference metric based on the movement information received from a preprocessor that potentially performs a deinterlacing compensated by the movement is as follows. The deinterlacer performs a bidirectional motion estimate and therefore a bidirectional motion vector and SAD information is available. A difference of frames represented by SAD_MV can be deduced for each macro block as follows:

SAD_MV = log10 [SAD * exp (-min (1, MV))] [29]

where MV = Square_root (MVx2 + MVy2), SAD = min (SADN, SADP), in which SADN is the SAD calculated from the backward reference chart and SADP is the SAD calculated from the reference chart to ahead.

Another approach to estimating a difference in tables was described above with reference to Equations 6-8. A ratio of SAD (or contrast ratio) γ can be calculated as described above in Equation 6. A luminance histogram of each frame can also be determined, the histogram difference λ being calculated using Equation 7. The metric Difference of tables D can be calculated as shown in Equation 8.

In an illustrative example, a contrast ratio and a frame difference metric are used as follows to obtain a classification of the video content, which could reliably predict the characteristics in a given video sequence. Although described herein as what happens in encoder 228, a preprocessor 226 can also be configured to determine a content classification (or other content information) and pass the content classification to encoder 228 by means of metadata. The process described in the example below classifies the content into eight possible classes, similar to the classification obtained from an analysis based on the R-D curve. The classification process produces the output of a value in the range between 0 and 1 for each superframe depending on the complexity of the scene and the number of scene change events in that superframe. The content classification module in the preprocessor can execute the following steps (1) - (5) for each superframe to obtain a content classification metric from the frame contrast and the frame difference values.

one.

 Calculate the Contrast of Average Frame and the Deviation of Frame Contrasts from the contrast values of macro blocks.

2.

 Normalize the Frame Contrast and the Frame Difference values using the values obtained from simulations, which are 40 and 5 respectively.

3.

 Calculate a content classification metric using, for example, the generalized equation:

CCMétrica = CCW1 * Media_Contraste_Box_I + CCW2 * Media_Difference_Box -

- CCW3 * Contrast_Deviation_I ^ 2 * exp (CCW4 * Difference_Deviation_Box ^ 2) [30]

in which CCW1, CCW2, CCW3 and CCW4 are weighting factors. In this example, the values choose to be 0.2 for CCW1, 0.9 for CCW2, 0.1 for CCW3 and -0.00009 for CCW4.

4. Determine the number of scene changes in the super frame. In general, a super frame refers to a group of images or frames that can be displayed in a particular period of time. Typically, the period of time is one second. In some aspects, a super frame comprises 30 frames (for 30 video

E06804241 04-11-2011

images per second). In other aspects a super frame comprises 24 frames (video of 24 images per second). Depending on the number of scene changes, the execution of one of the following cases can be obtained.

(a) No Scene Changes: when there are no scene changes in a super frame, the metric is completely dependent only on the difference values of frames as shown in the following equation:

CCMétrica = (CCW2 + (CCW1 / 2)) * Half_Difference_Box -

- (CCW3 - (CCW1 / 2)) * 1 * exp (CCW4 * Deviation_Difference_Box ^ 2) [31]

(b) A Single Scene Change: When there is a single scene change observed in the super frame, the default equation would be used to calculate the metric, as shown below:

CCMétrica = CCW1 * Media_Contraste_Box_I + CCW2 * Media_Difference_Box -

- CCW3 * Contrast_Deviation_I ^ 2 * exp (CCW4 * Difference_Deviation_Box ^ 2) [32]

(c) Two Scene Changes: When it is observed that there are at least 2 scene changes in the given super frame, more weight is agreed for the last super frame than for the first given that the first one will be refreshed in any case by the last quickly, as shown in the following equation:

CCMétrica = 0.1 * Media_Contraste_Box_I1 + CCW1 * Media_Contraste_Box_I2 +

 + (CCW2–0.1) * Media_Difference_Box -

 - CCW3 * Contrast_Deviation_I1 ^ 2 * Contrast_Deviation_I2 ^ 2 *

  * exp (CCW4 * Deviation_Difference_Box ^ 2) [33]

(d) There are Three or more Scene Changes: if in the given super frame it is observed that it has more than 3 frames I (say N), more weight is given to the last frame I and all other frames I are given a weight of 0.05, as the following equation is shown:

CCMétrica = 0.05 * Media_Contraste_Box_I (1, ... N-1) + CCW1 * Media_Contraste_Box_I (N) +

+ (CCW2– (0.05 * (N-1))) * Medium_Difference_Box -

- CCW3 * Contrast_Deviation_I (N) ^ 2 * Contrast_Deviation_I (1, ... N-1) ^ 2 *

  * exp (CCW4 * Deviation_Difference_Box ^ 2) [34]

5. A correction for the metric can be used in the case of low moving scenes when the average frame difference is less than 0.05. A displacement of 0.33 (CC DISPLACEMENT) would be added to the CCMétrica.

The content classification module 702 uses the Dcsat value, motion vectors and / or the sum of the absolute differences to determine a value that indicates a spatial complexity for the macro block (or the designated amount of video data). The temporal complexity is determined by measuring the metric difference of frames (the difference between two consecutive frames taking into account the amount of movement, with motion vectors and the sum of the absolute differences between the frames).

In some aspects, the content classification module 712 can be configured to generate a bandwidth map. For example, the generation of the bandwidth map can be performed by the content classification module 712 if the preprocessor 226 does not generate a bandwidth map.

Determination of texture and movement values

For each macro block in multimedia data, the content classification module 712 associates a texture value with spatial complexity and a movement value with temporal complexity. The texture value refers to the luminescence values of the multimedia data, in which a low texture value indicates small changes in the luminescence values of the pixels adjacent to the data and a high texture value indicates large changes in the luminescence values of the pixels adjacent to the data. Once the texture and movement values are calculated, the content classification module 712 determines a content classification by considering both the movement and texture information. The content classification module 712 associates the texture for the classification of the video data with a relative texture value, for example "Low" texture, "Medium" texture or "High" texture, which generally indicates the complexity of the values of luminance of the macro blocks. Also, the content classification module 712 associates the calculated movement value for the classification of the video data with a relative movement value, for example, "Low" movement, "Medium" movement or "High" movement which generally indicate the amount of movement of

60 E06804241 04-11-2011

The macro blocks. In alternative aspects, fewer or more categories can be used for movement and texture. Next, a content classification metric can be determined by considering the associated texture and motion values.

FIGURE 8 illustrates an example of a classification chart that illustrates how texture and motion values are associated with a content classification. A person skilled in the art will be familiar with many ways to implement such a ranking chart, for example, in a search table or a database. The ranking chart is generated based on predetermined evaluations of the video data content. To determine the classification of the video data, a texture value of “Low”, “Medium” or “High” (on the “x-axis”) is cross-referenced with a movement value of “Low”, “Medium ”Or“ High ”(on the“ y axis ”). A classification of the content indicated by the intersection block is assigned to the video data. For example, a "High" texture value and a "Medium" movement value results in a classification of seven (7). FIGURE 8 illustrates several combinations of relative texture and motion values that are associated with eight different content classifications, in this example. In other aspects, more or less classifications can be used. An additional description of an illustrative aspect of the classification of the content is disclosed in the pending US Patent Application together with this US Patent Application No. 11/300 73,577 entitled "Content Classification for Multimedia Processing" filed on March 10, 2006, assigned to the assignee of this.

Assignment of rate control bits

As described herein, a classification of multimedia data content can be used in coding algorithms to effectively improve bit management while maintaining a constant in perceptual video quality. For example, classification metrics can be used in algorithms for scene change detection, bit rate allocation control coding and up-frame rate conversion (FRUC). Compressor / decompressor (codec) systems and digital signal processing algorithms are commonly used in video data communications and can be configured to conserve bandwidth, but there is a compromise between quality and conservation of bandwidth. band. The best codecs provide the best preservation of bandwidth while producing the least degradation of video quality.

In an illustrative example, the rate control bit allocation module 714 uses content classification to determine a bit rate (for example the number of bits allocated for encoding multimedia data) and stores the bit rate in memory for use by other encoder processes and components

228. A bit rate determined from the classification of video data can help conserve bandwidth while providing multimedia data with a consistent quality level. In one aspect, a different bit rate can be associated with each of the eight different content classifications and then that bit rate is used to encode multimedia data. The resulting effect is that although different classifications of multimedia data content are assigned to a different number of bits for encoding, the perceived quality is similar or consistent when viewed on a screen.

In general, multimedia data with a higher content classification are indicative of a higher level of movement and / or texture and are assigned more bits when encoded. Multimedia data with a lower classification (indicative of lower texture and movement) have fewer assigned bits. For multimedia data of a particular content classification, the bit rate can be determined based on a perceived target quality level selected for viewing multimedia data. The determination of the quality of multimedia data can be determined by humans who view and rate multimedia data. In some alternative aspects, multimedia data quality estimates can be made by automatic test systems that use, for example, signal-to-noise ratio algorithms. In one aspect, a set of standard quality levels (for example five) and a corresponding bit rate necessary to achieve that particular quality level are previously determined for the multimedia data of each content classification. To determine a set of quality levels, the multimedia data of a particular content classification can be evaluated by generating an Average Opinion Score (MOS) that provides the numerical indication of the perceived visual quality of the multimedia data when encoded using a certain bit rate. The MOS can be expressed as a unique number in the range of 1 to 5 in which 1 is the lowest quality perceived and 5 the highest quality perceived. In other aspects, the MOS can have more than five or less than five quality levels, and different descriptions of each quality level can be used.

The determination of multimedia data quality can be determined by humans who view and rate multimedia data. In some alternative aspects, estimates of multimedia data quality can be performed by automatic test systems that use, for example, signal-to-noise ratio algorithms. In one aspect, a set of standard quality levels (for example five) and a corresponding bit rate necessary to achieve that particular quality level are previously determined for the multimedia data of each content classification.

Knowing the relationship between the level of visually perceived quality and the bit rate for multimedia data of a certain content classification can be determined by selecting an objective quality level (by

60 E06804241 04-11-2011

desired example). The target quality level used to determine the bit rate can be previously selected, selected by a user, selected through an automatic process or a semi-automatic process, an entry from a user or from another process is required or dynamically selected by the user. coding device or system based on predetermined criteria. An objective quality level can be selected based on, for example, the type of coding application or the type of client device that will receive the multimedia data.

In the example illustrated in FIGURE 7, the rate control bit allocation module 714 receives both data from the content classification module 712 and the metadata directly from the preprocessor 226. The rate control bit allocation module 714 resides in the part of the first step of the encoder 228 and a fine adjustment module of the rate control 738 resides in the part of the second step 706. This aspect of the two-step rate control is configured so that the first step ( 714 rate control bit allocation module) performs a context-adaptive bit allocation with an anticipation of the superframe (for example, a long-term objective of the 256 bitps average bit rates) and limits the rate of peak and the second step (fine adjustment module of the rate control 738) refines the results of the first step in a two-layer scalability that performs the rate adaptation. Rate control works on four levels: (1) GOP level - controls the bit distribution of tables I, P, B and F so that they are not uniform within a GOP; (2) superframe level - controls firm limits on maximum superframe size; (3) frame level - controls bit requirements according to the spatial and temporal complexity of multimedia data frames, which are based on content information (for example a content classification) and (4) macro level block controls the bit allocation of macro blocks based on spatial and temporal complexity maps, which are based on content information (for example a content classification).

An example flow chart of the operation of the rate control module 714 is illustrated in FIGURE 10. As shown in FIGURE 10, the process 1000 begins at an input block 1002. The rate control module 714 receives several entries, none of which is necessarily illustrated by FIGURE 7. For example, the input information may include metadata from preprocessor 226, an objective bit rate, an encoder buffer size (or, as equivalent, time maximum delay for rate control), an initial rate control delay and frame rate information. Additional input information may include entries at the image group level (GOP), including, for example, a maximum super frame size, length and distribution of GOP P / B frames (including scene change information), arrangement of the base layers and desired improvement, a complexity-distortion metric for images in the GOP for about 30 future frames. Other input information includes inputs at the image level, including the complexity-distortion map for the current image (received from the content classification module 712), quantization parameters (QP) and bit division of the last 30 frames (adjusted through a sliding window). Finally, the input information at the macro block level (MB) includes, for example, the mean absolute difference (MAD) or macro blocks not jointly located (MB) in a reference image and an encoded block pattern (CBP) of macro blocks after quantification (if they will skip or not).

After the entries in block 1002, the process 1000 continues in block 1004 for the initialization of the bit stream encoding. Simultaneously, an initialization of the buffer 1006 is performed. Next, a GOP is initialized as shown in block 1008, with the bit allocation of the GOP 1010 received as part of the initialization. After the initialization of the GOP, the flow continues in block 1012, in which a portion is initialized. This initialization includes an update of the header bits as shown by block 1014. After the initializations of block 1004, 1008 and 1012 are made, the rate control (RC) for a basic or macro unit is carried out. block (MB) as shown by block 1016. As part of determining the rate control of a macro block in block 1016, inputs are received via the interfaces in encoder 228. These inputs may include the assignment of macro block bits (MB) 1018, an update of the parameters of the quadratic model 1020 and an update of the average absolute deviation from the average parameters ("MAD", a robust estimate of the dispersion) 1022. Next the process 1000 continues in block 1024 for the execution of operations after encoding an image 1024. This procedure includes receiving an update of the memory parameters. at intermediate level as shown by block 1026. Process 1000 then proceeds to output block 1028 where the rate control module 714 produces the output of the quantization parameters QP for each macro block MB to be used by a module of mode decision 715 as shown in FIGURE 7.

Motion estimation

The motion estimation module 730 receives metadata and raw video inputs from the preprocessor 226 and provides an output that may include the block size, distortion metric of the motion vectors and identifiers of the reference frame to a mode 715 decision module FIGURE 11 illustrates an example operation of the motion estimation module 720. As shown, process 1100 begins with an input 1102. At the frame level, module 720 receives the ID input of the reference frame and Motion vectors At the macro block level, input 1102 includes input pixels and reference frame pixels. The process 1100 continues in step 1104 in which the color motion estimation (ME) and the motion vector forecast are performed. To carry out these processes, several inputs are received including MPEG-2 motion vectors, and luma motion vectors MV 1106, smoothed from the vector of

60 E06804241 04-11-2011

movement 1108 and non-causal motion vectors 1110. Next, process 1100 continues in block 1112 where an algorithm or methodology for searching the motion vector is performed, such as hexagonal or diamond search procedures. The entries to the process in block 1112 may include the sum of the absolute difference (SAD), the sum of the square difference (SSD) and / or other metrics as shown by block 1114. Once the search for the motion vector, the process 1100 continues in the completion block 1116, where the completion processing is performed. The process 1100 then ends at the output block 1118, which leads to an output of the block size, motion vector (MV), distortion metric and identifiers of the Reference Table.

R-D scalability for the base and enhancement layer

FIGURE 13 illustrates an example flowchart of a scalability process 1300 that can be performed by the scalability RD module 718. The process 1300 begins at starter block 1302 and continues at block 1304 in which the module of RD 718 scalability receives an input from the motion estimation module 720 performs a motion estimate. The movement estimate relies on the entry of reference tables of the base layer, reference tables of the improvement layer and the original frame to be coded as indicated by block 1306. Such information can be calculated by the GOP divisor. and communicate to the RD module of scalability 718 by means of, for example, metadata. Process 1300 continues in block 1308 to determine the scalability information of the base layer data and the improvement layer data. The coding of the base layer is then performed as shown in block 1310, followed by the coding of the improvement layer in block 1312. The coding of the improvement layer can use the coding results of the base layer to having a prediction between layers as input, as illustrated by block 1314, is therefore done temporarily after coding the base layer. This is further described in the pending US Patent Application together with this No. [representative file / reference No. 050078] entitled "SCALABLE VIDEO CODING WITH TWO LAYER ENCODING AND SINGLE LAYER DECODING". After coding is complete, process 1300 ends in block 1316.

Ordination Portion / Macro block

The part of the first step 302 also includes a portion / macro block module 722, which receives an input from an error resistance module 740 in the part of the second step and provides portion alignment information to the decision module 715 mode. Portions are independently encoded video data clippings (entropy decoding). Access units (AU) are encoded video frames each comprising a set of NAL units that always contains exactly one primary encoded image. In addition to the primary encoded image, an access unit may also contain one or more redundant encoded images or other NAL units that do not contain portions of data divisions in portions of an encoded image. Decoding an access unit always results in a decoded image.

The frames can be time division multiplexed blocks or physical layer packages (called TDM capsules) that offer the highest diversity of times. A super frame corresponds to a unit of time (for example, 1 sec.) And contains four frames. Alignment of the portion limits and AU with the frame limits in the time domain results in the more efficient separation and location of corrupted data. During a deep fade, most of the contiguous data in a TDM capsule is affected by errors. Due to the diversity of times, the remaining TDM capsules have a high probability of being intact. Uncorrupted data can be used to recover and hide lost data from the affected TDM capsule. Similar logic applies to frequency domain multiplexing (FDM) in which frequency diversity is achieved through the separation of frequency subcarriers that modulate data symbols. Additionally, similar logic is applied to spatial diversity (through separation in transmitting and receiving antennas) and other forms frequently applied in wireless networks.

To align the portions and the AUs to the frames, the creation of the outer code block (FEC) and the encapsulation of the MAC layer should also be aligned. FIGURE 20 illustrates the organization of the encoded video data or a bit stream of portions and AU video. The encoded video may be formed by one or more bit streams, for example the bit stream of the base layer and the bit stream of the enhancement layer when layered video coding is applied.

The video bit stream comprises the AUs as illustrated in FIGURE 20 by Table 1 '2005, Table 3' 2010 and Table M '2015. The AUs are composed of portions of data, as illustrated by the Portion 1 2020, Portion 2 2025 and Portion N 2030. Each portion start is identified by a start code and provides an adaptation to the network. In general, table I or intra-coded AUs are large, followed by P-frames or forward predicted frames followed by tables B. The coding of a multi-portion AU incurs a significant overhead cost in terms of the rate of bits coded because spatial prediction across portions is restricted and portion headers also contribute to overhead. Because portion limits are resynchronization points, restriction of physical layer packets adjacent to portions controls errors since when a PLP is corrupted, the error is confined to the portion

55 E06804241 04-11-2011

in the PLP while if the PLP contained multiple portions or parts of multiple portions, the error will impact all portions or portions of portions in the PLP.

Since frames I are typically large, for example, of the order of tens of kbits, the overhead due to multiple portions is not a large proportion of the size of the total frame I or the total bit rate. Also, having more portions in an intra-encoded AU allows for better and more frequent resynchronization and more efficient spatial error concealment. Also, frames I carry the most important information in the video bit stream since frames P and B are predicted outside of frames I. Tables I also serve as random access points for the acquisition of a channel.

Referring now to FIGURE 21, careful alignment of frames I to frame limits and portions with an AU I also to frame limits, allows for more efficient error control, error protection (since if a portion that belonged to Table 1 2105 is lost, portions that belong to Table 2 2110 have a high probability of being intact because Table 2 2110 has a significant time separation from Table 1 2105) error recovery is Can perform by resynchronization and error concealment.

Because the P frames are typically of a size of the order of a few kbits, the alignment of the portions to a P frame and an integer number of P frames to the limits of the frame allows resistance against errors without a harmful loss of efficiency (for similar reasons to that of tables I). The concealment of temporary error can be used in such aspects. Alternatively, dispersing consecutive P frames so that they arrive in different frames provides an added time diversity between the P frames, which may be due to the fact that temporary concealment is based on motion vectors and data from previously reconstructed I or P frames. The B frames can be extremely small (hundreds of bits) to moderately large (a few thousand bits). Hence, the alignment of an integer number of frames B to the limits of the frame is desirable to achieve resistance against errors without a detrimental loss of efficiency.

Mode decision module

FIGURE 2 illustrates some examples of the operation of the mode decision module 715. As shown, the process 1200 begins in an input block 1202. In an illustrative example, the various information inputs to the mode decision module 715 include serving type, 4x4 intra cost, 16x16 intra cost 8x8 intraUV cost, 16x16 intraY mode, intraUV mode, motion vector data (MVD), quantization parameters (QP), SpPredMB4x4Y, SpPredMB16x16Y, SpPredMB8x8U, SpPredMB8x8V, rate distortion marker, raw YMB pixels, raw UMB pixels and raw VMB pixels. The process 1200 then proceeds to block 1204 encoding the initialization, which can be initiated by an input signal or an interface that directs the initialization of the encoder as indicated by block 1206. Initialization may include the establishment of allowed modes ( including jump, director), the adjustment of the mode weights (if required, by default they will be equal weights for all modes) and the setting of the buffers. After initialization, the process 1200 continues in block 1208 where the main processing for the mode decision is performed, including: the calculation of the macro block mode (MB) cost for each allowed mode, the weighting of each cost of the MB mode with a weighting factor and the selection of a cost mode of the minimum MB mode. The inputs involved in these operations include motion estimation (for example MVD and prediction) and spatial prediction (for example intra-costs and forecasts) as illustrated by blocks 1210 and 1212. The interface with the mode decision module 715 is an entropy coding in block 1214 which, among other things, improves the compression rate. The process 1200 continues with block 1216 where the buffers are adapted to pass information to the encoder of the part of the second step 706. Finally, the process 1200 continues in block 1218 where the "best" coding mode can be communicated to the encoder of the part of the second step 706.

Encoder of the part of the second step

Referring again to FIGURE 7, the part of the second step 706 of the encoder 228 includes an encoder module of the second step 232 for carrying out the second step of the coding. The second step encoder 232 receives the output from the mode decision module 715. The second step encoder 232 includes a Quantification module of the MC / Transformed 726 and a Zig Zag (ZZ) / Entropy 728 encoder. The results of the Second-step encoder 232 is output to a scalability module 730 and a bit stream packaging module 731, which produces output to a base and enhancement layer encoded for transmission by transcoder 200 via a synchronization layer 240 (illustrated in FIGURE 2). As shown, in FIGURE 2, it is indicated that the base and enhancement layers of the second pass encoder 232 and the recoder 234 are assembled together by the synchronization layer 240 in a packet PES 242 which includes the base and base layers. enhancement, a PES of data 244 (for example, CC and other text data), and an audio PES 246. It is indicated that the audio encoder 236 receives audio information from the encoder 218 and, in turn, encodes the information and produces the output of the encoded information 238 towards the synchronization layer 240.

55 E06804241 04-11-2011

Recoder

With reference again to FIGURE 7, the part of the second passage of the encoder 706 also includes a recoder 234, which corresponds to the decoder 234 of FIGURE 2, the recoder 234 also receives the output of the part of the first step 702 and includes parts of the Quantification of the MC / Transformed 726 and coding of ZZ / Entropy 728. Additionally, the scalability module 730 produces the output to the recoder 234. The recoder 234 produces the output of the resulting base and improvement layer from the recoder to the module of bit stream 731 for transmission to a synchronizer (for example the synchronization layer 240 shown in FIGURE 2). The encoder 228 of the example in FIGURE 7 also includes a fine adjustment module of the rate control 738 which provides a feedback of the bit stream packaging for both the Quantification module of the MC / Transformed 234 in the encoder of the second step 232 and for the ZZ / Entropy module 736 in the recoder 234 to help adjust the coding of the second step (for example, to increase compression efficiency).

Error Resistance Module

The example of encoder 228 illustrated in FIGURE 7 also includes an error resistance module 740 in the part of the second step 706. The error resistance module 740 communicates with the bit flow packaging module 700 31 and the portion / MB sorting module 722. The error resistance module 740 receives metadata from the preprocessor 228 and selects an error resistance scheme, for example, the alignment of the portion and the units of access to the limits of the frame , predictive hierarchy and adaptive intra refreshment. The selection of the error resistance scheme can be based on the information received in the metadata or on the information communicated to the error resistance module from the bit stream packaging module 731 and the portion / MB 722 sorting module The error resistance module 740 provides information to the portion / macro block (MB) sorting module in the part of the first step 702 to implement the resistance processes against selected errors. Video streams through error-prone environments can employ error resistance strategies and algorithms that can result in the presentation of clearer and less error data to a vision user. The description of resistance to errors below can be applied to any individual application or combination of existing or future, transport and physical layers or other technologies. The robustness and error algorithms integrate an understanding of the susceptibility to error properties and the error protection capabilities between the OSI layers in conjunction with the desirable properties of the communication system such as low latency and high performance. The processing of the resistance against errors can be based on the information of the content of the multimedia data, for example, on the classification of the content of the multimedia data. One of the main advantages is the recoverability of fade and multi-path channel errors. The error resistance approaches described below pertain specifically to processes that can be incorporated in the encoder 228 (for example in particular in the error resistance module 740 and in the portion / MB 722 sorting module) and can be Generically extend to data communication in error-prone environments.

Resistance against errors

For a hybrid compression system based on prediction, intra-encoded frames are independently encoded without any temporal prediction. Intercoded frames can be temporarily predicted from past frames (P frames) and future frames (B frames). The best forecaster can be identified through a search process in the reference chart (one or more) and a distortion measure such as SAD is used to identify the best fit. The predictive encoded region of the current frame can be a block of variable size and shape (16x16, 32x32, 8x4, etc.) or a group of pixels identified as an object through, for example, segmentation.

Temporal prediction typically extends over many frames (for example 10 to tens of frames) and is terminated when one frame is encoded as one frame I, the GOP typically being defined by the frequency of frames I. For maximum coding efficiency, a GOP is a scene, for example, the limits of the GOP are aligned with the limits of the scene and the scene change frames are encoded as frames I. The low motion sequences are made up of a relatively static background and the movement is generally limited to a foreground object. Examples of content of such low motion sequences include news programs and weather predictions where no more than 30% of the most viewed content is of this nature. In low motion sequences, most regions are intra-encoded and predicted frames refer back to frame I through intermediate predicted frames.

With reference to FIGURE 22, the intra-encoded block 2205 in frame I is the forecast for the intracoded block 2210 in the encoded frame (or AU) P1. In this example, the region of these blocks is a fixed part of the background. Through consecutive temporal prediction, the sensitivity of the intra-coded block 2205 to errors grows until it is a "good" forecast that also implies that its "importance" is higher. Additionally, the intra-coded block 2205, by virtue of its temporal prediction chain called the prediction chain, persists more time on the screen (for the duration of the scene in the example in the figure).

60 E06804241 04-11-2011

The prediction hierarchy is defined as the tree of blocks created based on their level of importance or persistence measurement with the parent at the top (intra-coded block 2205) and the children at the bottom. Note that the inter-coded block 2215 in P1 is in the second hierarchy level and so on. Leaves are blocks that end a prediction chain.

The prediction hierarchy can be created for video sequences regardless of the type of content (also for music and sports and not just news) and can be applied to the understanding of video (and data) based on prediction in general (this applies to all aspects described in this application). Once the prediction hierarchy is established, error resistance algorithms such as adaptive intra refreshment, described below, can be applied more effectively. The measure of importance can be based on the possibility of recovering a given block of errors such as through concealment operations and application of adaptive intra refresh to improve the resistance of the coded bit streams to errors. An estimate of the importance measurement can be based on the number of times a block is used as a foresight also called a persistence metric. Persistence metrics are also used to improve coding efficiency by stopping the propagation of the prediction error. Persistence metrics also increase bit allocation for blocks with higher importance.

Intra adaptive soda

Adaptive intra refreshment is a technique of resistance to errors that can be based on the information of the multimedia data content. In an intra-refresh process, some MBs are intra-encoded even though the standard R-D optimization would dictate that they should be inter-encoded MB. The AIR uses a motion-weighted intracool to introduce intracoded MB in P or B frames. These intracoded MBs, contained in the base layer, can be encoded either with a QPb or with a QPe. If the QPe is used for the base layer, then no refinement is needed in the improvement layer. If QPb is used for the base layer, then a refinement may be appropriate, otherwise in the improvement layer, the quality drop will be noticeable. Since inter-coding is more efficient than intra-coding in the sense of coding efficiency, these refinements in the enhancement layer will be inter-coded. In this way, the coefficients of the base layer will not be used for the improvement layer, and quality improvement in the improvement layer is introduced without introducing new operations.

In some aspects, the adaptive intra refresh can be based on the information of the multimedia data content (for example a classification of the content) instead of, or in addition to, based on the weighting of the movement. For example, if the content classification is relatively high (for example, scenes that have a high spatial and temporal complexity) the adaptive intra refresh can introduce relatively more intracoded MB in P or B frames. Alternatively, if the content classification was relatively low (indicating a less dynamic scene with low spatial and / or temporal complexity), the adaptive intra refresh can introduce less intra-encoded MB in tables P and B. Such metrics and procedures for improving error resistance can be apply not only in the context of wireless multimedia communications but towards data compression and multimedia processing in general (for example in graphics generation).

Channel Change Chart

A channel change frame (CSF) is defined herein as a broad term describing a random access frame inserted in an appropriate location in a continuous broadcast transmission for a fast channel acquisition and hence a channel change. fast between continuous transmissions in an emission multiplex. Channel change frames also increase robustness against errors, since they provide redundant data that can be used if the primary frame is transmitted with an error. A frame I or a progressive frame I such as the progressive decoder refresh box in H.264 typically serves as a random access point. However, frequent frames I (or short GOP, shorter than the durations of the scenes) results in a significant reduction in compression efficiency. Since intracoding blocks may be required for error resistance, random access and error resistance can be effectively combined through the prediction hierarchy to improve coding efficiency while maintaining robustness against mistakes.

The improvement of random access change and robustness against errors can be achieved in concert and can be based on content information such as a content classification. For sequences of low movement, the prediction chains are long and a significant part of the information required to reconstruct a super frame or scene is contained in table I that appeared at the beginning of the scene. Channel errors tend to be bursts and when a fade appears and the FEC fails and the channel coding, there is a strong residual error that fails to hide it. This is particularly severe for low movement sequences (and therefore a low bit rate) since the amount of encoded data is not significant enough to provide a good diversity of times within the video bit stream and because they they are highly understandable sequences that generate each bit useful for reconstruction. High-movement sequences are more robust against errors due to the nature of the content - more new information in each frame increases the number of intra-encoded blocks that can be decoded independently and are inherently more resistant to errors. The intra adaptive soft drink based on prediction hierarchy

55 E06804241 04-11-2011

it achieves high performance for high motion sequences and performance improvement is not significant for low motion sequences. Therefore, a channel change frame containing most of the frame I is a good source of diversity for low motion sequences. When an error appears in a super frame, the decoding of the consecutive frames starts from the CSF that retrieves the lost information due to the prediction and resistance to errors is achieved.

In the case of high-movement sequences such as sequences that have a relatively high content classification (for example 6-8), the CSF may consist of blocks that persist in the SF - those that are good predictors. All other regions of the CSF are not to be coded since they are blocks that have short prediction chains that imply that they are terminated with intra blocks. Therefore, the CSF still serves to recover from lost information due to prediction when an error occurs. The CSFs for low motion sequences are up to the size of the I frames and can be encoded with a lower bit rate by means of a stronger quantification, with which the CSFs for high motion sequences are much smaller that the corresponding tables I.

Resistance to errors based on the prediction hierarchy can work well with scalability and can achieve highly efficient layered coding. The scalability to support hierarchical modulation in physical layer technologies may require the splitting of video bitstream data with specific bandwidth ratios. These may not always be the ideal relationships for optimal scalability (for example, with the least overhead). In some aspects, a 2-layer scalability with a 1: 1 bandwidth ratio is used. Splitting the video bit stream into 2 layers of equal size may not be as efficient for low motion sequences. For low motion sequences, the base layer that contains all the header and metadata information is larger than the enhancement layer. However, since the CSFs for low motion sequences are larger, they clearly fit the remaining bandwidth in the enhancement layer.

The high movement sequences have enough residual information that a partition of the data in 1: 1 can be achieved with the least overhead. Additionally, a channel change frame for such sequences is much smaller for high motion sequences. Therefore, resistance to errors based on the prediction hierarchy can also work well with scalability for high motion sequences. The extension of the concepts explained above for moderate motion clips is possible based on the descriptions of these algorithms and the proposed concepts are applied for video coding in general.

Multiplexor

In some aspects of the encoder, a multiplexer can be used for encoding multiple continuous multimedia transmissions produced by the decoder and used to prepare the encoded bits for broadcast. For example, in the illustrative aspect of the encoder 228 shown in FIGURE 2, the synchronization layer 240 comprises a multiplexer. The multiplexer can be implemented to provide bit rate allocation control. The estimated complexity can be provided to the multiplexer, which can then allocate the available bandwidth for a collection of multiplexed video channels according to the coding complexity anticipated by those video channels, which then allows the quality of a relatively constant particular channel even if the bandwidth for the collection of multiplexed video streams is relatively constant. This provides a channel with a collection of channels to have a variable bit rate and a relatively constant visual quality, rather than a relatively constant bit rate and a variable visual quality.

FIGURE 18 is a block diagram illustrating a multiple coding system of continuous transmissions.

or multimedia channels 1802. Continuous multimedia transmissions 1802 are encoded by the respective encoders 1804, which are in communication with a multiplexer (MUX) 1806, which in turn is in communication with a transmission medium 1808. For example, continuous transmissions Multimedia 1802 can correspond to various content channels, such as news channels, sports channels, movie channels and the like. The encoders 1804 encode continuous multimedia transmissions 1802 in the encoding format specified by the system. While described in the context of the coding of continuous video transmissions, the principles and advantages of the disclosed techniques are generally applicable to continuous multimedia transmissions that include, for example, audio transmissions. The encoded multimedia continuous transmissions are provided to a multiplexer 1806, which combines the various encoded multimedia continuous transmissions and sends the combined continuous transmission to the transmission medium 1808 for transmission.

The transmission medium 1808 can correspond to a variety of media, such as, but not limited to, digital satellite communication, such as DirecTV®, digital cable, wired and wireless Internet communications, optical networks, cell phone networks and others Similar. Transmission medium 1808 may include, for example, modulation in a radio frequency (RF). Typically, due to spectral and other similar limitations, the transmission medium has a limited bandwidth and the data from multiplexer 1806 to the transmission medium is maintained with a relatively constant bit rate (CBR).

In conventional systems, the use of a constant bit rate (CBR) at the output of multiplexer 1806 may require continuous encoded multimedia or video transmissions that are introduced into multiplexer 1806

55 E06804241 04-11-2011

are also from CBR. As described in the background, the use of CBR when encoding video content can result in a variable visual quality, which is typically undesirable.

In the illustrated system, two or more of the encoders 1804 communicate an early coding complexity of the input data. One or more of the encoders 1804 can receive control of the bit rate adapted from the multiplexer 1806 in response. This allows an encoder 1804 that expects to encode relatively complex video to receive a higher bit rate or a higher bandwidth (more bits per frame) for those frames of the video in an almost variable bit rate form. This allows continuous multimedia streaming 1802 to be encoded with consistent visual quality. The extra bandwidth used by the particular encoder 1804 that encodes relatively complex video comes from the bits that would otherwise have been used for encoding other continuous video transmissions 1804 if the encoders were implemented to operate with bit rates. constants This keeps the output of multiplexer 1806 at a constant bit rate (CBR).

While an individual multimedia streaming 1802 may be relatively "bursts," that is, variable in the bandwidth used, the cumulative sum of multiple video streaming can be less bursts. The bit rate of the channels that are encoded with less complex video can be reassigned by, for example, multiplexer 1806, to the channels that are encoding relatively complex video and this can improve the visual quality of the combined video streams as a whole

The encoders 1804 provide the multiplexer 1806 with an indication of the complexity of a set of video frames to be encoded and multiplexed together. The output of multiplexer 1806 should provide an output that is not higher than the bit rate specified for transmission medium 1808. The indications of complexity can be based on the classification of content as explained above to provide a level of selected quality. The multiplexer 1806 analyzes the indications of complexity and provides the various encoders 1804 with an assigned number of bits or bandwidth and the encoders 1804 use this information to encode the video frames in the set. This allows a set of video frames to individually have a variable bit rate and still achieve a constant bit rate as a group.

Content classification can also be used to enable a conversion based on multimedia quality in general for any generic compressor. The classification of the content and the procedures and apparatus described herein may be used in multimedia processing based on quality and / or based on the content of any multimedia data. An example is its use in multimedia compression in general by any generic compressor. Another example is in the decompression or decoding in any compressor or decoder or post-processor such as in interpolation, resampling, improvement, restoration and presentation operations.

Referring now to FIGURE 19, a typical video communication system includes a video compression system consisting of a video encoder and a video decoder connected via a communication network. Wireless networks are a class of error-prone networks in which communication channels exhibit logarithmic-normal or overshadowed fading and multi-path fading in mobile scenarios in addition to loss of trajectory. To combat channel errors and provide reliable communications for application layer data, the RF modulator includes a direct error correction that includes interleavers and channel coding such as convolutional or turbo coding.

Video compression reduces redundancy at the source of the video and increases the amount of information carried in each bit of encoded video data. This increases the impact on quality when even a small part of the encoded video is lost. The spatial and temporal prediction inherent in video compression systems aggravates the loss and produces errors that propagate resulting in visible artifice in the reconstructed video. The error resistance algorithms in the video encoder and the error recovery algorithms in the video decoder improve the robustness against errors of the video compression system.

Typically, the video compression system is agnostic with respect to the underlying network. However, in error-prone networks, the integration or alignment of error protection algorithms in the application layer with the FEC and channel coding in the link / physical layers is highly desirable and provides the highest efficiency in the performance improvement against system errors.

FIGURE 14 illustrates an example of a rate-distortion data stream that can happen in encoder 228 to encode frames. The process 1400 starts at the start 1402 and continues in the decision block 1404 in which it receives the input of the scene change detector 1410 from the preprocessor 226 (for example by means of metadata) and the resistance input is acquired against 1406 errors. If the information indicates a frame selected as a frame I, the process intra-encodes the frame. If the information indicates that the selected frame is a P or B frame, the process uses intra-coding and (Inter) motion estimation coding to encode the frame.

After an affirmative condition for the conditions of block 1404 occurs the process 1400 continues in a preparation block 1414 in which the rate R is set at the value R = Rcal, the desired target quality based on

60 E06804241 04-11-2011

R-D curves. The adjustment is received from data block 1416 comprising the R-D curves. The process 1400 then proceeds to block 1418 where the assignment of rate control bits {QPi} is performed based on the activity and image / video information (eg content classification) from a content classification process in block 1420.

The assignment block of the rate control bit 1418 is used, in turn, for the estimation of the movement in the block 1422. The estimation of the movement 1422 can also receive the input of metadata from the preprocessor 1412, the smoothing detector movement (MPEG-2 + Historical) from block 1424 and multiple reference frames (macro causal + non-causal MB blocks) from block 1426. Process 1400 then proceeds to block 1428 where the rate calculations for intra modes are determined -coded by the rate control bit allocation (QPi). The process 1400 then proceeds to block 1430 where the mode and quantification parameters are determined. The mode decision of block 1430 is made based on the motion estimation input of block 1422, the error resistance input 1406 and the scalability RD, which is determined in block 1432. Once the mode is decided , the flow proceeds to block 1432. It is indicated that the flow from block 1430 to 1432 occurs when the data is passed from the parts of the first step to the second step of the encoder.

In block 1432, the transformation and quantification is performed by the second step of the encoder 228. The transform / quantification process is adjusted or tuned as indicated with block 1444. This transformation / quantification process may be influenced by the module. of final adjustment of the rate control (FIGURE 7). Process 1400 then proceeds in block 1434 for a zigzag classification and entropy coding to produce the encoded base layer. Zigzag coding prepares quantified data in an efficient format for coding. Entropy coding is a compression technique that uses a series of bit codes to represent a set of possible symbols. The improved layer resulting from the transform / quantization block 1432 is also sent to an adder 1436, which subtracts the base layer and sends the result to a ZZ / entropy encoder 1438 for the improved layer, as previously described with reference to FIGURES. 31-36. It is also indicated that the improved layer is fed back (see line 1440 for updating the true rate) for updating the content classification in 1420 for the true rate and an operation for determining the long and short term histories of the Bit rates for use in rate control.

FIGURE 17 is a high level block diagram of a multimedia coding system. The multimedia coding system includes means for receiving multimedia data as illustrated by the module for receiving multimedia data 1705. Such means may comprise, for example, a transcoder, an encoder, a preprocessor, a processor configured to receive data. Multimedia, a receiver. More specifically, the receiving means may comprise components and modules described herein to receive multimedia data, including, in several examples, transcoder 200. The encoding system also comprises means for encoding multimedia data as illustrated by the module for encoding multimedia data 1710. Such encoding means 1710 may comprise transcoder 200, encoder 228 or preprocessor 226.

FIGURES 23, 24, 27 and 28 are process flow diagrams that exemplify multimedia data coding procedures that perform the aspects described herein. FIGURE 23 is a process flow diagram illustrating a process 2300 for encoding multimedia data based on content information. Block 2305 of process 2300 receives the encoded multimedia data and in block 2310 process 2300 decodes the multimedia data. In block 2315, process 2300 determines the information of the content associated with the decoded multimedia data. In block 2320, process 2300 encodes multimedia data based on content information.

FIGURE 24 is a process flow diagram illustrating a multimedia data encoding process 2400 so that it aligns the data boundaries based on the level of content information. In block 2405, process 2400 obtains information about the content associated with multimedia data, which can be done, for example, by preprocessor 216 or content classification module 712 shown in FIGURE 7. In block 2410, The 2400 process encodes multimedia data so that it aligns the limits of the data based on the content information. For example, portion limits and access unit limits align with frame limits based on the content classification of the multimedia data being encoded. The encoded data is then available for further processing and / or transmission to a mobile device and the 2400 process ends.

FIGURE 27 is a process flow diagram illustrating a process 2700 for data coding using an adaptive intra refresh scheme based on the content information. When the 2700 process begins the multimedia data has been obtained. In block 2705, process 2700 obtains the information of the content of the multimedia data. The content information can be obtained by, for example, the preprocessor 226 or the content classification module 712 as described above. The process 2700 continues in block 2710, in which it encodes the multimedia data using a resistance scheme against adaptive intra refresh errors, in which the resistance scheme against adaptive intra refresh errors is based on the content information . The functionality of block 2700 can be performed by encoder 228. The encoded data is made available for further processing and transmission and the 2700 process

60 E06804241 04-11-2011

ends next.

FIGURE 28 is a process flow diagram illustrating a process of encoding multimedia data using redundant tables I based on the information of the multimedia content. When the 2800 process begins the multimedia data is available for processing. In block 2805, process 2800 obtains the information of the multimedia data content. As described above, this can be done by, for example, preprocessor 226 and / or encoder 228. In block 2810, process 2800 encodes multimedia data so that it inserts one or more additional I frames of the data encoded based on content information. This can be done by the encoder 228 as described above in connection with the error resistance scheme, by inserting the frames I in the base layer or in the improvement layer depending on the error resistance scheme employed. After block 2810, the encoded data is available for further processing and / or transmission to a mobile device.

It should be noted that the procedures described herein can be implemented in a variety of communication hardware, processors and systems known to those skilled in the art. For example, the general requirement for the client to operate as described herein is that the client has a screen to display the information content, a processor to control the operation of the client and a memory to store data and programs. related to customer operation. In one aspect, the customer is a cell phone. In another aspect, the customer is a laptop that has communication capabilities. The other aspect, the customer is a personal computer that has communication capabilities. In addition, hardware such as a GPS receiver can be incorporated into the client to implement the various aspects. The various logical blocks, logic, modules and illustrative circuits described in connection with the aspects disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor (DSP), a specific application integrated circuit ( ASIC), a field programmable door array (FPGA) or other programmable logic devices, discrete doors or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described in this document. A general purpose processor can be a microprocessor but, in the alternative, the processor can be any processor, controller, microcontroller or conventional state machine. A processor can also be implemented as the combination of calculation devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one more microprocessors in conjunction with a DSP core or any such combination.

The various logic, logic blocks, modules and illustrative circuits described in connection with the aspects disclosed in this document can be implemented or performed with a general purpose processor, digital signal processor (DSP), a specific application integrated circuit (ASIC ), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor but, in the alternative, the processor can be any processor, controller, microcontroller or conventional state machine. A processor can also be implemented as the combination of calculation devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one more microprocessors in conjunction with a DSP core or any other such combinations.

The disclosed methods and apparatus provide the transcoding of video data encoded in one format to video data encoded in another format where the encoding is based on the content of the video data and the encoding is resistant to errors. The procedures or algorithms described in connection with the examples disclosed herein can be performed directly on hardware, in a software module executed by a processor, firmware or in a combination of two or more of these. A software module may reside in a RAM, flash memory, ROM memory, EPROM memory, EEPROM memory, recorders, a hard disk, a removable disk, a CD-ROM or any other form of storage media known in the art. . An example storage medium is connected to the processor, so that the processor can read the information from, and write the information to, the storage medium. In the alternative, storage media can be an integral part of the processor. The processor and storage medium may reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

The examples described above are merely exemplary and those skilled in the art can now make numerous uses of, and separations of, the examples described above without separating from the inventive concepts disclosed herein. Various modifications to these examples may be readily apparent to those skilled in the art and the generic principles defined herein may be applied to other examples, for example in an instant messaging service or in any general wireless data communication application.

Therefore, the scope of the disclosure is not intended to be limited to the examples shown in this document but to be in accordance with the broader scope consistent with the novel principles and characteristics disclosed in this document. The term "example" is used exclusively herein to indicate "serving as an example, case or illustration". Any example described herein

E06804241 04-11-2011

The document as “example” is not necessarily to be interpreted as preferred or advantageous compared to other examples. Accordingly, the novel aspects described herein are to be defined only by the scope of the following claims.

45 E06804241 04-11-2011

Claims (28)

1. A multimedia data processing procedure comprising: the reception of multimedia data and the coding of the multimedia data (361) in a first coded data layer and a second coded data layer based on the content of the multimedia data, the first coded data layer being configured to be encodable from independent mode of the second coded data layer and in which the first data layer is coded with a lower quality than the second data layer, characterized in that the method comprises: the coding of the first data layer by the steps of: transformation of the residual error coefficients of the first data layer, the residual error being the difference between the multimedia data that is being encoded (361) and the best fit macro block of a temporary reference frame; the calculation of a coefficient of the first data layer, C'base, from the transformed residual error coefficients, Cbase and the quantification of the coefficient of the first data layer, C'base, by a first quantification parameter , Qb; and the coding of the second data layer through the steps of: the transformation of the residual error coefficients in the second data layer, the residual error being the difference between the multimedia data being encoded (361) and the macro block best fit of a temporary reference chart; the calculation of a coefficient of the second data layer, C'mej, from the transformed residual error coefficients, Cmej and the quantification of the coefficient of the second data layer, C'mej, by a second quantification parameter , in which: 10, if Base and Cmej have opposite signs' = C base 1min (Cbase, Cmej), otherwise or −1 1Cbase, if QQ (C) - C <C b b base mej mej C 'base = 10, otherwise Y C’mej = Cmej - Qb -1 (Qb (C’base)) in which Qb-1 is the decuantification parameter of the first data layer and in which the first quantification parameter is set to be higher than the second quantization parameter, so that the first coded data layer can be decode to form multimedia data that can be displayed if the second coded data layer is not available and the first coded data layer and the second coded data layer can be decoded in combination to form multimedia data that can be displayed if both The first coded data layer as the second coded data layer are available.

2.

 The method of claim 1, wherein the first data layer comprises I, intra-encoded, and P-tables, predictive and in which the second data layer comprises I, intra-encoded, P, predictive and B frames, bidirectional.

3.

 The method of claim 1, wherein the first data layer comprises a base layer and the second data layer comprises an improvement layer.

Four.

 The method of claim 1, wherein the encoding comprises assigning a bit rate for at least a portion of the multimedia data based on the content classification.

5.

 The method of claim 1, wherein the coding further comprises:

the use of content classification to detect scene changes and the determination of whether to include intra-encoded I frames in the first data layer and in the second data layer based on the scene changes detected.

6.

 The method of claim 1, wherein the coding further comprises determining a frame rate for the coding of the multimedia data based on the content classification.

50 E06804241 04-11-2011

7.

 The method of claim 1, wherein the coding comprises performing the motion estimation of the multimedia data based on the content classification.

8.

 The method of claim 1, further comprising determining a first frame rate for coding the first data layer and determining a second frame rate for coding the second data layer, wherein the First frame rate is lower than the second frame rate.

9.

 The method of claim 1, wherein the coding comprises performing error resistance processing in the multimedia data based on the content classification.

10.

 The method of claim 1, wherein the first quantization parameter comprises a first step size for data coding and wherein the second quantization parameter comprises a second step size for data coding, in the that the first jump size is larger than the second step size.

eleven.

 The method of claim 1, wherein the coding comprises the reduction of noise in the multimedia data based on the content classification.

12.

 The method of claim 11, wherein the noise reduction comprises performing an elimination of artifice.

13.

 The method of claim 11, wherein the noise reduction comprises processing at least part of the multimedia data with a scrap filter, wherein the intensity of the scrap filter is based on the content of the multimedia data. .

14.

 The method of claim 11, wherein the noise reduction comprises processing at least part of the multimedia data with a block suppression filter, wherein the intensity of the block suppression filter is based on the content of multimedia data.

fifteen.

 The method of claim 11, wherein the noise reduction comprises the filtering of selected frequencies of the multimedia data.

16.

 The method of claim 13, wherein the intensity of the stripping filter is based on the classification of the content of the multimedia data.

17.

 The method of claim 14, wherein the intensity of the block suppression filter is based on the classification of the content of the multimedia data.

18.

 The method of claim 1, wherein the coding comprises the reduction of multimedia data sampling.

19.

 The method of claim 1, wherein the coding comprises the association of a quality level with the multimedia data and the use of the quality level and the information of the content of the multimedia data to determine a bit rate for the coding of Multimedia data

twenty.

 An apparatus for processing multimedia data comprising an encoder (228) configured to receive multimedia data and encode the multimedia data in a first layer of encoded data and a second layer of encoded data based on the content of the multimedia data, the first configured coded data layer to be decoded independently of the second data layer and in which the first data layer is coded with a lower quality than the second data layer; the apparatus characterized in that the encoder (328) is configured to encode the first data layer by:

the transformation of the residual error coefficients of the first data layer, the residual error being the difference between the multimedia data being encoded (361) and the best fit macro block of a temporary reference frame; the calculation of a coefficient of the first data layer, C'base, for the transformed residual error coefficients, Cbase and the quantification of the coefficient of the first data layer, C'base, by means of a first quantification parameter, Qb ; and in which the encoder is configured to encode the second data layer by: the transformation of the residual error coefficients of the second data layer, the residual error being the difference between the multimedia data being encoded (361) and the best fit macro block of a temporary reference frame; the calculation of a coefficient of the second data layer, C'mej, from the transformed residual error coefficients, Cmej and the quantification of the coefficient of the second data layer, C'mej, by a second quantification parameter , in which: E06804241 04-11-2011 10, if Base and Cmej have opposite signs C 'base = 1min (Cbase, Cmej), otherwise or −1 1Cbase, if QQ (C) - C <C b b base mej mej C 'base = 10, otherwise Y C’mej = Cmej - Qb -1 (Qb (C’base)) in which Qb-1 is the quantification parameter of the first data layer and in which the encoded is further configured to determine the first quantification parameter of the second quantification parameter, the first quantification parameter being higher than the second quantification parameter, so that the first coded data layer can be decoded to form multimedia data that can be displayed if the second coded data layer is not available and the first coded data layer and the second coded data layer are they can decode in combination to form multimedia data that can be displayed if both the first coded data layer and the second coded data layer are available.

twenty-one.

 The apparatus of claim 20, wherein the encoder (228) comprises a motion estimation module (720) configured to perform the estimation of the movement of the multimedia data based on the classification of the content and generate compensation information of the data movement and in which the coding module (228) is further configured to use the motion compensation information to encode the multimedia data.

22

 The apparatus of claim 20, wherein the encoder (228) further comprises a quantization module

(726) for the determination of the first and second quantification parameters.

2. 3.

 The apparatus of claim 20, wherein the encoder (228) further comprises a bit allocation module (714) configured to provide a bit rate for at least a portion of the multimedia data based on the content classification.

24.

 The apparatus of claim 20, wherein the encoder (228) further comprises a scene change detection module configured to detect scene changes and wherein the coding module (228) is further configured to include frames I, intra-encoded in the multimedia data encoded based on the scene changes detected.

25.

 The apparatus of claim 20, wherein the encoder (228) further comprises a frame rate module configured to determine the frame rate of the multimedia data based on the content classification and in which the coding module ( 228) encodes multimedia data based on frame rate.

26.

 The apparatus of claim 20, wherein the encoder (228) is further configured to perform error processing in the multimedia data based on the content classification.

27.

 A means that can read a machine comprising instructions that after its execution cause a machine to perform the procedure of claim 1.

28.

 A processor comprising a configuration for performing the method of claim 1.

E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04 -11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11-2011 E06804241 04-11 -2011 E06804241 04-11-2011