RU2673704C1 - Method, device and system for encoding and decoding conversion units for encoding units - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: invention relates to the field of video processing. Method for converting a plurality of arrays of residual coefficients from a video bitstream configured for a 4:2:2 format into a plurality of arrays of residual bins contains the steps of: decoding four arrays of residual luminance coefficients for luminance channel, each of these four arrays of residual luminance corresponding to one 4x4 brightness block out of four 4x4 brightness blocks, and these four 4x4 brightness blocks together occupy a 8x8 brightness region; decoding two arrays of residual chromaticity coefficients for the first chrominance channel; decoding two arrays of residual chromaticity coefficients for the second chroma channel; and applying a transform to each of the four decoded arrays of residual luminance coefficients for the luminance channel, decoded two arrays of residual chromaticity coefficients for the first chrominance channel and decoded two arrays of residual chromaticity coefficients for the second chroma channel to create residual discretization luminance elements for the luminance channel, residual chrominance bin of the first chrominance channel and residual chrominance bin of the second chrominance channel.

EFFECT: improved efficiency of encoding/decoding arrays of residual coefficients of the bit stream.

6 cl, 28 dwg, 4 tbl

Description

Cross reference to related application (s)

[0001] This application claims priority under Section 119 of Section 35 of the US Code for the filing date of Australian Patent Application No. 2012247040, filed November 8, 2012, which is hereby incorporated by reference in its entirety. This application is a highlighted application of Australian Patent Application No. 2012232992, filed September 28, 2012, which is hereby incorporated by reference in its entirety.

FIELD OF THE INVENTION

[0002] The present invention relates generally to digital processing of a video signal, and in particular, to a method, apparatus, and system for encoding and decoding residual coefficients of a transform unit (TU), wherein the transform unit (TU) includes one or more conversion unit (TU) and can be configured for several color formats, including a 4: 2: 2 color format, and the residual conversion unit (TU) coefficients can be data in either the frequency domain or spatial th field.

BACKGROUND OF THE INVENTION

[0003] Currently, there are many video encoding applications, including applications for transmitting and storing video data. Many video coding standards have also been developed, while others are under development. Recent developments in the standardization of video coding have led to the formation of a group called the “Joint Video Coding Team” (JCT-VC). The Joint Video Coding Team (JCT-VC) includes members of Study Group 16, Issue 6 (SG16 / Q6) of the Telecommunication Standardization Sector (ITU-T) of the International Telecommunication Union (ITU), also known as the Video Coding Experts Group (VCEG) ) and members of the International Organization for Standardization / Joint Technical Committee 1 of the International Electrotechnical Commission / Subcommittee 29 / Working Group 11 (ISO / IEC JCT1 / SC29 / WG11), also known as the Expert Group on Cinematography (MPEG).

[0004] The Combined Video Encoding Team (JCT-VC) aims to create a new video encoding standard to provide significant superiority over the current video encoding standard known as “H.264 / MPEG-4 AVC”. The H.264 / MPEG-4 AVC standard in itself is a big improvement over previous video coding standards such as MPEG-4 and ITU-T H.263. The new coding standard, which is under development, has been called "High Performance Video Coding (HEVC)." The Joint Video Coding Team (JCT-VC) also takes into account implementation problems associated with the proposed technology for high-performance video coding (HEVC), which creates difficulties with scalable implementations of the standard for operation at high resolutions in real time or at high frame rates. One implementation problem is the complexity and size of the logic used to provide multiple ‘transform’ sizes for converting video data between the frequency domain and the spatial domain.

SUMMARY OF THE INVENTION

[0005] An object of the present invention is essentially to overcome or at least improve one or more of the disadvantages of existing arrangements.

[0006] In accordance with one aspect of the present invention, there is provided a method for decoding luma transforms and a plurality of color transforms from a video bitstream, the plurality of color transforms comprising color data for one color channel, the method comprising the steps of:

determining a luminance conversion skip flag for luminance conversion, the luminance conversion skipping flag indicating whether the luminance conversion data is encoded in the video bitstream as a representation in the spatial domain;

determining a color conversion skip flag value for a first color conversion from a plurality of color transforms, the color conversion skipping flag indicating whether the color conversion data is encoded in the video bitstream as a representation in the spatial domain; and

decode the luminance conversion in accordance with a specific value of the luminance conversion skip flag and a plurality of color conversions in accordance with the determined value of the color mismatch flag for the first color conversion.

[0007] In accordance with another aspect, a method for decoding a transform unit with a luminance conversion and two color transforms from a video bit stream is provided, wherein two color transforms comprising color data for one color channel in accordance with a 4: 2: 2 color format, wherein The method comprises the steps of:

determining a luminance conversion skip flag for luminance conversion, the luminance conversion skipping flag indicating whether the luminance conversion data is encoded in the video bitstream as a representation in the spatial domain;

 determining a color conversion skip flag for a first color conversion from two color transforms, the color conversion skipping flag indicating whether the color conversion data is encoded in the video bitstream as a representation in the spatial domain; and

decode the luminance conversion in accordance with a specific value of the luminance conversion skip flag, and decode two color transforms in accordance with the determined value of the mismatch color conversion flag for the first color conversion.

[0008] In accordance with yet another aspect, a method for decoding luma transforms and a plurality of color transforms from a video bitstream is provided, the plurality of color transforms containing data for one color channel, the method comprising the steps of:

breaking at least one rectangular one of the transformations into a plurality of square transformations; and

decode square transforms.

[0009] It is desirable that the step at which to split is comprised of a step at which all rectangular transformations are partitioned into square transforms, so that the step at which decode is carried out only on the square transforms.

[0010] In accordance with another aspect, a method is provided for decoding a conversion unit containing residual chroma coefficients from a video bitstream, wherein a conversion unit comprising at least one array of residual chroma coefficients associated with one chroma channel, the method comprising the steps of , where:

determining the size of the conversion unit, the size related to the hierarchical level of the conversion unit in the corresponding coding unit;

at least one array of residual chroma coefficients is decoded from the video bitstream using a predetermined maximum number of transformations for the chroma channel of the transform unit;

selecting an inverse transform for the decoded arrays of residual chroma, the inverse transform being selected from a predefined set of inverse transforms; and

apply the selected inverse transform to each of the arrays of residual chroma coefficients to decode the residual chroma sampling elements for the chroma channel of the transform unit.

[0011] In yet another aspect, a method is disclosed for decoding residual data for a region in a transform unit (TU) in a color channel encoded in a video bitstream, the method comprising the steps of:

first of all, it is determined from the bitstream that the conversion skip flag is enabled;

secondly, it is determined whether the region is the first region in the color channel and in the transform unit (TU) with the flag value of the encoded block (CBF) equal to one, and if so, the value of the transform skip flag is decoded and stored, otherwise, the value of the skip flag is retrieved transformations; and

decode the residual region data using the transform skip flag value.

[0012] Here, preferably, the first determination step further comprises determining that the coding unit transform quantization bypass flag is not enabled and the transform size is 4x4.

[0013] In accordance with another aspect of the present invention, there is provided a method for inverting a plurality of arrays of residual coefficients from a video bitstream configured for a 4: 2: 2 color format, the method comprising the steps of:

a 4 × 4 brightness unit from a plurality of 4 × 4 brightness units, each 4 × 4 brightness unit being co-located with one 4 × 4 conversion unit from a plurality of 4 × 4 conversion units, wherein a plurality of 4 × 4 brightness units together occupy an 8 × brightness area 8;

decode, after the arrays of residual luminance coefficients are decoded, a plurality of arrays of residual chromaticity coefficients for the first color channel, wherein each array of residual chroma coefficients corresponds to a 4 × 4 color block and each 4 × 4 chroma block for the first color channel is co-located with two of sets of 4 × 4 transformation units;

decode, after the arrays of residual chroma coefficients for the first color channel are decoded, a plurality of arrays of residual chroma coefficients for the second color channel, wherein each array of residual chroma coefficients corresponds to a 4x4 color block and each chroma block for the second color channel is co-located with two from a plurality of 4x4 transformation units; and

apply an inverse transform to each of the decoded plurality of arrays of residual luma coefficients, the decoded plurality of arrays of residual luma coefficients for the first color channel, and the decoded plurality of arrays of residual luma coefficients for the second color channel.

[0014] Preferably, the number of arrays of residual brightness coefficients in the plurality of arrays of residual brightness coefficients is four. Preferably, the number of arrays of residual color coefficients in the set of arrays of residual color coefficients is equal to two. Advantageously, one array of residual coefficients includes all coefficients necessary for the inverse transform of one 4x4 block.

[0015] In accordance with another aspect, a method is disclosed for directly converting multiple arrays of residual coefficients into a video bitstream configured for a 4: 2: 2 color format, the method comprising the steps of:

apply a direct conversion to each of: a plurality of arrays of residual luma coefficients, a plurality of arrays of residual chroma for the first color channel, and a plurality of arrays of residual chroma for the second color channel;

a 4 × 4 brightness unit from a plurality of 4 × 4 brightness units, each 4 × 4 brightness unit being co-located with one 4 × 4 conversion unit from a plurality of 4 × 4 conversion units, wherein a plurality of 4 × 4 brightness units together occupy an 8 × brightness area 8;

encode, after the arrays of residual luminance coefficients have been encoded, a plurality of arrays of residual chromaticity coefficients for the first color channel, wherein each array of residual chroma coefficients corresponds to a 4x4 color block and each 4x4 color block for the first color channel is co-located with two of sets of 4 × 4 transformation units; and

encode, after the arrays of residual chroma coefficients for the first color channel are encoded, a plurality of arrays of residual chroma coefficients for the second color channel, wherein each array of residual chroma coefficients corresponds to a 4x4 color block and each chroma block for the second color channel is co-located with two from the set of 4 × 4 transformation units.

[0016] Other aspects, including additional encoders, are also disclosed.

Brief Description of the Drawings

[0017] At least one embodiment of the present invention will now be described with reference to the following drawings, in which:

[0018] FIG. 1 is a schematic structural diagram showing a video encoding and decoding system;

[0019] FIG. 2A and 2B form a schematic structural diagram of a general-purpose computer system on which one or both of the video encoding and decoding systems of FIG. one.

[0020] FIG. 3 is a schematic structural diagram showing functional modules of a video encoder;

[0021] FIG. 4 is a schematic structural diagram showing functional modules of a video decoder;

[0022] FIG. 5A and 5B schematically illustrate color formats for representing frame data;

[0023] FIG. 6A is a schematic representation of an example encoding unit transform tree;

[0024] FIG. 6B is a schematic representation of an exemplary transform tree of grid-arranged brightness discretization elements;

[0025] FIG. 6C is a schematic representation of an exemplary transform tree of a grid-arranged color sampling element;

[0026] FIG. 7 is a schematic illustration of a data structure representing a luminance channel of an exemplary transform tree;

[0027] FIG. 8 illustrates a data structure representing a color channel of an exemplary transform tree;

[0028] FIG. 9A and 9B schematically show a bitstream structure that encodes an exemplary transform tree;

[0029] FIG. 9C, 9D, and 9E schematically show an alternative bitstream structure that encodes an example transform tree;

[0030] FIG. 10 is a schematic flowchart showing a method for encoding an example transform tree;

[0031] FIG. 11 is a schematic flowchart showing a method for decoding an example transform tree;

[0032] FIG. 12A-12C schematically show residual 4x8 scan unit conversion units;

[0033] FIG. 13 is a schematic flowchart showing a method for encoding an example conversion unit;

[0034] FIG. 14 is a schematic flowchart showing a method for decoding an example conversion unit;

[0035] FIG. 15 schematically shows possible arrangements of 4 × 4 transformations for 4 × 4 and 8 × 8 transform units (TU);

[0036] FIG. 16 schematically illustrates exemplary color regions for implementation;

[0037] FIG. 17 is a schematic flowchart showing a method for decoding residual data of an exemplary conversion unit; and

[0038] FIG. 18 schematically illustrates a conversion skip operation applied to a 4 × 8 color region with a (non-square) 4 × 8 transform.

Detailed Description Including Preferred Embodiment

[0039] Where reference is made to steps and / or features that have the same numeric designations in one or more of the accompanying drawings, these steps and / or features, for the purposes of this description, have the same function (s) or operation (s) until the opposite intention appears.

[0040] FIG. 1 is a schematic block diagram showing the functional modules of a video encoding and decoding system 100 that can use techniques to encode syntax elements, which are the implicit division of conversion units into multiple transformations for a color channel. System 100 includes a source device 110 and a receiver device 130. A communication channel 120 is used to transmit encoded video information from the source device 110 to the receiver device 130. In some cases, the source device 110 and the receiver device 130 may be configured in the form of corresponding mobile handsets, in which case, communication channel 120 is a wireless channel. In other cases, the source device 110 and the recipient device 130 may be made in the form of equipment for video conferencing, and in this case, the communication channel 120, as a rule, is a wired channel, such as an Internet connection. Moreover, the source device 110 and the receiver device 130 can be made in the form of any of a wide range of devices, including devices that support broadcast television, cable television applications, Internet video applications, and including applications where the encoded video is captured on a storage medium or file server.

[0041] As illustrated, the source device 110 includes a video source 112, a video encoder 114, and a transmitter 116. The video source 112 is typically configured as a source: captured video frame data, such as an image forming sensor; a previously captured video sequence stored on a non-temporary recording medium; or video from a remote imaging sensor. Examples of source devices 110, which may include an image forming sensor as a video source 112, include smart phones, video cameras, and network video cameras. A video encoder 114 converts the captured frame data from a video source 112 into encoded video data and will be described later with reference to FIG. 3. The encoded video data is typically transmitted by the transmitter 116 over the communication channel 120 as encoded video information. It is also possible that the encoded video data is stored on some storage device, such as a “flash” memory or a hard disk drive, until the latter is transmitted over the communication channel 120.

[0042] The recipient device 130 includes a receiver 132, a video decoder 134, and a display device 136. The receiver 132 receives the encoded video information from the communication channel 120 and forwards the received video data to the video decoder 134. The video decoder 134 then outputs the decoded frame data to the display device 136. Examples of the display device 136 include a cathode ray tube, a liquid crystal display, such as in smart phones, tablet computers, computer monitors, or stationary televisions. It is also possible that the functionality of each of the source device 110 and the recipient device 130 are embodied in one device.

[0043] Despite the above-mentioned exemplary devices, each of the devices: the source device 110 and the receiver device 130, can be implemented within a general-purpose computing system, typically by a combination of hardware and software components. FIG. 2A illustrates such a computer system 200 that includes: a computer module 201; input devices such as a keyboard 202, a pointing device 203, a mouse, a scanner 226, a camera 227 that can be configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214 that can be configured as a display device 136, and loudspeakers 217. An external modulator-Demodulator (Modem) transceiver device 216 can be used by a computer module 201 to transmit to and receive from a communication network 220 via a connection 221. A communication network 220, which may be a communication channel 120, may be a wide area network (WAN), such as the Internet, a cellular network, or a private WAN. In the case where the connection 221 is a telephone line, the modem 216 may be a traditional “switched” modem. Alternatively, in the case where connection 221 is a high capacity connection (eg, cable), modem 216 may be a broadband modem. A wireless modem may also be used to wirelessly connect to a communications network 220. The transceiver device 216 may provide the functionality of the transmitter 116 and the receiver 132, and the communication channel 120 may be embodied in the connection 221.

[0044] Computer module 201 typically includes at least one processor node 205 and a memory node 206. For example, the memory unit 206 may have a random access memory (RAM) and a semiconductor read only memory (ROM). Computer module 201 also includes a number of input / output (I / O) interfaces, including: an audio-video interface 207, which interfaces with a video display 214, a speaker 217, and a microphone 280; an I / O interface 213 that interfaces with a keyboard 202, a mouse 203, a scanner 226, a camera 227, and an optional joystick or other human interface device (not illustrated); and an interface 208 for an external modem 216 and a printer 215. In some implementations, the modem 216 may be integrated into a computer module 201, for example, within an interface 208. The computer module 201 also has a local area network interface 211 that allows the computer system 200 to be coupled through connection 223 with a local area network 222 connection known as a Local Area Network (LAN). As illustrated in FIG. 2A, a local area network 222 can also interface with a wide area network 220 via a connection 224, which will typically include a so-called “firewall” device or device with similar functionality. The LAN interface 211 may include an Ethernet ™ circuit board, a Bluetooth ™ wireless configuration, or an IEEE 802.11 wireless configuration; however, numerous other types of interfaces can be practiced with respect to the interface 211. The interface 211 of the local area network 211 can also provide the functionality of the transmitter 116 and the receiver 132, and the communication channel 120 can also be implemented in the local communication network 222.

[0045] The I / O interfaces 208 and 213 can provide either one or both of serial and parallel connectivity, the former typically being implemented in accordance with Universal Serial Bus (USB) standards and using appropriate USB connectors (not illustrated). Storage devices 209 are provided, and typically include a hard disk drive (HDD) 210. Other storage devices, such as a floppy disk drive and a magnetic tape drive (not illustrated), may also be used. An optical disk drive 212 is typically provided for operation as a non-volatile data source. Portable memory devices such as optical discs (e.g. CD-ROM, DVD, Blu-ray Disc ™), USB-RAM, portable, external hard drives, and floppy disks, for example, can be used as appropriate data sources for a computer system 200. As a rule, any of the components: HDD 210, optical drive 212, networks 220 and 222, can also be configured to function as a video source 112, or as a receiver in relation to decoded video data, for storage with reproduction Res display 214.

[0046] Components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner that leads to the normal operation of the computer system 200 known to those skilled in the art. For example, processor 205 interfaces with system bus 204 via connection 218. Similarly, memory 206 and optical drive 212 are coupled to system bus 204 via connections 219. Examples of computers on which the described arrangements may be practiced include IBM-PC and compatible, Sun SPARCstations, Apple Mac ™ or similar computer systems.

[0047] If necessary or desired, the video encoder 114 and the video decoder 134, as well as the methods described below, can be implemented using the computer system 200, the video encoder 114, the video decoder 134 and the processes of FIG. 10-13, which will be described, can be implemented as one or more software applications 233 executable on a computer system 200. In particular, a video encoder 114, a video decoder 134, and steps of the described methods are performed by instructions 231 (see FIG. . 2B) in software 233 that runs on computer system 200. Software instructions 231 may be formed as one or more code modules, each for one or more specific tasks. The software can also be divided into two separate parts, of which the first part and the corresponding code modules perform the described methods, and the second part and the corresponding code modules control the user interface between the first part and the user.

The software may be stored on a computer-readable storage medium including storage devices described below, for example. The software is loaded into a computer system 200 readable by a computer from a computer readable storage medium, and then executed by a computer system 200. A computer readable storage medium with such software or a computer program recorded on a computer readable storage medium is a computer program product. The use of a computer program product in a computer system 200 preferably embodies an advantageous device for implementing a video encoder 114, a video decoder 134, and the described methods.

[0049] The software 233 is typically stored on the HDD 210 or in the memory 206. The software is downloaded to a computer system 200 readable by a computer from a computer readable medium, and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium 225 information (for example, CD-ROM), which is read by the drive 212 on the optical disk.

[0050] In some cases, application programs 233 can be delivered to a user encoded on one or more CD-ROMs 225 and read through an appropriate drive 212, or, alternatively, can be read by a user from networks 220 or 222. In yet another case, , the software can also be downloaded to the computer system 200 from other computer readable storage media. Computer-readable storage media refers to any non-temporary physical storage media that provide recorded instructions and / or data to computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-ray Disc, hard disk drive, ROM or integrated circuit, USB memory, magneto-optical disk, or a computer-readable card such as a PCMCIA card and the like, whether or not such devices are internal or external to computer module 201. Examples of temporary or non-material computer-readable transmission media that may also be involved in providing software applications frames, instructions and / or video data or encoded video data to computer module 401, include radio or infrared transmission channels, as well as a network connection to another computer or network device, the Internet or Intranets, including email and information transfers, recorded on web sites and the like.

[0051] The second part of the application programs 233 and corresponding code modules mentioned above can be executed to implement one or more graphical user interfaces (GUIs) for visualization or other presentation on the display 214. By manipulating, as a rule, the keyboard 202 and the mouse 203 , the user of the computer system 200 and the application can manipulate the interface in a functionally flexible manner to provide control and / or input commands to GUI-related applications. Other forms of functionally flexible user interfaces may also be implemented, such as an audio interface using the output of voice prompts through the speakers 217 and inputting voice commands from the user through the microphone 280.

[0052] FIG. 2B is a detailed schematic structural diagram of a processor 205 and “memory” 234. Memory 234 is a logical aggregation of all memory modules (including HDD 209 and semiconductor memory 206) that can be accessed by computer module 201 of FIG. 2A.

[0053] When power is initially supplied to the computer module 201, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. A hardware device, such as a software storage ROM 249, is sometimes referred to as firmware. The POST program 250 examines the hardware inside the computer module 201 to ensure proper functioning, and typically checks the processor 205, memory 234 (209, 206), and basic input / output system (BIOS) software module 251, also typically stored in ROM 249 for proper operation. Once the POST program 250 is completed successfully, the BIOS 251 activates the hard disk drive 210 of FIG. 2A. Activating the hard disk drive 210 calls the bootloader program 252, which is resident on the hard disk drive 210, for execution through the processor 205. This loads the operating system 253 into the RAM memory 206, after which the operating system 253 starts to function. Operating system 253 is a system-level application executed by processor 205 to perform various high-level functions, including processor control, memory management, device management, storage management, a software application interface, and a common user interface.

[0054] The operating system 253 controls the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has enough memory to execute without conflict with the memory allocated to another process. In addition, the various types of memory available in the computer system 200 of FIG. 2A must be used correctly so that each process can be performed efficiently. Accordingly, aggregated memory 234 is not intended to illustrate how particular segments of memory are allocated (unless otherwise stated), but rather to provide a general view of the memory available to computer system 200 and how it is used.

[0055] As shown in FIG. 2B, the processor 205 includes a number of functional modules, including a control unit 239, an arithmetic logic unit 240 (ALU), and local or internal memory 248, sometimes referred to as cache memory. The cache memory 248 typically includes a number of storage registers 244-246 in the register section. One or more internal buses 241 functionally interconnect these function modules. The processor 205 typically also has one or more interfaces 242 for communicating with external devices via a system bus 204 using a connection 218. A memory 234 is coupled to a bus 204 using a connection 219.

[0056] The application program 233 includes a sequence of instructions 231, which may include conditional branches and loop instructions. Program 233 may also include data 232 that is used to execute program 233. Instructions 231 and data 232 are stored in memory cells 228, 229, 230, and 235, 236, 237, respectively. Depending on the relative size of the instructions 231 and the memory cells 228-230, a particular instruction may be stored in one memory cell, as depicted by the instruction shown in the memory cell 230. Alternatively, instructions may be segmented into a number of parts, each of which is stored in a separate memory location, as depicted by instruction segments shown in memory locations 228 and 229.

[0057] In general, the processor 205 is issued a set of instructions that are executed in it. The processor 205 awaits subsequent input, to which the processor 205 responds by executing another set of instructions. Each input can be provided from one or more of a number of sources, including data generated by one or more input devices 202, 203, data extracted from one of the storage devices 206, 209, or data extracted from a storage medium 225 of information inserted into an appropriate reader device 212, all depicted in FIG. 2A. The execution of a set of instructions may in some cases lead to data output. Execution may also cause the storage of data or variables in memory 234.

[0058] The video encoder 114, the video decoder 134, and the described methods can use the input variables 254, which are stored in the memory 235 in the corresponding memory cells 255, 256, 257. The video encoder 114, the video decoder 134 and the described methods create output variables 261, which are stored in the memory 234 in the corresponding memory cells 262, 263, 264. Intermediate variables 258 may be stored in memory cells 259, 260, 266, and 267.

[0059] Turning to the processor 205 of FIG. 2B, registers 244, 245, 246, the arithmetic logic unit 240 (ALU), and the control unit 239 work together to perform the sequences of microoperations required to complete the “fetch, decode, and execute” cycles for each instruction in the instruction set constituting the program 233. Each cycle of sampling, decoding, and execution contains:

(a) a sampling operation that selects and reads instruction 231 from memory cell 228, 229, 230;

(b) a decoding operation in which the control unit 239 determines which instruction has been selected; and

(c) an execution operation in which the control unit 239 and / or the ALU 240 executes the instruction.

[0060] After this, an additional sampling, decoding, and execution cycle for the next instruction may be executed. Similarly, a save cycle may be performed by which the control unit 239 stores or writes a value to the memory cell 232.

[0061] Each step or subprocess in the processes of FIG. 10-13, which will be described, is associated with one or more segments of the program 233 and, as a rule, is performed by the register section 244, 245, 247, ALU 240, and the control unit 239 in the processor 205, working together to perform sampling, decoding, and the execution for each instruction in the instruction set for the marked segments of program 233.

[0062] FIG. 3 is a schematic block diagram showing the functional modules of a video encoder 114. FIG. 4 is a schematic block diagram showing the functional modules of a video decoder 134. Video encoder 114 and video decoder 134 may be implemented using general purpose computer system 200, as shown in FIG. 2A and 2B, where various functional modules may be implemented by specialized hardware in a computer system 200, by software executing in a computer system 200, such as one or more software code modules of a software application 233 residing on a drive 205 on the hard drive and controlled by the processor 205, or alternatively, through a combination of specialized hardware and software executing in the computer system 200. The encoder 114, video decoder 134 and the described methods may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or subfunctions described methods. Such specialized hardware may include graphic processors, digital signal processors, problem oriented integrated circuits (ASICs), programmable gate arrays (FPGAs), or one or more microprocessors and associated storage devices. In particular, video encoder 114 comprises modules 320-344, and video decoder 134 comprises modules 420-434, each of which may be implemented as one or more software code modules of a software application 233.

[0063] Although the video encoder 114 of FIG. 3 is an example of a high-performance video coding (HEVC) video coding pipeline, the processing steps performed by modules 320-344 are common to other video codecs such as VC-1 or H.264 / MPEG-4 AVC. Video encoder 114 receives captured frame data, such as captured frame data, as a series of frames, with each frame including one or more color channels. Each frame contains one grid of bins per color channel. Color information is represented using ‘color space’, such as ITU-R BT.709 (‘YUV’), although other color spaces are also possible. When the YUV color space is used, color channels include a luminance channel (‘Y’) and two color channels (‘U’ and ‘V’). Moreover, different amounts of information can be included in the grid of discretization elements of each color channel, depending on the sampling of the image or by applying filtering to oversampling the captured frame data. There are several resampling approaches known as ‘color formats’, some of which will be described with reference to FIG. 5A and 5B.

[0064] The video encoder 114 divides each frame of the captured frame data, such as frame data 310, into areas commonly referred to as “coding tree blocks” (CTBs). Each block of a coding tree (CTB) includes a hierarchical subdivision of a quad tree in the form of a section of a frame in a collection of ‘coding units’ (CU). A coding tree block (CTB) typically occupies an area of 64 × 64 brightness sampling elements, although other sizes, such as 16 × 16 or 32 × 32, are possible. In some cases, even larger sizes, such as 128 × 128, can be used. A coding tree block (CTB) can be subdivided into four equally sized regions to create a new hierarchical level. Partitioning can be applied recursively, leading to the formation of a quad tree hierarchy. Since the lateral dimensions of a coding unit (CTB) block are always powers of two and splitting a quad tree always leads to halving the width and height, the dimensions of the side of the region are always powers of two. When further partitioning of the region is not performed, it is said that there is a ‘coding unit’ (CU) in the region. When the splitting is not performed at the upper level of the coding tree block, the area occupying the entire coding tree block contains one coding unit (CU), which is usually referred to as the ‘largest coding unit’ (LCU). For each coding unit, there is also a minimum size, such as the area occupied by 8x8 brightness sampling elements, although other minimum sizes are also possible. Coding units of a given size are generally referred to as ‘smallest coding units’ (SCU). As a result of this quad tree hierarchy, the entire coding tree block (CTB) is occupied by one or more coding units (CUs).

[0065] The video encoder 114 creates one or more arrays of sampling units, typically referred to as ‘prediction units’ (PUs) for each coding unit (CU). Different prediction unit (PU) arrangements are possible in each coding unit (CU), provided that the prediction units (PU) do not intersect and that the entire coding unit (CU) is occupied by one or more prediction units (PU). This scheme ensures that prediction units (PUs) cover the entire frame area.

[0066] The video encoder 114 operates by outputting, from the multiplexer module 340, the prediction unit (PU) 382. Difference module 344 derives the difference between the prediction unit (PU) 382 and the corresponding 2D array of data sampling elements, in the spatial domain, from the coding unit (CU) of the coding tree block (CTB) of frame data 310, the difference known as the 'residual sampling array' 360. An array 360 of residual sampling elements may be converted to a frequency domain in the transform module 320, or an array 360 of residual sampling elements may remain in the spatial domain, with two x options are performed by the multiplexer 321, operating under the control of the conversion skip control module 346, and is signaled by the conversion skip flag 386. The transform skip control module 346 determines a transform skip flag 386, which indicates whether the transform module 320 is used to convert the residual sampler array 360 to the residual coefficient array 362, or if the transform module 320 is skipped. The omission of the transform module 320 is referred to as a “transform omission”. When the conversion is not skipped, the array 360 of residual bins from the difference module 344 is received by the transform module 320, which converts (or ‘encodes’) the array 360 of residual bins from the spatial representation to the frequency domain representation by applying ‘direct transform’. The transform module 320 creates transform coefficients configured as an array 362 of residual transforms for each transform in a transform unit (TU) in the hierarchy of a coding unit (CU) in one or more transform units (TU), typically referred to as a “transform tree’. When the conversion skip is performed, the array 360 of residual sampling elements is represented in the encoded bitstream 312 in the spatial domain and the bypass of the transform module 320 is performed, which leads to the direct transfer of the array 360 of residual sampling elements to the scaling and quantization module 322 through the multiplexer 321, which operates under control flag 386 pass conversion. Conversion skip control module 346 can test the bit rate required in encoded bitstream 312 for each value of the conversion skip flag 386 (i.e., a skipped conversion, or a normal conversion operation). Transmission skipping control module 346 may select a value for the conversion skipping flag 386, which results in a lower bit rate in the encoded bitstream 312, thereby achieving higher compression efficiency. Each test performed by the conversion skip control module 346 increases the complexity of the video encoder 114, and therefore it is desirable to reduce the number of cases for which the conversion skip module 346 performs the test to those for which the advantage of selecting the conversion skip outweighs the cost of performing the test. For example, this can be achieved by restricting the conversion skip to specific transform sizes and block types, such as only 4x4 transforms for intra-predicted blocks (as described further below) in the emerging high-performance video coding (HEVC) standard. The conversion skipping functionality is especially useful for encoding arrays of 360 residual sampling elements that contain a large amount of ‘high-frequency’ information. High frequency information is typically present in frame data 310 containing many sharp edges, such as where alphanumeric characters are included in frame data 310. Other frame data sources 310, such as computer-generated graphics, may also contain a significant amount of high-frequency information. The DCT-like transform of the transform module 320 is optimized for frame data 310 containing mainly low-frequency information, such as that obtained from an imaging sensor capturing a natural image. Thus, the presence of the conversion skipping functionality provides a significant increase in coding efficiency for applications that are related to the emerging standard for high-performance video coding (HEVC). For video encoder 114, one drawback of supporting the skip conversion functionality is the need to test two possible modes for the skip conversion flag 386. As discussed below, the conversion skipping functionality is supported for an array of residual bins of 4 × 4 bins and when the array 360 of residual bins corresponds to an intra-prediction block, as described with reference to the intra-frame prediction module 336. However, the conversion skip flag 386 is desirably signaled separately for each color channel, and therefore, a separate test can be performed by the conversion skip control module 346 for each color channel. Separate signaling for each color channel is advantageous since the high-frequency information can be concentrated in one or both color channels, thereby being suitable for skipping the conversion, while the luminance channel can have minimal high-frequency information and, therefore, the advantage of using the conversion. For example, colored text on a colored background will lead to this scenario.

[0067] In relation to the emerging high-performance video coding (HEVC) standard, conversion to frequency domain representation is implemented using a modified discrete cosine transform (DCT), in which a conventional DCT is modified to be implemented using shifts and additions. Various sizes are possible with respect to the array 360 of residual sampling elements and transform coefficients 362, in accordance with the supported transform sizes. In the emerging standard for high-performance video coding (HEVC), conversions are performed on 2D arrays of sampling elements with specific sizes, such as 32 × 32, 16 × 16, 8 × 8 and 4 × 4. Therefore, it can be said that there is a predefined set of transform sizes available to video encoder 114. Moreover, as foreshadowed above, a set of transform sizes may vary between a luminance channel and a color channel. Two-dimensional transformations are usually performed 'separate', making it possible to implement as a first set of 1D transformations that operate on a 2D array of discretization elements in one direction (for example, in rows), followed by a second set of 1D transformations that operate on a 2D array of elements discretization output from the first set of 1D transforms in the other direction (for example, in columns). Transformations with the same width and height are usually referred to as ‘square transforms’. Additionally, transformations with different widths and heights are also possible and they are generally referred to as ‘non-square transformations’. Optimized transform implementations can combine one-dimensional transforms in rows and columns in specific hardware or software modules, such as a 4x4 transform module or an 8x4 transform module. Transformations with large dimensions require more schemes to implement, even though they can rarely be used. Accordingly, in the emerging standard for high-performance video coding (HEVC), there is a maximum conversion size of 32 × 32. The complex nature of the transformation implementation also makes it preferable to reduce the number of supported non-square transform sizes, since they will usually require completely new hardware to implement, instead of reusing the existing one-dimensional transformation logic present in the corresponding square transforms.

[0068] The transforms apply to both luminance and chrominance channels. There are differences between luminance and chrominance channel processing with respect to transform units (TUs) and will be described below with reference to FIG. 5A and 5B. Each transformation tree occupies one coding unit (CU) and is defined as a decomposition of the coding unit (CU) quad tree into a hierarchy containing one transformation unit (TU) at each end vertex of the transformation tree (quad tree) hierarchy, with each conversion unit (TU) has the ability to use transforms supported transform sizes. Similar to the coding tree block (CTB), it is necessary that the entire coding unit (CU) is occupied by one or more transform units (TU). At each level of the hierarchy of the quadtree of the transformation tree, the 'encoded block flag value' signals that there may be a conversion in each color channel, either at the current hierarchy level when there are no further partitions, or to signal that lower hierarchy levels may contain at least one Conversion from Resulting Transformation Units (TU). When the flag value of the encoded block is equal to zero, no conversion unit (TU) of the transformation tree is converted for the corresponding color channel, either at the present hierarchical level or at lower hierarchical levels. When the flag value of the encoded block is zero, the region contains a transform that must have at least one non-zero residual coefficient. Thus, for each color channel, zero or more transformations can span a portion of the area of the coding unit (CU), ranging from none to a full coding unit (CU). For each color channel, there are separate flag values for the encoded block. Each flag value of the encoded block is not required to be encoded, as there are cases where there is only one possible flag value of the encoded block.

[0069] Thus, the output of multiplexer 321 is one of the following: an array 360 of residual sampling elements or an array 362 of transform coefficients, and are simply labeled as an array 363 in FIG. 3. Array 363 is the input to scaling and quantization module 322, where its sample elements are scaled and quantized in accordance with a specific quantization parameter 384 to create a residual data array 364. The scaling and quantization process leads to a loss of accuracy, depending on the value of a specific quantization parameter 384. A higher value of the determined quantization parameter 384 leads to more information that is lost from the residual data. This increases the compression achieved by video encoder 114, at the cost of reducing the visual quality of output from video decoder 134. The determined quantization parameter 384 may be adapted during the encoding of each frame of frame data 310, or it may be fixed for a portion of frame data 310, such as a full frame. Other adaptations of a certain quantization parameter 384 are also possible, such as quantization of different residual coefficients using separate values. The residual data array 364 and the determined quantization parameter 384 are taken as input to the reverse scaling module 326, which reverses the scaling performed by the scaling and quantization module 322 to create rescaled data arrays 366, which are rescaled versions of the residual data array 364. The high-performance video coding standard (HEVC) under development also supports ‘lossless’ coding mode. When lossless coding is used, both the transform module 320 and the scaling and quantization module 322 are crawled, resulting in the residual coding array 360 being directly inserted into the entropy encoder 324. In lossless mode, crawling of the inverse scaling module 326 is also performed and module 328 inverse transformation. The selection of a lossless coding mode (as opposed to the conventional ‘lossy’ mode) is encoded in the encoded bitstream 312 by an entropy encoder 324. The logic for implementing a bypass with respect to the lossless mode is not illustrated in FIG. 3. Bypassing the scaling and quantization module 322 results in the absence of quantization of the residual coefficient array 362 or the residual sampling array 360, and a strict representation of the frame data 310 is encoded in the encoded bitstream 312 by the entropy encoder 324. The lossless encoding mode results in low encoder compression efficiency 114 videos and, as a result, is usually used only in applications where lossless coding is highly desirable, such as in medical applications.

[0070] The residual data array 364, the determined quantization parameter 384, and the conversion skip flag 386 are also taken as input to the entropy encoder module 324, which encodes the values of the residual data array 364 in the encoded bitstream 312 (or видео video bitstream ’). An array of 364 residual data in each transform unit is encoded in groups, collectively referred to as ‘subblocks’. The subunits should preferably have the same dimensions regardless of the size of the transformation, since this allows reuse of logic that relates to the processing of the subunit. Residual data in one subunit is usually referred to as a “data group” (or “coefficient group”, even when a conversion skip is applied and the “coefficient group” includes a spatial representation instead of a frequency domain representation) and for each data group, typically, a data group flag is encoded to indicate whether at least one value of the residual data in the data group is non-zero. In some cases, a data group flag may be implied and therefore not encoded. A flag is encoded for each residual data value belonging to the data group with the data group flag value equal to one to indicate whether the residual data value is non-zero (‘significant’) or zero (‘insignificant’). Due to the loss of accuracy resulting from the scaling and quantization module 322, the rescaled data arrays 366 are not identical to the original values in the 363 array. The rescaled data arrays 366 from the downscaling unit 326 are then output to the inverse transform unit 328. The inverse transform module 328 performs the inverse transform from the frequency domain to the spatial domain to create a spatial representation 368 of the arrays 366 of rescaled transform coefficients identical to the spatial domain representation that is created in the video decoder 134. The multiplexer 369 is configured to complement the operation of the multiplexer 321. The multiplexer 369 is configured to receive each of the arrays 366 of the rescaled data and the (converted) representation 368 in the spatial domain as input data and, under the control of the conversion skip flag 386, select one of the input data 366 or 368 as input to summing module 342.

[0071] The motion estimation module 338 creates motion vectors 374 by comparing frame data 310 with previous frame data from one or more frame sets stored in the frame buffer module 332, typically executed in memory 206. The frame set is known as the “reference picture list” '. The motion vectors 374 are then input to the motion compensation unit 334, which creates a mutual predictive prediction unit (PU) 374 by filtering the bins stored in the frame buffer module 332, taking into account the spatial displacement extracted from the motion vectors 374. Not illustrated in FIG. 3, but it is also possible to send motion vectors 374 as syntax elements to an entropy encoder module 324 for encoding in encoded bitstream 312. The intra prediction unit 336 creates an intra prediction unit (PU) 378 using the sampling elements 370 received from the module 342 an adder that sums the prediction unit (PU) 382 from the multiplexer module 340 and outputs in the spatial domain of the multiplexer 369. The intra-frame prediction module 336 also creates t intra prediction mode 380, which is sent to the entropy encoder 324 for encoding in the encoded bitstream 312.

[0072] Prediction units (PUs) may be generated using either the intra prediction method or the inter prediction method. The intra-prediction modes use the bins adjacent to the prediction unit (PU) that were previously decoded (typically above and to the left of the prediction unit) in order to generate reference bins in the prediction unit (PU). Various directions of intra prediction, referred to as ‘intra prediction mode’, are possible. Mutual prediction methods use a motion vector to refer to a block from a selected reference frame. Since a block can have any alignment up to an accuracy of a sub-sampling element, for example, in one-eighth of a sampling element, filtering is required to create a block of reference sampling elements for a prediction unit (PU). The decision as to which method to use is made in accordance with a speed-distortion trade-off between the required bit rate of the resulting encoded bitstream 312 and the amount of image quality distortion introduced by either intra prediction or mutual prediction. If intra prediction is used, one intra prediction mode is selected from a set of possible intra prediction modes, also in accordance with a speed-distortion tradeoff. The multiplexer module 340 selects either intra-prediction reference bins 378 from the intra-frame prediction module 336 or a cross-prediction unit (PU) 376 from the motion compensation module 334, depending on the decision made by the rate-distortion algorithm. Summing module 342 creates a sum 370 that is input to deblocking filter module 330. The deblocking filter module 330 performs filtering by block boundaries, creating the deblocked sampling elements 372, which are written to the frame buffer module 332 made in the memory 206. The frame buffer module 332 is a buffer of sufficient capacity to hold data from one or more past frames for future reference as part of the reference picture list.

[0073] In relation to the emerging high-performance video coding (HEVC) standard, the encoded bitstream 312 created by the entropy encoder 324 is delimited in units of the network abstraction layer (NAL). Typically, each frame slice is contained in one NAL unit. Entropy encoder 324 encodes a residual array 364, intra prediction mode 380, motion vectors, and other parameters collectively referred to as “syntax elements” in encoded bitstream 312 by executing a context-sensitive adaptive binary arithmetic coding (CABAC) algorithm. Syntax elements are grouped together into ‘syntax structures’, and these groups may contain recursion to describe hierarchical structures. In addition to ordinal values, such as intra prediction mode, or integer values, such as a motion vector, syntax elements also include flags, such as those to indicate a split quad tree. The motion estimation module 338 and the motion compensation module 334 operate on the motion vectors 374 with an accuracy of 1/8 of the brightness sampling element, providing accurate simulation of the motion between frames in frame data 310.

[0074] Although the video decoder 134 of FIG. 4 is described with reference to a high-performance video coding (HEVC) video decoding pipeline, processing steps performed by modules 420-434 are common to other video codecs that use entropy coding, such as H.264 / MPEG-4 AVC, MPEG -2 and VC-1. The encoded video information may also be read from memory 206, a hard disk drive 210, a CD-ROM, a Blu-ray ™ disc, or other computer-readable storage medium. Alternatively, the encoded video information may be received from an external source, such as a server, connected to a communications network 220 or radio frequency receiver.

[0075] As seen in FIG. 4, received video data, such as encoded bitstream 312, is input to video decoder 134. The encoded bitstream 312 may be read from memory 206, a hard disk drive 210, a CD-ROM, a Blu-ray ™ disc, or other computer-readable storage medium. Alternatively, encoded bitstream 312 may be received from an external source, such as a server, connected to a communications network 220 or radio frequency receiver. The encoded bitstream 312 contains encoded syntax elements representing captured frame data that must be decoded.

[0076] The encoded bitstream 312 is input to an entropy decoder module 420 that extracts syntax elements from the encoded bitstream 312 and transfers the values of the syntax elements to other blocks in the video decoder 134. Entropy decoder module 420 employs a context-sensitive adaptive binary arithmetic coding (CABAC) algorithm to decode syntax elements from encoded bitstream 312. Decoded syntax elements are used to reconstruct parameters in video decoder 134. Parameters include zero or more of the following: residual data array 450, motion vectors 452, prediction mode 454, and transform skip flag 468. The residual data array 450 is sent to the backscaling unit 421, the motion vectors 452 are sent to the motion compensation unit 434, and the prediction mode 454 is sent to the intra-frame prediction unit 426 and the multiplexer 428. The backscaling unit 421 backscales the residual data to create recreated data 455. When the conversion skip flag 468 is zero, the inverse scaling unit 421 outputs the recreated data 455 to the inverse transforming unit 422. The inverse transform module 422 uses ‘inverse transform’ to convert (or ‘decode’) the reconstructed data, which in this case are transform coefficients, from the frequency domain representation to the spatial domain representation, outputting an array of 456 residual sampling elements through the multiplexer module 423. When the value of the conversion skip flag 468 is one, the reconstructed data 455, which in this case is in the spatial domain, is output as an array 456 of residual sampling elements through the multiplexer module 423. The inverse transform module 422 performs the same operation as the inverse transform module 328. The inverse transform module 422, therefore, must be configured to provide a predetermined set of transform sizes required to decode the encoded bitstream 312, which is compatible with the emerging high-performance video encoding (HEVC) standard. When the signaling in the encoded bitstream 312 indicates that a lossless mode has been used, the video decoder 134 is configured to bypass the inverse scaling unit 421 and the inverse transform unit 422 (not illustrated in FIG. 4), resulting in an array 450 residual data is directly input to summing unit 424.

[0077] The motion compensation module 434 uses motion vectors 452 from an entropy decoder module 420 combined with reference frame data 460 from a frame buffer unit 432 made in memory 206 to create a mutual predictive prediction unit (PU) 462 for a prediction unit (PU ), which is the prediction of the output decoded frame data. When prediction mode 454 indicates that the current prediction unit has been encoded using intra prediction, intra prediction unit 426 creates intra prediction unit (PU) 464 for intra prediction unit (PU) using sampling elements spatially adjacent to the prediction unit (PU), and a prediction direction also provided by prediction mode 454. Spatially adjacent bins are obtained from the sum 458 output from the summing unit 424. Multiplexer module 428 selects an intra-prediction prediction unit (PU) 464 or an inter-prediction prediction unit (PU) 462 for the prediction unit (PU) 466, depending on the current prediction mode 454. The prediction unit (PU) 466, which is output from the multiplexer module 428, is added to the array 456 of residual sampling elements from the inverse scaling and conversion module 422 by means of the summing module 424 to create a sum 458, which is then input into each of the modules: deblocking filter module 430 and an intra-frame prediction module 426. The deblocking filter module 430 filters by the boundaries of the data block, such as transform unit (TU) boundaries, to smooth out visible artifacts. The output of the deblocking filter module 430 is written to a frame buffer module 432 made in the memory 206. The frame buffer module 432 provides sufficient storage to hold one or more decoded frames for future reference. The decoded frames 412 are also output from the frame buffer module 432 to a display device, such as a display device 136.

[0078] FIG. 5A and 5B each show a grid of discretization elements of a frame portion 500 and a frame portion 510 encoded with 4: 2: 0 and a color format of 4: 2: 2, respectively. The color format is indicated as a configuration parameter to the video encoder 114 and the video encoder 114 encodes a syntax element ‘chroma_format_idc’ in the encoded bitstream 312, which indicates the color format. The video decoder 134 decodes the syntax element ‘chroma_format_idc’ from the encoded bitstream 312 to determine the color format used. For example, when a 4: 2: 0 color format is used, the chroma_format_idc value is one, when a 4: 2: 2 color format is used, the chroma_format_idc value is two, and when a 4: 4: 4 color format is used, the chroma_format_idc value is three. In FIG. 5A and 5B, the locations of luminance bins, such as the location 501 of the luma bins, are illustrated with ‘X’ symbols, and the locations of the color bins, such as the location 502 of the bins, are illustrated with ‘O’ characters. By sampling a portion of frame 500 at the indicated points, a grid of sampling elements is obtained for each color channel when a 4: 2: 0 color format is applied. At each location X of the luma sampling element, the luma channel is sampled (‘Y’), and at each location O of the chroma sampling element, both color channels are sampled (‘U’ and ‘V’). As shown in FIG. 5A, for each location of the chroma element, there is a 2x2 arrangement of the locations of the luma elements. By sampling the brightness sampling elements at the locations of the brightness sampling elements and the color sampling elements at the locations of the color sampling elements indicated on the frame portion 510, a grid of sampling elements for each color channel is obtained when the 4: 2: 2 color format is applied. Exactly the same distribution of bins for color channels is made for section 510 of the frame, as for section 500 of the frame. In contrast to the frame portion 500, in the frame portion 510 there are twice as many locations of the color sampling elements. In a portion 510 of the frame, the locations of the color sampling elements are co-located with each second location of the brightness sampling element. Accordingly, in FIG. 5B, for each chroma location, there is a 2x1 arrangement of luma sampling locations.

[0079] The various allowable dimensions of the conversion units have been described above in units of brightness sampling units. The area covered by the transformation applied to the luminance channel, therefore, will have the same dimensions as the dimensions of the transformation unit. Since conversion units also encode color channels, the applied conversion for each color channel will have dimensions adapted to the particular color format used. For example, when a 4: 2: 0 color format is used, a 16 × 16 transform unit (TU) will use a 16 × 16 transform for the luminance channel, and an 8x8 transform for each chrominance channel. One special case is that when a 4x4 conversion is used for a luminance channel, there is no corresponding available 2x2 conversion (when a 4: 2: 0 color format is applied) or an available 4 × 2 conversion (when a 4: 2: 2 color format is applied ), which can be used for color channels. In this particular case, the 4x4 transformation for each color channel may span the area occupied by several brightness transformations.

[0080] FIG. 6A is a schematic representation of an example transform tree of a coding unit (CU) 602 (depicted by a thick border) in a frame coding tree (CTB) block 600. One quad-tree division divides the coding tree block (CTB) 600 into four 32 × 32 coding units (CUs), such as coding unit 602 (CUs). An exemplary transform tree exists in coding unit 602 (CU). An exemplary coding tree includes several quad-tree divisions that result in ten transformation units (TUs), numbered such as those in FIG. 6A, for example, conversion unit 604 # 9 (TU). Conversion units # 1 to # 10 span the entire coding unit (CU) 602. Each quad-tree unit spatially divides the area into four quarters, giving four smaller areas. Each transformation unit (TU) has a conversion depth value corresponding to the hierarchical level of a transformation unit (TU) in the transformation tree. The hierarchical level indicates the number of quad-tree divisions that are completed before the quad-tree divisions complete, resulting in a transform unit (TU) instance that spans the corresponding area. For example, conversion unit # 9 (TU) 604 occupies one fourth of the zone of a coding unit (CU) 602 and, therefore, has a conversion depth of one. Each transform unit (TU) has an associated size (or преобразования transform size ’), typically described by the dimensions of the region containing the transform unit (TU) over the grid of luminance sampling elements. The size depends on the size of the coding unit (CU) and the conversion depth. Conversion units (TUs) with a conversion depth of zero are equal to the size of the corresponding coding unit (CU). Each increment of the conversion depth leads to halving the size of the transformation units (TU) present in the transformation tree at a given conversion depth. Since the frame includes a luminance channel and chrominance channels, a coding unit (CU) occupies an area both in the grid of luma sampling elements and in the grid of chroma sampling elements, and therefore, each transform unit (TU) includes information, describing both luma sampling elements on a grid of luma sampling elements and chroma sampling elements on a grid of chroma sampling elements. The nature of the information for each transform unit (TU) depends on the processing stage of the video encoder 114 or video decoder 134. At the input of the transform module 320 and the output of the inverse scaling and transform module 422, the array 360 and 456 of residual sampling elements respectively contains information for each transform unit (TU) in the spatial domain. An array of 360 and 456 residual sampling units can be further divided into ‘an array of residual sampling elements’ and ‘an array of residual sampling elements’, due to differences in processing between the luminance channel and the color channels. At the output of the scaling and quantization module 322 and at the input of the inverse scaling and transform module 422, the residual data array 364 and 450 respectively contains information for each transform unit (TU) in the frequency domain. Arrays 364 and 450 of residual data can be further divided into ‘an array of residual chroma data’ and ‘an array of residual luminance data’, due to differences in processing between the luminance channel and the color channels.

[0081] FIG. 6B illustrates an example transform tree 630 corresponding to the example transform tree of FIG. 6A, for a 32x32 luminance channel, a coding unit (CU) containing a set of transform units (TU), and occupying a coding unit 602 (CU), which occupies a 32 × 32 array of brightness sampling elements over a grid of brightness sampling elements. FIG. 7 illustrates a data structure 700, which is an example transform tree 630. In FIG. 6B, frames numbered 1 to 10 indicate conversion units present in region 632 (illustrated by several transformation units (TU) 640), and each frame is contained in an area that is no longer subdivided (indicated by a dotted border) .

[0082] FIG. 6B, frames numbered 1 and 9 contain 16 × 16 transforms for a luminance channel, frames numbered 2, 3, and 8 contain 8 × 8 transforms for a luminance channel, and frames numbered 4 to 7 contain 4 × 4 transforms for channel brightness. The corresponding area (dashed frame) for each of these frames has a flag value of the encoded block equal to one, to indicate the presence of conversion.

[0083] The presence or absence of a conversion for each color channel is indicated by a separate flag value of the encoded block, which is used both in encoding and decoding the bitstream, but which does not have to be transmitted in the bitstream, as discussed below. Therefore, the number of arrays 450 of residual coefficients output from the entropy decoder 420 depends on the flag values of the encoded block. When there are no significant coefficients (i.e., all coefficients are zero) in any color channel, the number of arrays 450 of residual data (coefficients) output from the entropy decoder 420 is zero.

[0084] FIG. 7, the circles represent the values of the transform split flag with the value of the transform split flag indicated inside the corresponding circle. In FIG. 7, the triangles are the flag values of the encoded block, with the flag value of the encoded block indicated inside the corresponding triangles. The squares are transformation units, with each transformation numbered to match the numbering of the transformation shown in FIG. 6B.

[0085] The topmost hierarchical level of the exemplary transform tree 630 comprises a region 632 occupying a 32x32 coding unit (CU). The conversion splitting flag value 702 indicates that the area 632 is subdivided into four 16x16 areas, such as the area 634, thereby setting the “non-terminal” vertex of the example transformation tree 630. For each 16 × 16 area, a further transform split flag value, such as a transform split flag value 704, indicates that the corresponding 16 × 16 area should be further subdivided into four 8 × 8 areas. For example, the region 634 is not further subdivided, as indicated by the value of the conversion splitting flag 704 to zero, thereby setting the ‘end’ vertex of the example transformation tree 630. In contrast, region 638 is further subdivided into four 4x4 regions (such as region 636), as indicated by a conversion split flag value 712 of one. The recursive splitting structure present in the transform tree 630 is similar to the quad tree splitting present in the coding tree block (CTB). With respect to the luminance channel, at the ых end ’vertices of the quad tree, the presence of a transformation in a transform unit (TU) is signaled by the flag value of the encoded block, for example, a value of 708 flag of the encoded block equal to one indicates the presence of transform 710 in region 634.

[0086] Since the transform can be used to represent residual data in each region, regions are not allowed to be smaller than the smallest supported transform size, such as 4x4 luma sampling elements for a luminance channel. Additionally, for areas larger than the largest available transform size, the value of the transform split flag is assumed to be one. For example, for a transform tree with a top level of 64x64 coding units, automatic division (i.e., not signaled in encoded bitstream 312) into four 32x32 regions occurs when the largest supported transform size is 32x32 brightness sampling elements.

[0087] The lower right 16 × 16 region 642 contains a transform unit (TU) (numbered 10 (ten) and shaded) with no transform for the luminance channel and, therefore, has a corresponding encoded block flag value 716 equal to zero.

[0088] FIG. 6C and 8 illustrate an example transform tree 630 corresponding to the example transform tree of FIG. 6A, for a color channel configured for a 4: 2: 2 color format, and comprising a transform set for a color channel corresponding to a transform tree 630 for a luminance channel, and represented by a data structure 800. Since the hierarchy of the transformation tree is common due to the structure of FIG. 6A between the luminance channel and the color channels, conversion splitting flag values are shared between data structures 700 and 800. In contrast to the data structure 700, the data structure 800 includes a coded block flag value with each transform split flag value equal to one (i.e., at non-terminal vertices of the transform tree). For example, a coded block flag value of 802 equal to one is associated with a conversion splitting flag 702. If the flag value of the encoded block at the non-terminal vertex of the transformation tree is zero, the flag values of the encoded block at the vertices of the descendants are assumed to be zero (and the corresponding flags of the encoded block are not encoded in the encoded bitstream 312). The flag values of the encoded block at non-terminal areas allow completion of the encoding of the flags of the encoded block at lower levels of the transformation tree for each color channel if significant residual coefficients are not present in any of the descendant regions, even though significant residual coefficients may be present in the channel chroma. This is a common situation for typical captured frame data, since most of the information is presented in the luminance channel.

[0089] When the video encoder 114 and the video decoder 134 are configured for a 4: 4: 4 color format, a color region of each color channel of any predetermined transform unit (TU) size that is not one of a predetermined set of transform unit (TU) sizes, has dimensions identical to the brightness regions of a given transformation unit (TU) (i.e., when an implied partition does not occur). When the video encoder 114 and the video decoder 134 are configured for a 4: 4: 4 color format, the color region of each color channel of any given transform unit (TU) size, which is one of a predefined set of transform unit (TU) sizes, has dimensions less than brightness areas of a given transformation unit (i.e., when an implied partition does not occur).

[0090] When a 4: 2: 2 color format is used, this leads to a coding unit (CU) 602 including a 16 × 32 region 662 of FIG. 6C from chroma sampling elements for each chroma channel and, therefore, occupying a 16 × 32 region along the grid of chroma sampling elements. FIG. 6C illustrates areas on a grid of color sampling elements drawn as an array of color sampling elements, with each color sampling element equidistant horizontally and vertically (as opposed to Fig. 5B). Due to the use of a 4: 2: 2 color format, each of the color areas of FIG. 6C becomes horizontally compressed with respect to the corresponding luminance region of FIG. 6B. A conversion splitting flag value of 702 equal to one divides the 16 × 32 region 662 corresponding to the coding unit 602 (CU) into four 8 × 16 regions, such as the 8 × 16 region 664. The 8 × 16 region 664 is non-square and also larger than other non-square areas illustrated in FIG. 6C, such as a 4 × 8 region 670. For each 8 × 16 region, a transform split flag value, such as a transform split flag value 704, indicates whether the corresponding 8 × 16 region should be further subdivided into four smaller 4 × 8 regions, in the same way the quad tree partition present in the transform tree 630 for the array of luminance sampling elements. The upper right 8 × 16 region 672 is further subdivided into four 4 × 8 regions. A coded block flag value of 804 equal to one indicates that each of the four 4 × 8 regions may contain significant residual coefficients. The flag of the coded block for each 4 × 8 region, therefore, is required to indicate the presence of a transform for the corresponding region. Of these four 4 × 8 regions, the lower left 4 × 8 region 674 (shaded) contains a transform unit (TU) but does not contain a transform and, therefore, has a coded block flag value 814 equal to zero. The remaining 4 × 8 regions, such as region 670, each have a transform and, therefore, has corresponding flag values of the encoded block equal to one. The upper left 8 × 16 region is divided into two equally sized 8 × 8 regions. In contrast to the quad tree division, the corresponding transform split flag is not present in the encoded bitstream 312.

[0091] A partition of a channel region, such as a color channel, transform units (TUs) into several regions (each of which may have a transform), without the presence of signaling in encoded bitstream 312, is referred to as “implied splitting”. Implied partitioning eliminates the need for maintaining hardware that supports non-square transform for this case (8 × 16). Instead, transforms are used, such as the first 8 × 8 transform of 666. Since it is possible that each area obtained from an implied partition contains all zero residual information, it is necessary to indicate the presence of a transformation in each region obtained from an implied partition. Accordingly, separate flag values of the encoded block are required for each region obtained from the implied partition. In this case, the flag values 806 and 808 of the encoded block correspond to the first 8 × 8 transform 666 and the second 8 × 8 transform 668, respectively. For conversion units (TUs) in which there is no implied splitting, the flag value of the coded block for each color channel indicates the presence or absence of conversion for the area occupied by the conversion unit (TU) for the color channel. When an implied splitting takes place, a separate flag value of the encoded block (not illustrated in Fig. 8) is required for each of the resulting areas, however, implementations can retain the flag value of the encoded block, which is typical for the entire transform unit (TU). A single coded block flag value may be implied as ца one ’in all cases, or a single coded block flag value may be determined by performing a logical OR operation with respect to the coded block flag value of each region obtained from the partition. If a single encoded block flag value is determined from a coded block flag value of each region obtained from the partition, a single encoded block flag value can be encoded in encoded bitstream 312 by entropy encoder 324 and decoded from encoded bitstream 312 by entropy decoder 420 as an additional flag encoded block (not illustrated in Fig. 9). In this case, when an individual flag value of the encoded block is equal to zero, the flag value of the encoded block of each region from the partition can be assumed to be zero, and when a separate flag value of the encoded block is equal to one, the flags of the encoded block for each region of the partition are encoded in the encoded bitstream 312 entropy encoder 324 and are decoded from encoded bitstream 312 by entropy decoder 420.

[0092] The lower left 8 × 16 region 680 16 × 32 of region 662 illustrates an implied partition, where an 8 × 8 transform is present in the upper 8 × 8 implied region 682, but an 8x8 transform is not present in the lower 8 × 8 implied region 684. Lower right The 8 × 16 array 676 (shaded) contains the transform unit (TU), but does not contain the conversion in any square 8 × 8 region obtained from the implied partition, and, therefore, has the encoded block flag values 810, 812 equal to zero.

[0093] The presence of two color channels leads to duplication of the structure depicted in FIG. 6C, with individual encoded block flag values used to indicate the presence of a transform for each chroma channel. In this implementation, a partition was implied for region sizes with respect to color other than 4 × 8, resulting in the use of a 4 × 8 rectangular transform, such as the 4 × 8 transform 816 (contained in region 670), and reusing existing square transforms in other cases (e.g. 8 × 8, 16 × 16). Thus, we can say that there is a set of predefined sizes of regions (such as 8 × 16 and 16 × 32), for which a split into two regions can be used, and, therefore, two transformations (sizes 8 × 8 and 16 × 16) . Different definitions of a predefined set of sizes of regions for which an implied partition occurs is also possible and will provide the possibility of using a different combination of existing square transformations and rectangular transformations. Also, in relation to certain implementations, it is possible to always imply a partition, and in this case a rectangular transformation for color channels 4: 2: 2 is not introduced. In this case, the predefined set of sizes of the regions for which the implied splitting occurs contains all possible sizes of the chroma region (for example, 4 × 8, 8 × 16 and 16 × 32 for the color format 4: 2: 2, or 4 × 4, 8 × 8, 16 × 16 and 32 × 32 for 4: 4: 4 color formats).

[0094] FIG. 16 is a schematic view showing an example of ‘non-rectangular transform’ for implementing ‘always’ implied splitting for all possible area sizes (4 × 8, 8 × 16 and 16 × 32) for 4: 2: 2 color formats. As illustrated in FIG. 16 by marking ‘1’ (one) and ‘2’ (two) for each color region obtained from the implied partition.

[0095] When a 4: 2: 0 color format is used, an implied partition does not occur for any color region in a transform unit (TU), therefore, the maximum number of conversions for each color channel is always one (the value of the encoded block flag for each channel chroma controls whether chroma conversion occurs).

[0096] Although the video encoder 114 and the video decoder 134 are described regardless of differences between the luminance and chrominance channels, different nets of sampling elements obtained from chroma formats inevitably entail a need for differences in modules. Practical implementations may have separate обработки processing routes ’for the luminance channel and for the color channels. Such an implementation, therefore, can disconnect the processing of luma sampling elements and chroma sampling elements. Since encoded bitstream 312 is one bitstream for luminance and chrominance channels, the entropy encoder 324 and the entropy decoder are not disconnected. Additionally, a single frame buffer, such as a frame buffer 332432, holds luma and chroma sampling elements and, therefore, is not disconnected. However, modules 322-330 and 334-340 and modules 422-430 and 434 can have separate luminance and chrominance processing, allowing implementations to have separate logic for luminance and chrominance, thereby creating a 'luminance processing route' and 'processing route color '.

[0097] Some implementations may imply a partition for a 16x32 region of a color channel of a transform unit (TU) into two 16x16 regions, but do not imply a partition for cases 8x16 and 4x8. Such implementations avoid the need to introduce a 32-point conversion logic into a color processing route, instead relying on 4, 8, or 16-point conversion logic, well developed in the art.

[0098] FIG. 9A and 9B illustrate a syntax structure that can be used to encode or otherwise represent the hierarchical level of a transform tree. At non-terminal vertices of the transform tree, the syntax structure 900 is recursively expanded in accordance with data structures, such as data structures 700 and 800, to define syntax elements present in a portion of the encoded bitstream 312 corresponding to the transform tree. At the ends of the transform tree (where there is no further division in the transform tree), the syntax structure 930 defines the syntax elements present in the portion of the encoded bitstream 312. As a rule, there is one data structure for luminance and two data structures for chroma, despite that additional data structures are possible, such as for encoding an alpha channel or depth map. Alternatively, fewer data structures may be used, as in the case where one data structure is shared by color channels and it is possible to share the flag values of the encoded block with color channels. The non-end vertex syntax structure of the transform tree 902 defines encoding of one hierarchical level of the transform tree, such as transform tree 630. The conversion splitting flag 910 encodes a conversion splitting flag value of one, such as the conversion splitting flag value 702. This value indicates that the syntax structure 902 of the non-end vertex of the transform tree includes a lower hierarchical level that contains additional instances of the syntax structure 902 of the non-end vertex of the transform tree or the syntax structure 902 of the end vertex of the transform tree, or ‘descendants’. The coded block flag 912 encodes the coded block flag value 802 equal to one for the color channel ‘U’, and the coded block flag 914 encodes the additional coded block flag value for the color channel ‘V’. If the syntax structure 902 of the non-end vertex of the transform tree is the upper level hierarchy of the transform tree, then there are flags 912,914 of the encoded block. If the syntax structure 902 of the non-end vertex of the transform tree does not define the top level of the hierarchy of the transform tree, then the flags of the encoded block 912 914 are present only if the corresponding flags of the encoded block at the parent level of the hierarchy of the transform tree are present and have a value of one. Since in the transformation tree 630 there is a lower hierarchical level (relative to the upper hierarchical level), there is a subdivision of the quad tree. This subdivision leads to four transformation tree syntax structures 916, 918, 920, 922 (identified by the ‘blkIdx’ variable (block index), numbered from zero to three) that are included in the non-end vertex syntax structure 902 of the transformation tree.

[0099] The syntax structure 903 defines the encoding of the end vertex of the end vertex 932 of the transform tree (that is, where no further division takes place). The transform splitting flag 940 encodes a transform splitting flag value of zero, such as the transform splitting flag value 704.

[0100] The conversion split flag is only encoded if the corresponding area is larger than the minimum size. For example, region 636 has the smallest possible size for a region of 4 × 4 luma sampling elements (corresponding to the smallest supported luma conversion size), so that the transform split flag value 714 is assumed to be zero and the transform split flag is not encoded for the corresponding transform tree syntax structure.

[0101] With respect to region 636, the residual chroma sampling elements are converted using a 4 × 8 chroma conversion, therefore, no implied conversion splitting is present. Coded block flags, such as coded block flag 942 and coded block flag 946, may be present to signal the presence of a transform for each of the color channels. A coded block flag 950 signals the presence of a transform for the luminance channel. Residual coefficients for luminance and chrominance channels (if present) are present in the transform unit (TU) syntax structure 952. If the value of the encoded block flag 950 is equal to one, the luminance conversion skip flag 964 and the luminance residual data unit 954 encoding either the residual coefficients for the luminance conversion or the residual sampling elements when the conversion is skipped are present in the encoded bitstream 312. The value of the mismatch conversion flag 964 brightness indicates whether the conversion module 320 in the video encoder 114 and the rotary module 422 are used (during normal operation) or bypassed (during the conversion skip operation) atnogo conversion to video decoder 134. If the flag value of the encoded block for each color channel is one, the corresponding color conversion skip flags 966 and 968 and residual color blocks 956 and 960 are present in the encoded bitstream 312. The conversion skip flag 966 signals the conversion skip mode for the residual color block 956, and the flag The conversion skip 968 signals the conversion skip mode for the residual color block 960. When an implied splitting of a transform does not occur, the encoded block flag 944 and 948 and color residual blocks 958 and 962 are absent in the encoded bitstream 312. When an implied splitting of a transform does not occur, the transform skip flag for each color channel therefore signals the conversion skip mode for the corresponding color channel in the entire area 636.

[0102] With respect to region 664, residual chroma sampling units are converted using two 8x8 color transforms, therefore, an implied partition of the transform is present. Flags 942 and 946 of the encoded block, if present, signal the presence of 8x8 transformations for each color channel of the first 8x8 transform 666. Flag 944 of the encoded block and flag 948 of the encoded block, if present, signal the presence of 8x8 transforms for each color channel of the second 8x8 transform 668. If the value of the flag 944 of the encoded block is one, the residual chroma block 958 is present in the encoded bitstream 312. If the value of the flag 948 of the encoded block is one, the residual block 962 of color a validity value is present in the encoded bitstream 312. The conversion skipping flag 966 signals the conversion skipping mode for the residual color blocks 956 and 958, and the conversion skipping flag 968 signals the conversion skipping mode for the residual color blocks 960 and 962. When an implied transform partition is present, the transform skip flag for each color channel is therefore a signal of the transform skip mode for the corresponding chroma channel in the entire region 664, in accordance with the behavior when the implied transform partition is not present.

[0103] The syntax structure 930, as illustrated in FIG. 9B shows the first and second transforms of each chroma channel encoded adjacent to the implicit splitting transform. Alternatively, other arrangements may be used, such as adjacent coding of syntax elements for each chroma, or coding of syntax elements for each chroma channel alternating with other syntax elements.

[0104] FIG. 9C, 9D, and 9E illustrate an alternative syntax structure 9100 that can be used to encode or otherwise represent the hierarchical level of a transform tree. At non-terminal vertices of the transform tree, an alternative syntax structure 9100 is recursively expanded in accordance with data structures, such as data structures 700 and 800, to specify syntax elements present in a portion of encoded bitstream 312 corresponding to the transform tree. An instance of an alternative syntax structure 9100 exists for each vertex in the transform tree, including end vertices, each of which contains a transform unit (TU). Where there is an 'implied split' for subdividing a transform unit (TU) for each color channel, syntax structure 9130 defines syntax elements present in a portion of encoded bitstream 312 for the first sub-region obtained from the implied split (e.g., the upper half of the chrominance, when used 4: 2: 2 chroma format or upper left quarter of the chroma area when 4: 4: 4 chroma format is used). In addition, syntax structure 9160 defines syntax elements present in a portion of encoded bitstream 312 for subsequent subregions obtained from an implied partition (e.g., another subdomain for the lower half of the chroma region when the 4: 2: 2 color format is used, and the remaining three subregions of the chroma region when a 4: 4: 4 chroma format is used). Sending to the ‘first’ subregion and ‘the subsequent’ subregion (for example, the second and possibly the third and fourth subregions) is implied in the sequence of scanning subregions of the area in the quad tree. The scanning sequence is such that the subregions intersect first from left to right, and then from top to bottom. Typically, there is one data structure for luminance and two data structures for chroma, although additional data structures are possible, such as for encoding an alpha channel or depth map. Alternatively, fewer data structures may be used, such as in the case where one data structure is shared by color channels and it is possible to share the flag values of an encoded block between color channels. The transform tree syntax structure 9102 defines encoding of one hierarchical level of the transform tree, such as transform tree 630.

[0105] With respect to an instance of the transform tree syntax structure 9102 at the non-terminal top of the transform tree, such as transform tree 630, the transform split flag 9110 encodes the transform split flag value of one, such as the transform split flag value 702. This value indicates that an instance of the transform tree syntax structure 9102 includes a lower hierarchical level containing additional instances of the transform tree syntax structure 9102 or ‘descendants’. The coded block flag 9112 encodes the coded block flag value in accordance with the description of the coded block flag 912. The coded block flag 9114 encodes the coded block flag value in accordance with the description of the coded block flag 914. Since there is a lower hierarchical level in the transform tree 630 (relative to the upper hierarchical level), there is a subdivision of the quad tree. This subdivision leads to four transform tree syntax structures 9116, 9118, 9120, 9122 (identified by the variable ‘blkIdx’, numbered from zero to three) that are included in the transform tree top syntax structure 9102. Each of the transform tree syntax structures 9116, 9118, 9120, 9122 is a different instance of the transform tree syntax structure 9102. The encoded block flag 9124 and the luminance conversion unit section 9126, encoding either the residual coefficients for the luminance conversion or the residual sampling elements when the transform is skipped, will be absent from the transform tree syntax structure 9102.

[0106] Implementations can also compose the transform tree syntax structure 9102 so that the encoded block flag 9124 and the luminance conversion unit section 9126 (if present) are placed earlier in the transform tree syntax structure 9102, such as between the encoded block flag 9114 and the syntax structure 9116 transformation tree.

[0107] With respect to an instance of the transform tree syntax structure 9102 at the end vertex of the transform tree, such as the transform tree 630, the transform split flag 9110 encodes the transform split flag value of zero, such as the transform split flag value 704. An instance of the transform tree syntax structure 9102 therefore corresponds to a transform unit (TU) in the transform tree 930. A transform unit (TU) has a size that is determined in accordance with a coding unit (CU) containing a transform unit (TU), such as a coding unit (CU) 602, and a transform depth. The coded block flag 9112 encodes a coded block flag value of one to indicate that any of the color areas obtained from the implied partition for the ‘U’ color channel may have a coded block flag value of one. If the encoded block flag 9112 encodes a value of zero, then the encoded block flag value for each color region obtained from the implicit splitting for the color channel ‘U’ has the encoded block flag value implied as zero. Even when the encoded block flag 9112 encodes a value equal to one, implementations can still encode the encoded block flag with a value equal to zero for each color region obtained from the implied partition. Because of this, implementations may omit the encoded block flag 9112 in encoded bitstream 312, instead of the always-implied coded block flag value of one for the omitted encoded block flag 9112. The coded block flag 9114 encodes the additional coded block flag value for the канала V ’color channel in a manner similar to that used for the coded block flag 9112. With respect to transform unit (TU) sizes that are consistent with those for which an implied division into four color regions occurs (the maximum number of arrays of residual chroma coefficients is four), four transform tree syntax structures 9116 9118 9120 9122 (identifiable by 'blkIdx' from zero up to three) vertices of the transform tree are included in the syntax structure 9102. For transform unit (TU) sizes that are consistent with those for which an implied split into two color regions occurs (the maximum number of arrays of residual chroma coefficients is two), two transform tree syntax structures, such as transform tree syntax structures 9116 9118 (identifiable 'blkIdx' from zero to one) are included in the top structure of the transform tree syntax 9102. Each of the transform tree syntax structures 9116 9118 9120 9122 is an instance of a transform tree for the chroma syntax structure 9132. A coded block flag 9124 encodes a coded block flag value, such as a coded block flag value 708, indicating that there is no conversion for a luminance channel of a transform unit (TU). The luminance section of the transform unit 9126 encodes a luminance conversion skip flag as a transform skip flag 9127 and an array of residual luminance factors as residual luminance syntax elements 9128.

[0108] The transform tree for the luminance syntax structure 9132 that exists only for the first color region (or ‘subregion’), when an implied partition occurs, includes a reduced syntax set of the transform tree syntax structure 930. The coded block flag 9142 encodes the coded block flag value for the chroma channel ‘U’ of the chroma region. A coded block flag 9144 encodes a coded block flag value for a color channel ‘V’ of a color domain. The color section of the transform unit (TU) 9146 encodes a subset of the transform unit (TU) syntax structure 952. The color section of the conversion unit 9146 (TU) encodes color transforms containing color data for one color channel. The color transforms are encoded in the form of an array of residual chroma coefficients as residual chroma syntax elements 9150 for the color channel 'U' if the value of flag 9142 of the encoded block is one, and an array of residual chroma coefficients as residual chroma syntax elements 9152 for the chroma channel 'V' if the value of flag 9144 of the encoded block is equal to one (collectively, arrays of residual coefficients for 'color conversions'). The conversion skipping flag 9148 is associated with the residual chroma syntax elements 9150 and encodes the value of the conversion skipping flag for the канала U ’color channel, for each color region obtained from the implied partition. The conversion skipping flag 9151 is associated with the residual color syntax elements 9152 and encodes the value of the conversion skipping flag for the color channel ‘V’, for each color area obtained from the implied partition. This relationship is through the fact that the conversion skip flag is encoded in a ‘residual encoding’ syntax structure that includes the corresponding residual syntax elements.

[0109] The transformation tree for a chroma syntax structure 9162 that exists only for chroma regions other than the first chroma region (or ‘subregion’), when an implied partition occurs, includes an abbreviated syntax set of the transform tree syntax structure 930. The coded block flag 9172 encodes the coded block flag value for the chrominance channel ‘U’ of the chroma region. A coded block flag 9174 encodes a coded block flag value for a color channel ‘V’ of a color domain. The color section of the transform unit (TU) 9176 encodes a subset of the transform unit (TU) syntax structure 952. The luminance portion of a transform unit (TU) 9176 encodes an array of residual chroma coefficients as residual chroma syntax elements 9180 for a color channel ‘U’ if the value of flag 9172 of the encoded block is one. The color portion of the transform unit 9176 (TU) encodes an array of residual chroma coefficients as residual chroma syntax elements 9182 for a color channel ‘V’ if the value of flag 9174 of the encoded block is one. The conversion skipping mode for the area corresponding to the residual color syntax elements 9180 is determined from the conversion skipping flag 9148. The conversion skipping mode for the area corresponding to the region corresponding to each residual color syntax elements 9182 is determined from the conversion skipping flag 9151. Implementations may use hardware registers, such as registers 246, or memory 206, to store the skip flag of the transform from the first color region for use in subsequent sub-region (s).

[0110] Syntax structures 9130 and 9160, as illustrated in FIG. 9D and 9E, show the first and second flag of the encoded block encoded adjacent to, followed by the first and second array of residual chroma coefficients of each chroma channel for the implicit splitting of the transform. Alternatively, other arrangements may be used, such as encoding the flag of the encoded block and the array of residual chroma coefficients adjacent to each chroma channel.

[0111] Although the implicit splitting of the transform is illustrated using the 8 × 16 area 664, divided into two 8 × 8 areas, alternative implementations can perform the splitting for other areas. For example, some implementations may involve splitting a 16 × 32 region into two 16 × 16 regions. Such implementations mainly avoid the need for 32-point 1D conversion along the color processing path. Since a 32-point 1D conversion is not required for a color processing route when a 4: 2: 0 color format is applied, the requirement for a 32-point 1D conversion is completely removed from the color processing route. Implementations that use a separate processing scheme to decouple the luminance and color channels, therefore, can receive a lower implementation cost in the color processing scheme.

[0112] 4: 4: 4 a color format exists where there is one location of a color sampling element for each location of a brightness sampling element. Accordingly, with this format, the transformations for the color channel and the luminance channel can have the same dimensions. With the largest conversion size equal to 32 × 32 in the brightness processing route, this will require entering 32 × 32 conversion into the chroma processing route for a disconnected implementation. Specific implementations may involve splitting for each color channel, splitting a 32 × 32 region into four 16 × 16 regions, allowing reuse of the existing 16 × 16 transform in the color processing path. Since the 32 × 32 transform will be used only in the color processing path for a 4: 4: 4 color format, implying a split for each color channel to divide a 32 × 32 region into four 16 × 16 regions will remove the 32 × 32 transform from the color processing path, reducing the required processing scheme. Such implementations will require four encoded block flag values for each chroma channel, and therefore, up to four encoded block flags, are encoded in the syntax structure 930 for each chroma channel in encoded bitstream 312.

[0113] Implementations that support a 4: 2: 2 color format may also include dividing for each color channel to divide a 32 × 16 region into four 8 × 16 regions. Such implementations require four encoded block flag values for each color channel, and therefore, four encoded block flags are encoded in the syntax structure 930 for each color channel in encoded bitstream 312, therefore, the flag is 'CU3', 'CU4', 'CV3' and a 'CV4' encoded block (not illustrated in FIG. 9B) may be introduced into a transform unit (TU) syntax structure 952. Such implementations avoid introducing a 32-point conversion logic into the color processing route and, where 8 × 16 areas are not subdivided, they can reuse the 8 × 16 conversion logic required for 16 × 16 transformation unit (TUs) (in the luminance channel) that require conversion on an 8 × 16 size conversion for color channels.

[0114] FIG. 10 is a schematic flowchart showing a method 1000 for encoding a transform unit (TU) by encoding a non-end vertex syntax syntax structure 902 and a transform tree end syntax 932. Method 1000 is described with reference to a color unit of a transform unit (TU), however, method 1000 can be applied to any color unit of a transform unit (TU). Since the non-end vertex syntax structure of the transform tree 902 and the end tree syntax structure of the transform tree 932 describe one vertex in the transform tree, method 1000 encodes one vertex of the transform tree in encoded bitstream 312. Method 1000 may be implemented in hardware or software, executable on processor 205, for example. The method 1000 is initially called for the top level of the transform tree and is configured to call itself (recursively) to encode the vertices of the descendants of the transform tree. Step 1002 determining the size of the conversion unit determines the size of the conversion unit (TU) in the conversion tree in accordance with the size of the coding unit (CU) that contains the conversion tree, and the conversion depth value of the conversion unit (TU). When method 1000 is invoked at the top level of the transform tree, the transform depth value is set to zero; otherwise, the transform depth value is provided by the parent instance of method 1000. The value of the transform split flag, such as the transform split flag value 702, is encoded in encoded bitstream 312 as a flag 910 conversion splitting if the conversion depth value is less than the maximum allowed conversion depth

[0115] When the value of the conversion split flag is one, the color coded block flags 912 and 914 are encoded for each color channel only if the parent node of the transformation tree hierarchy has a corresponding coded block flag value equal to one. Method 1000 then calls up a new instance of method 1000 for each vertex of the descendant (represented in the encoded bitstream portion 312 by transform tree syntax structures 916, 918, 920, and 922) of the transform tree. Each instance of method 1000 invoked for descendant vertices is provided with a conversion depth value equal to the conversion depth value of the current instance of method 1000 increased by one.

[0116] When the value of the transform split flag is zero, the step 1004 of identifying the maximum number of direct transforms determines the maximum number (n) of transforms for each color channel of the encoded area. When an implied partition takes place, this number n will be equal to one. When a 4: 2: 2 color format is used and they encounter a rectangular region of the color channel, such as an 8 × 16 region 664, and the size of the region corresponds to one of a predefined set of region sizes (such as 16 × 32 and 8 × 16), the implied partition and the maximum number of conversions will be equal to two (otherwise the number of transformations will be equal to one). Otherwise (the size of the region does not correspond to one of the predefined set of sizes of the region), the maximum number of transformations will be equal to one. For example, if 4 × 8 is not one of a predefined set of area sizes, then the maximum number of transformations will be one. When the 4: 4: 4 format is used, the color format and size of the area that is encountered is one of a predefined set of area sizes (such as the 32 × 32 area), an implied splitting takes place and the maximum number of transformations will be four. Otherwise (the area size is not one of a predefined set of area sizes) the maximum number will be equal to one. For example, if 8 × 8 is not one of a predefined set of area sizes, then the maximum number of transformations will be one. Although a predefined set of area sizes includes 8 × 16, another predefined set of area sizes is possible, such as only 16 × 32 when using a 4: 2: 2 color format or 32 × 32 when using 4 : 4: 4 color format.

[0117] For each color channel, if the parent vertex had a coded block flag value equal to one, then for each of n, the coded block flag is encoded in encoded bitstream 312. For example, when the number of conversions is two, coded block flags 942 and 944 indicate for the presence of a transformation for each of the two areas implied by the partition. The direct conversion selection step 1006 selects the direct transformation from a predefined set of direct transformations, for each of the maximum number of transformations, based on the size of the transform unit (TU), which in turn depends on the depth of the transform, and therefore relates to the hierarchical level of the transform unit in the largest coding unit. When the conversion depth is zero, the size of the transformation unit (TU) is equal to the size of the coding unit (CU). For each increment of transform depth, the transform unit size (TU) is halved. For a 32 × 32 coding unit (CU) size, a conversion depth of zero, and using a 4: 2: 2 color format, the conversion unit (TU) size will therefore be 32 × 32, and the conversion size for chroma will therefore be 16 × 32. For example, when the maximum number of transformations is two and the size of the region for chroma is 16 × 32, then a 16 × 16 direct transformation is selected for each of the 16 × 16 regions for chroma obtained as a result of the implicit partition.

[0118] Step 1008 of applying the direct transform performs a direct transform for each of the maximum number of transformations in the corresponding region, which has a flag value of the encoded block equal to one. A step 1008 of encoding arrays of residual chroma samples is typically performed by the transform module 320. This leads to the conversion of each array of residual chroma sampling elements (representations in the spatial domain) into an array of residual chroma coefficients (representations in the frequency domain).

[0119] Step 1010 of encoding the arrays of residual chroma coefficients encodes an array of residual chroma coefficients for each of the maximum number of transform areas of each chroma channel with the flag value of the encoded block equal to one in the encoded bitstream 312. The number of arrays of residual chroma coefficients encoded for a given conversion unit for of a given color channel, depends on the flag value of the encoded block of each transform and, therefore, will vary range from zero to (in most) the maximum number of conversions. For example, when the number of conversions is two and both chroma channels have a coded block flag value equal to one for each of the counter values, then the residual chroma blocks 956, 958, 960 and 962 are encoded in the encoded bitstream. If the flag value of the encoded block for each transform for a given chroma channel is zero, then the residual chroma block is not encoded in the encoded bitstream for the chroma channel. An encoding step 1010 of an array of residual chroma coefficients is typically performed by an entropy encoder 324.

[0120] FIG. 11 is a schematic flowchart showing a method 1100 for decoding a transform unit (TU) by decoding a non-end syntax structure of a transform tree and syntax structure 932 of an end vertex of a transform tree. Method 1100 is described with reference to a color unit of a transform unit (TU), however, method 1100 can be applied to any color unit of a transform unit (TU). Since the syntax structure 902 of the non-end vertex of the transform tree and the syntax structure 932 of the end vertex of the transform tree describes one vertex in the transform tree, method 1100 decodes one vertex of the transform tree from encoded bitstream 312. Method 1100 can be executed in appropriate hardware or alternatively in software, for example, executed by the processor 205. Method 1100 is initially called for the top level of the transformation tree and, configured to, call yourself (recursively) to decode the vertices of the descendants of the transformation tree. The transform unit size (TU) determining step 1102 determines the transform unit (TU) size in a manner identical to the transform unit size determining step 1002. Step 1102 determining the size of the transform unit determines the size of the transform unit (TU) in the transform tree in accordance with the size of the coding unit (CU) that contains the transform tree and the depth value of the transform unit (TU). When method 1100 is invoked at the upper level of the transform tree, the transform depth value is set to zero, otherwise the transform depth value is provided by the parent instance of method 1100. The transform split flag value, such as the transform split flag value 702, is decoded from the encoded bitstream 312 as a conversion splitting flag 910 if the conversion depth value is less than the maximum allowed conversion depth.

[0121] When the value of the conversion split flag is one, the flags of the color coded block 912 and 914 are decoded for each color channel only if the parent node of the transformation tree hierarchy has the corresponding flag value of the coded block equal to one. Method 1100 then calls a new instance of method 1100 for each vertex of the child (represented in the encoded bitstream portion 312 by transform tree syntax structures 916, 918, 920 and 922) of the transform tree. Each instance of method 1100 invoked for descendant vertices is provided with a conversion depth value equal to the conversion depth value of the current instance of method 1100 increased by one.

[0122] When the value of the transform split flag is zero, the step 1104 of identifying the maximum number of inverse transforms determines the (maximum) number (n) transforms for each of the at least one arrays of residual color coefficients present in each color channel of the decoded region in a manner identical to the step 1004 identifying the maximum number (n) of direct conversions. When the implied partition does not take place, the given number n will be equal to one. When a 4: 2: 2 color format is used and they encounter a rectangular region of the color channel, such as an 8 × 16 region 664, and the size of the region corresponds to one of a predefined set of region sizes (such as 16 × 32 and 8 × 16), the implied partition and the maximum number of conversions will be two (otherwise, the number of transformations will be one). Otherwise (the size of the region does not correspond to one of the predefined set of sizes of the region), the maximum number of transformations will be equal to one. For example, if 4 × 8 is not one of a predefined set of area sizes, then the maximum number of transformations will be one. When using a 4: 4: 4 color format and the size of the area that is encountered is one of a predefined set of area sizes (such as a 32 × 32 area), an implied splitting takes place and the maximum number of transformations will be four. Otherwise (the area size is not one of a predefined set of area sizes) the maximum number will be equal to one. For example, if 8 × 8 is not one of a predefined set of area sizes, then the maximum number of transformations will be one. Although the predefined set of area sizes includes 8 × 16, another predefined set of area sizes is possible, such as only 16 × 32 when using a 4: 2: 2 color format, or 32 × 32 when using 4: 4: 4 color format. For each color channel, if the parent vertex had a coded block flag equal to one, then for each of the (n) transforms, the coded block flag is decoded in encoded bitstream 312. For example, when the maximum number of conversions is two, coded block flags 942 and 944 indicate the presence of a transformation for each of the two areas implied by the partition.

[0123] Step 1106 of decoding the arrays of residual chroma coefficients decodes the array of residual chroma coefficients for each of the maximum number of transform areas of each chroma channel from the encoded bitstream 312 with the flag value of the encoded block equal to one. The number of arrays of residual coefficients decoded for a given transform unit for a given chroma channel depends on the flag value of the encoded block of each transform and, therefore, will vary from zero to (in most) ‘the number of (n) transformations’. For example, when the number of conversions is two and both chroma channels have coded block flags equal to one for each counter value, then chroma residual blocks 956, 958, 960 and 962 are decoded from the encoded bitstream 312. Step 1106 of decoding the array of residual chroma coefficients as typically performed by an entropy decoder 420 for each array of residual chroma with a flag value of the encoded block equal to one.

[0124] Step 1108 select the inverse transform then selects the inverse transform from a predefined set of inverse transforms, for each of the maximum number of transformations with the flag value of the encoded block equal to one for each color channel. For example, when the maximum number of transformations is two and the size of the region is 16 × 32, and the flag value of the encoded block for each of the two transformations is one, then the 16 × 16 inverse transform is selected for each of the 16 × 16 regions resulting from the implied partition.

[0125] Step 1110 applying the inverse transform then performs the inverse transform for each of the maximum number of transform regions in the corresponding region with the flag value of the encoded block equal to one. This leads to the conversion of each array of residual chroma (representation in the frequency domain) into an array of residual chroma sampling elements (representation in the spatial domain), which is a decoded video frame. An inverse transform application step 1110 is typically performed by the inverse scaling and transform module 422.

[0126] FIG. 12A shows a diagonal scan pattern 1201, FIG. 12B shows a horizontal scan pattern 1202, and FIG. 12C shows a vertical scan pattern 1203, each for a 4 × 8 conversion unit 1200. These implementations, which scan a 4 × 8 conversion unit 1200 using the illustrated scanning patterns, have the property that residual coefficients are grouped into 4 × 4 blocks, known as ‘subblocks’. The ‘coefficient group’ flag present in the encoded bitstream 312 can therefore be used to indicate, for each subunit, the presence of at least one significant (non-zero) residual coefficient. The application of a 4 × 4 sub-block size for a 4 × 8 transform achieves coherence with the scan pattern present in other transform sizes, where the coefficients are always grouped in sub-blocks.

[0127] Specific implementations may apply a coefficient group flag to signal the presence of at least one non-zero residual coefficient in each subunit. Advantageously, these scanning patterns permit reuse of control software or a digital circuit that processes residual coefficients by reusing subunit processing for all transform sizes. The particular scan pattern used may be selected according to a criterion, such as an intra-prediction direction of a co-located prediction unit (PU). Where the conversion encodes chroma sampling elements over a 4: 2: 2 chroma format sampling element grid, the relationship between the intra prediction direction and the scanning pattern changes, since each chroma sampling element corresponds to a non-square (2 × 1) array of brightness sampling elements, influencing to the 'direction' or angle of the intra prediction mode. The scan is shown in the ‘reverse’ direction in FIG. 12A-12C, ending in a DC coefficient located in the upper left corner of the transform unit (TU). In addition, the scan does not need to start in the lower right corner of the conversion unit (TU). Due to the predominance of non-zero residual coefficients in the upper left region of the transformation unit (TU), scanning can start from the ‘position of the last significant coefficient’ and go in the opposite direction until the upper left coefficient is reached.

[0128] Other implementations may apply a single scan for a given area to encode residual coefficients and then apply more than one transform to these residual coefficients. In this case, only one flag of the encoded block is used for the region and, therefore, for all the transformations covered by the scan pattern. The flag of the coded block is set to unity if at least one significant residual coefficient exists in any of the scans. For example, 4 × 8 scan patterns in FIG. 12A-12C may be used to code the residual coefficients of two 4x4 transforms. Two 4 × 4 arrays of residual coefficients can be concatenated to form a 4 × 8 array suitable for the scan pattern. Since one scan is performed on the array, one position ‘the last significant coefficient’ is encoded in the bitstream for the scan pattern and one flag value of the encoded block is enough for the array. The energy compaction property of the modified discrete cosine transform (DCT) provides an advantage over other schemes, such as alternating the coefficients of each square transform along the scan pattern path in a rectangular coefficient array. This gives the advantage that the density of residual coefficients in each 4 × 4 array of residual coefficients is approximately equalized in the combined 4 × 8 array, providing a higher compression efficiency by the entropy encoder 324, for subsequent decoding by the entropy decoder 420.

[0129] Some color channel color coding implementations may use the first transform to encode residual bins at the locations of chroma bins corresponding to a 4: 2: 0 grid of color bins, and the second transform to encode residual bins at additional locations of chroma applied at a 4: 2: 2 grid of chroma sampling elements, relative to a 4: 2: 0 grid of chroma sampling elements. Such implementations may advantageously use a simplified transform for the second transform, such as a Hadamard transform, wherein the output of the second transform is added (or otherwise combined) with the residual bins for the first transform to create residual bins for the second transform. Advantageously, a preprocessing step that implements a transform, such as a Hadamard transform, can be used to sample a grid of chroma elements for a 4: 2: 2 chroma format into a grid of chroma bins for a 4: 2: 0 chroma format. Such configurations must transmit additional residual coefficients from the preprocessing stage as collateral information, such a residue applied to each largest coding unit (LCU) in the case where the preprocessing transform is applied at the level of the largest coding unit (LCU).

[0130] Implementations with multiple transforms for a given area can use either a single combined scan covering the entire area, or a separate scan for each transformation. If a scan for several transformations is combined into one scan, then only one flag of the encoded block is required for each scanned area. These implementations using one combined scan can achieve higher compression of the residual coefficients by alternating the residual coefficients of each transform, such as coefficient-by-coefficient interleaving, in order to co-locate the residual coefficients from each transform with similar spectral properties.

[0131] FIG. 13 is a schematic flowchart showing a conversion unit encoding method 1300. The method 1300 performed by video encoder 114 encodes a luminance channel and a color channel of a transform unit. In step 1302 of determining the value of the luminance conversion skip flag, the conversion skipping control module 346 determines the value of the conversion skipping flag, such as the conversion skipping flag 964 or 9127, for the luminance channel, as a rule, by testing the costs of encoding an array of 360 residual bins as in spatial area (skipping conversion is performed), and the frequency domain (skipping conversion is not performed). In step 1304 of determining the value of the color conversion skip flag, the conversion skipping control module 346 determines or otherwise sets the value of the conversion skipping flag, such as the conversion skipping flag 966 or 9148, for one of the color channels that will be applied to all subregions resulting from the implied partitions, and belonging to the same color channel. Transmission skip control module 346 can apply logic similar to that applied to the luminance channel, however, the determination of the bit rate costs should be considered for each of the arrays of residual color sampling elements obtained from the implicit partition when determining the costs or to perform the conversion skipping for all arrays of residual chroma sampling elements in the chroma channel (or 'color channel'), or to perform conversion skipping for none of the ma Sivov residual chroma sampling elements in the chrominance channel. The step 1304 for determining the value of the color conversion skipping flag is repeated for each color channel, determining the values of the conversion skipping flag for other color channels, such as the conversion skipping flag 968 or 9151. Luminance conversion and chroma conversion encoding step 1306 encodes an array of residual luma sampling elements in the encoded bitstream 312 using an entropy encoder 324 and encodes arrays of residual chroma sampling elements for a color channel in the encoded bitstream 312 using an entropy encoder 324. An array of luma residual sampling elements determined in accordance with the luminance conversion skip flag, or by conversion in the mass conversion module 320 iva residual sample elements in an array of residual coefficients, or by performing bypass module 320 conversion when conversion is performed pass video encoder 114. Subsequently, the residual array 363 is sent to the scaling and quantization module 322 to create the residual data array 364. When at least one of the values in the residual data array 364 is non-zero, the values of the residual data array 364 are encoded in the encoded bitstream 312 by an entropy encoder 324 (in the residual data block, such as residual data block 954, 956, 958, 960, or 962 residual data ) and the corresponding flag of the encoded block is set to unity. Arrays of residual chroma sampling elements are defined similarly to arrays of residual chroma sampling elements, except that arrays of residual chroma sampling elements other than the first share the conversion skipping flag with the first array of chroma residual sampling elements. Step 1306 of encoding arrays of residual chroma sampling elements is repeated for each chroma channel.

[0132] FIG. 14 is a schematic flowchart showing a method 400 for decoding a transform unit. The method 1400 performed by video decoder 134 decodes a luminance channel and a color channel of a transform unit. Step 1402 for determining the luminance skipping flag value determines the conversion skipping flag for the luminance channel by decoding the conversion skipping flag, such as conversion skipping flag 964 or 9127, from the encoded bitstream 312 using the entropy decoder 420. Step 1404 for determining the color conversion skipping flag value determines the value of the conversion skipping flag for one of the arrays of residual chroma sampling elements in the chroma channel, which lzhno be applied to all arrays residual chroma sampling the chrominance components in the channel and the same conversion unit (TU). Step 1404 decodes a transform skip flag, such as transform skip flag 966 or 9148, from encoded bitstream 312, using an entropy decoder 420. Implementations that associate the transform skip flag with the first array of residual chroma chips avoid the need for buffering earlier arrays residual sampling elements before defining the conversion skip flag from a later array of residual coefficients (which will then be used to continue I processing of an earlier array of residual bins, thereby introducing internal buffering). Step 1404 may also determine a conversion skip flag for additional color channels, such as by decoding a conversion skip flag 968 or 9151 from encoded bitstream 312 using an entropy decoder 420. The brightness conversion and color conversion decoding step 1406 instructs the entropy decoder 420 to decode an array of residual coefficients luminance, such as residual luminance data block 954, when the corresponding flag of the encoded block is one, such as a flag 950 of the encoded block, and arrays of residual chroma coefficients associated with a particular chroma channel, such as arrays 956 and 958 of residual chroma, when each corresponding flag of the encoded block, such as flags of the encoded block 942 and 944, is equal to one. When decoding the luminance conversion, the array of residual luminance coefficients is sent only through the inverse transform module 422 if a skipping conversion is not performed, otherwise the array of residual luminance coefficients bypasses the inverse transform module 422. When decoding a chroma conversion, for each array of residual chroma sampling elements in a transform unit, a transform skip flag is present in the encoded bitstream 312 and associated with the first array of residual luma sampling elements.

[0133] The description of methods 1300 and 1400 relates to a преобразования conversion unit ’, which may contain several arrays of residual chroma sampling elements for a given chroma channel when an implied partition occurs. This is consistent with syntax structure 930. When syntax structures 9100, 9130, and 9160 are used, each color region obtained from the implied partition is illustrated as a separate transform unit (TU), labeled as color transform unit (CTU) in FIG. 9C, 9D and 9E. For the purposes of methods 1300 and 1400, color conversion units (CTUs) are merely an artifact of using the transform tree syntax structure 9100 to partition color areas. In FIG. 9C, the spatial area occupied by the luminance conversion unit (LTU) 9126 may be considered a преобразования conversion unit ’because it occupies exactly the same spatial area as the conversion unit 952. The color conversion units (CTUs) 9116 9118 and 9120-9122 (if present) can be considered color sub-regions derived from an implied partition.

[0134] Advantageously, both methods 1300 and 1400 result in a single conversion skip flag encoded for each color channel, regardless of the presence or absence of an implied splitting operation (which can be applied when 4: 2: 2 and 4: 4: 4 are used color formats). This characteristic leads to a uniform character with a 4: 2: 0 color format, where one conversion skip flag is present for each array of residual coefficients, and only one array of residual coefficients is present for each color channel in relation to a given conversion unit. For example, an 8 × 8 4: 2: 0 conversion unit will have 8 × 8 conversion for luminance and 4 × 4 color conversion for each color channel. One conversion skip flag will be present for each color channel in this case. In the case of 4: 2: 2, with implied splitting, two 4x4 color transformations will be present in each color channel. The conversion skip flag encoded with the first 4 × 4 color conversion, but applied to both 4 × 4 color conversions, will control the conversion skip status for the same spatial area as in the case of 4: 2: 0. This uniform nature leads to processing of the conversion skip for 4: 2: 2, which is backward compatible with the 4: 2: 0 case (i.e., there is no rearrangement of the syntax elements in 4: 2: 0 due to the support of the conversion skip to 4 : 2: 2). Having a common conversion gap for chroma results in an implied split that avoids artificially dividing the conversion unit into the upper half and lower half in order to indicate the conversion gap.

[0135] FIG. 15 is a schematic view showing possible arrangements of 4 × 4 transformations in a 4 × 4 and 8 × 8 transform unit, for a video encoder 114 and a video decoder 134. Color channels Y, U, and V are shown in FIG. 15 in columns, and three cases are shown along the rows. In all the cases depicted, video encoder 114 and video decoder 134 are configured to use a 4: 2: 2 color format. Also, in all cases, the video encoder 114 and the video decoder 134 support the implicit splitting of the 4 × 8 chroma region into two 4 × 4 chroma regions, and therefore, two 4 × 4 color transforms are shown for each color channel. The three cases depicted are:

case 1: 8 × 8 conversion unit (TU) (top row);

case 2: four 4 × 4 transformation units (TU) with the first ordering (order 1) of transformations (middle row); and

case 3: four 4 × 4 transform units (TU) with second ordering (order 2) of transformations (bottom line).

[0136] For each case, the transforms are numbered in the order in which they appear in the encoded bitstream 312. Case 1 shows a transform unit (TU) with 8 × 8 brightness conversion and two 4 × 4 transformations, for each color channel. The luminance conversion does not have a skip conversion flag because the luminance conversion is 8 × 8. Case 2 and 3 further illustrate the case where four 4 × 4 conversion units result in luminance regions for each color conversion that apply to multiple conversion units (TUs). In Cases 2 and 3, four conversion units (TUs) are numbered from zero to three and indexed using the ‘blkIdx’ variable, as used in the emerging high-performance video coding (HEVC) standard. For each transformation depicted in FIG. 15, if a conversion skip is supported, the frame is included in the upper left corner of the transformation. For conversions in which the conversion skip flag is always explicitly encoded, the frame is hatched (such as hatched frame 1502). An unshaded frame (such as an unshaded frame 1504) illustrates a case where the transform skip flag for the present transform is extracted from an earlier one (such as the transform on top). Implementations that do not support this extraction will explicitly encode the conversion skip flag in encoded bitstream 312 for unshaded frames transforms. In Case 2 and Case 3, a conversion unit syntax structure, such as conversion unit syntax structure 952, is called four times (with a value for ‘blkIdx’ increasing from zero to three), once for each 4x4 conversion unit. Thus, four instances of the transform unit syntax structure are present in the encoded bitstream 312. On each call, a residual luminance block, such as a residual luminance data block 954, is present in the encoded bitstream 312 if the corresponding flag of the encoded block, such as the encoded flag 950 block, has a value equal to one. In Case 2, on the fourth call ('blkIdx' is three), residual chroma blocks for chroma channels, such as chroma residual blocks 956, 958, 960, 962, are encoded in encoded bitstream 312 (if the corresponding flags of the encoded block, such as flags 942, 944, 946, 948 of the encoded block have a value of one). The ordering of the residual luma and chroma blocks of FIG. 9B corresponds to the ordering of the transformations present in Case 2. In Case 3, the ordering changes due to the following: Residual chroma blocks for the upper half (such as residual chroma 956, 960) are processed on the second call of the transform unit syntax structure (i.e., when 'blkIdx' is one), and residual chroma blocks for the lower half (such as chroma residual blocks 958, 962) are processed on the fourth call of the transform unit syntax structure (i.e., when 'blkIdx' is three).

[0137] Another case not illustrated in FIG. 15, consists of that of a 4 × 4 conversion unit, when a 4: 2: 0 color format is used, one 4 × 4 conversion for color is applied to the area along the color sampling grid, which corresponds to four 4 × 4 conversion units for brightness at the same hierarchical level of the quad tree (together occupying an 8 × 8 region along the grid of brightness discretization elements). When a 4 × 8 conversion is available in chroma, the conversion skip for the 4: 2: 2 case applies to the 4 × 8 conversion (in addition to the 4 × 4 conversion), as described with reference to FIG. 18 below. When a 4 × 8 transform is not available in chroma and a 4: 2: 2 chroma format is used, implementations must use two 4 × 4 transforms for each color channel and can encode a transform skip flag for one 4 × 4 transform, such as the top 4 × 4 transform, but apply the encoded transform skip flag for both 4x4 transforms for a given color channel.

[0138] FIG. 17 is a schematic flowchart showing a method 1700 for decoding residual data for a transform unit (TU), developing aspects of the method 1400 of FIG. 14. Method 1700 determines a conversion skip flag for a given area and decodes residual data for the area. When method 1700 is invoked for a luminance channel of a transform unit (TU), there is only one area. For a single color channel of a transform unit (TU), and when an implied partition occurs, two areas are present and a method 1700 is called for each area with a flag value of the encoded block equal to one. The method 1700 begins with step 1702 testing supported conversion skip. Step 1702 tests the enabled transform skip flag and the coding unit transform quantization bypass flag and the transform size for the present domain. The enable conversion skip flag encoded in encoded bitstream 312 indicates whether the conversion skip function is available in encoded bitstream 312. The encoding unit transform quantization bypass flag, encoded in encoded bitstream 312, indicates whether the 'lossless' encoding mode has been selected by the video encoder 114, by which both the transform module 320 and the quantization module 322 are bypassed, and therefore video encoder 114 operates in a lossless mode, allowing video decoder 134 to accurately reproduce captured frame data from video source 112. The transform size for the real area, indicated by the variable ‘log2TrafoSize’ in the emerging High-Performance Video Coding (HEVC) standard, which is defined as log2 of the side dimension of the square transform. When the conversion skip flag matches true (i.e., enabled) and the coding unit conversion quantization bypass flag matches false (i.e., not enabled) and the conversion size is 4 × 4 (i.e., log2TrafoSize is 2), control proceeds to step 1704 of testing the first area with the true flag of the encoded block (CBF) in the color channel, otherwise, control proceeds to step 1712 of decoding the residual data. Testing step 1704 determines whether the present area is the first area in the color channel (and in the transform unit (TU) with the coded block flag value (CBF) equal to one). Since the method 1700 is called only if the flag value of the encoded block for this area is equal to one, two cases are possible. If method 1700 is invoked for the first chroma region (upper region when a 4: 2: 2 chroma format is used, for example, region 682 or 666 in FIG. 6C) of the implied partition, then test block 1704 gives an estimate corresponding to true and control proceeds to step 1706 decoding the flag skip conversion. If method 1700 is invoked for the subsequent color region (s) of an implied partition (lower region when a 4: 2: 2 color format is used, for example, region 684 or 668 in Fig. 6C), test step 1704 gives an estimate corresponding to false when method 1700 was previously called for the first chroma region (in relation to the present conversion unit), and the corresponding value is true when method 1700 was not previously called for the first chroma region (in relation to the present conversion unit). When test step 1704 provides an estimate corresponding to true, control proceeds to step 1706 of decoding the skip conversion flag. At step 1706, the entropy decoder 420 decodes the conversion skip flag from the encoded bitstream 312 to determine the value of the conversion skip flag. Step 1708 storing the value of the conversion skip flag stores the value of the conversion skip flag in memory, such as hardware registers or registers 246, for use later in subsequent calls to method 1700. If test step 1704 gives an estimate corresponding to false, control proceeds to retrieve step 1710 conversion skipping flag, where the conversion skipping flag value defined and stored in the previous call to method 1700 is retrieved from memory, such as hardware registers software or registers 246. In a residual data decoding step 1712, a residual data block, such as residual data block 954, 956, 958, 960 or 962, is decoded from encoded bitstream 312 by an entropy decoder 420. The determined value of the conversion skip flag is sent as the value 468 conversion skipping flag to control the conversion skipping operation, as described above with reference to multiplexer 423. Steps 1702-1710 correspond to step 1402 of FIG. 14, when method 1700 is invoked for the luminance channel, and steps 1702-1710 correspond to step 1404 of FIG. 14, when method 1700 is invoked for a color channel. The residual data decoding step 1712 corresponds to decoding the brightness remainder of step 1406 of FIG. 14 and decoding the color residue of step 1406 of FIG. 14. The 1700 method also follows the syntax structure of ‘residual coding’ as defined in the emerging high-performance video coding (HEVC) standard.

[0139] FIG. 18 is a schematic view 1800 showing a skip conversion operation applied to a 4 × 8 color region (with 4 × 8 non-square transform) for each color channel. In FIG. 18 shows a luminance channel (‘Y’) and each color channel (‘U’ and ‘V’). In FIG. 18 shows two cases:

[0140] Case 1: The '8 × 8 TU' (top row in Fig. 18) represents an 8 × 8 transform unit (TU), with an 8 × 8 transform 1802 for a luminance channel and a 4 × 8 (non-square or rectangular) transform 1804 for each color channel. The conversion skipping flag is shown with a hatched frame in the upper right corner of the conversion for which the conversion skipping operation is supported. In this case, the skip conversion operation is also supported in the 4 × 8 transform case (in addition to the 4 × 4 transform case) and, therefore, each of the 4 × 8 transforms includes a transform skip flag 1806, as illustrated in FIG. eighteen.

[0141] Case 2: 'Four 4 × 4 TUs' (bottom row in Fig. 18) depicts four 4 × 4 transform units (TU), with four 4 × 4 transforms 1808 for the luminance channel and 4 × 8 (not square or rectangular) conversion 1810 for each color channel. The 4 × 8 transform for each chroma channel is co-located (along the grid of chroma sampling elements) with the luminance conversion (along the grid of chroma sampling elements) and shared among the four 4 × 4 transformation units (TU). In this implementation, the conversion skipping operation is also supported in the 4 × 8 conversion case (in addition to the 4 × 4 conversion case) and, therefore, the 4 × 8 conversion includes a conversion skipping flag 1812, as illustrated in FIG. eighteen.

[0142] With respect to an implementation supporting Cases 1 and 2 of FIG. 18, the modified test step 1702 and steps 1706 and 1712 of method 1700 are performed by video decoder 134. The modified testing step 1702 functions as the testing step 1702 of FIG. 17, except that the 4 × 8 transform size is included (in addition to the 4 × 4 transform size) as a possible transform size for which the skip conversion operation is supported, thereby allowing the modified test step 1702 to provide an estimate corresponding to true as in the case of 4 × 4, and in the case of 4 × 8 transformation.

[0143] Appendix A illustrates a possible ‘text’ in relation to a high-performance coding standard (HEVC) being developed that corresponds to syntax structure 900 and syntax structure 930. Each instance of the transform_tree () function in Appendix A is depicted as a portion of the syntax structure marked ‘TT’ in FIG. 9A and 9C, and each instance of the transform_unit () function in Appendix A is depicted as a portion of the syntax structure marked ‘TU’ in FIG. 9A and 9B. The text provided in Appendix A is one example of a text that is consistent with syntax structures 900 and 930 and other examples are possible. Text that is consistent with syntax structures 900 and 930 assumes that video encoder 114 performs method 1000 for encoding a bitstream, and video decoder 134 performs method 1100 for decoding a bitstream.

[0144] Appendix B illustrates possible text in relation to a high-performance video coding standard (HEVC) being developed that corresponds to syntax structure 9100 and syntax structure 9130. Each instance of the transform_tree () function in Appendix B is depicted as a portion of the syntax structure marked ‘TT’ in FIG. 9C, 9D, and 9E, and each instance of the transform_unit () function in Appendix B is depicted as a portion of the syntax structure marked ‘TU’ in FIG. 9C, 9D and 9E. The text provided in Appendix B is one example of text that is consistent with syntax structures 9100 and 9130 and other examples are possible. Text that is consistent with syntax structures 9100 and 9130 also assumes that video encoder 114 performs method 1000 for encoding a bitstream, and video decoder 134 performs method 1100 for decoding a bitstream.

[0145] The text in Appendix A and Appendix B leads to an implementation whereby a 32 × 32 color region encountered in a 32 × 32 size conversion unit (TU) configured for a 4: 4: 4 color format results in applicable ( the maximum number) of four 16 × 16 color conversions, and the 16 × 32 color region encountered in a 32 × 32 transform unit (TU) configured for a 4: 2: 2 color format results in two (16) applied × 16 color transforms. An implementation derived from the text in Appendix A and Appendix B, when applied to smaller conversion units (TUs) and configured for a 4: 2: 2 color format, applies (maximum) one color conversion. For example, an 8 × 16 transform is applied to an 8 × 16 chroma region and a 4 × 8 transform is applied to a 4 × 8 chroma region.

Industrial applicability

[0146] The described arrangements are applicable in the computer and data processing industries, and in particular for digital signal processing as applied to encoding and decoding signals, such as video signals.

[0147] The foregoing describes only certain embodiments of the present invention, and modifications and / or changes may be made therein without departing from the scope and spirit of the invention, the embodiments being illustrative and not limiting.

[0148] (Australia only) In the context of this technical description, the word “comprising” means “including mainly, but not necessarily exclusively” or “possessing” or “including”, and not “consisting only of”. Variations of the word “comprising”, such as “contain” and “contains”, have correspondingly changed meanings.

APPENDIX A

TRANSFORM_TREE () AND TRANSFORM_UNIT () IMPLEMENT AN IMPLIED DIVISION OF COLOR USING THE CYCLE OPERATOR

7.3.11 Transformation tree syntax

7.3.12 Syntax of conversion unit

7.4.8.1 General semantics of the coding unit

The variables TrafoCrCbHorCnt and TrafoCrCbVertCnt are retrieved as follows:

- If log2TrafoSize is 5 and split_transform_flag is 0, TransformIdxMax is retrieved as follows:

- If chroma_format_idc is 1, TrafoCrCbHorCnt and TrafoCrCbVertCnt are 1.

- If chroma_format_idc is 2, TrafoCrCbHorCnt is 1, and TrafoCrCbVertCnt is 2.

- Otherwise, if chroma_format_idc is 3, TrafoCrCbHorCnt and TrafoCrCbVertCnt are 2.

- Otherwise, TrafoCrCbHorCnt and TrafoCrCbVertCnt are 1.

The variable TrafoCrCbCnt is retrieved as TrafoCrCbHorCnt * TrafoCrCbVertCnt.

The variables log2CrCbTrafoHorSize and log2CrCbTrafoVertSize are retrieved as follows:

- If chroma_format_idc is 1, log2CrCbTrafoHorSize and log2CrCbTrafoVertSize are log2TrafoSize - 1.

- Otherwise, if chroma_format_idc is 2, log2CrCbTrafoHorSize is log2TrafoSize and log2CrCbTrafoVertSize is min (log2TrafoSize - 1, 4).

- Otherwise, if chroma_format_idc is 3, log2CrCbTrafoHorSize and log2CrCbTrafoVertSize are min (log2TrafoSize, 4).

End of Appendix A

APPENDIX B

TRANSFORM_TREE () is called ONCE ONE PAIR OF COLOR CHANNELS FOR EVERY COLOR CONVERSION OBTAINED FROM AN IMPORTANT DIVISION

7.3.11 Transformation tree syntax

7.3.12 Syntax of conversion unit

7.4.8.1 General semantics of the coding unit

The variables TrafoCrCbHorCnt and TrafoCrCbVertCnt are retrieved as follows:

- If log2TrafoSize is 5 and split_transform_flag is 0, TransformIdxMax is retrieved as follows:

- If chroma_format_idc is 1, TrafoCrCbHorCnt and TrafoCrCbVertCnt are 1.

- If chroma_format_idc is 2, TrafoCrCbHorCnt is 1, and TrafoCrCbVertCnt is 2.

- Otherwise, if chroma_format_idc is 3, TrafoCrCbHorCnt and TrafoCrCbVertCnt are 2.

- Otherwise, TrafoCrCbHorCnt and TrafoCrCbVertCnt are 1.

The variable TrafoCrCbCnt is retrieved as TrafoCrCbHorCnt * TrafoCrCbVertCnt.

End of Appendix B

Claims (22)

1. A method of converting multiple arrays of residual coefficients from a video bitstream configured for a 4: 2: 2 format to a plurality of arrays of residual sampling elements, the method comprising the steps of: decode four arrays of residual luminance factors for the luminance channel, wherein each of these four arrays of residual luminance coefficients corresponds to one 4x4 luminance block of four 4x4 luminance blocks, and these four 4x4 luminance blocks together occupy an 8x8 luminance region; decode, after the four arrays of residual luminance coefficients are decoded, two arrays of residual chroma for the first color channel, wherein each of these two arrays of residual chroma for the first chroma channel corresponds to a 4x4 chroma block and each 4x4 chroma block for the first chroma channel co-located with two of the four 4x4 brightness units; decode, after the two arrays of residual chroma coefficients for the first chroma channel are decoded, two arrays of residual chroma coefficients for the second chroma channel, wherein each of these two arrays of residual chroma coefficients for the second chroma channel corresponds to a 4x4 chroma block and each chroma block for a second color channel is co-located with two of the four 4x4 luma blocks; apply a conversion to each of the decoded four arrays of residual luma coefficients for the luminance channel, the decoded two arrays of residual luma coefficients for the first chroma channel and the decoded two arrays of residual luma coefficients for the second chroma channel to create residual luma sampling elements for the luma channel, residual chroma sampling elements the first color channel and the residual color discretization elements of the second color channel. 2. A method for converting a plurality of arrays of residual sampling elements into a plurality of arrays of residual coefficients of a video bitstream configured for a 4: 2: 2 format, the method comprising the steps of: applying the transform to a plurality of arrays of residual sampling elements to create four arrays of residual luminance coefficients for a luminance channel, two arrays of residual chroma for a first color channel, and two arrays of residual chroma for a second chroma channel; encode four arrays of residual brightness coefficients, wherein each of these four arrays of residual brightness coefficients corresponds to one 4x4 brightness block of four 4x4 brightness blocks, and these four 4x4 brightness blocks together occupy an 8x8 brightness region; encode after the four arrays of residual luminance coefficients are encoded, two arrays of residual chroma for the first color channel, wherein each of these two arrays of residual chroma for the first channel corresponds to a 4x4 chroma block and each 4x4 chroma block for the first chroma channel together with two blocks of brightness 4x4; encode after said two arrays of residual chroma coefficients for the first chroma channel are encoded, two arrays of residual chroma coefficients for the second chroma channel, wherein each of these arrays of residual chroma coefficients corresponds to a 4x4 chroma block and each of these two chroma blocks for the second channel color is located in conjunction with two blocks of brightness 4x4. 3. A decoder for converting a plurality of arrays of residual coefficients from a video bitstream configured for a 4: 2: 2 format into a plurality of arrays of residual sampling elements, the decoder comprising: a first decoder for decoding four arrays of residual luminance coefficients for the luminance channel, wherein each of these four arrays of residual luminance coefficients corresponds to one 4x4 luminance block of four 4x4 luminance blocks, wherein a plurality of 4x4 luminance blocks together occupy an 8x8 luminance region; the second decoder for decoding, after the four arrays of residual luminance coefficients are decoded, two arrays of residual coefficients for the first chroma channel, wherein each of these two arrays of residual chroma coefficients corresponds to a 4x4 chroma block and each 4x4 chroma block for the first chroma channel with two of the four blocks of brightness; a third decoder for decoding, after said two arrays of residual chroma coefficients for the first chroma channel are decoded, two arrays of residual chroma coefficients for the second chroma channel, wherein each of these two arrays of residual chroma coefficients corresponds to a 4x4 chroma block and each chroma block for the second the color channel is located in conjunction with two of the four four 4x4 brightness units; and a conversion unit for applying a conversion to each of the decoded four arrays of residual luma coefficients, the decoded two arrays of residual chroma for the first color channel, and the decoded two arrays of residual chroma for the second color channel to create residual luma sampling elements for the luma channel, residual chroma the first color channel and the residual color discretization elements of the second color channel awns. 4. An encoder for converting multiple arrays of residual sampling elements into multiple arrays of residual coefficients of a video bitstream configured for a 4: 2: 2 format, the encoder comprising: a transform unit for applying the transform to the plurality of arrays of residual sampling elements to create each of the plurality of arrays of residual luminance coefficients for the luminance channel, two arrays of residual chroma for the first color channel, and two arrays of residual chroma for the second chroma channel; a first encoder for encoding four arrays of residual brightness coefficients, wherein each array of residual brightness coefficients corresponds to one 4x4 brightness block of four 4x4 brightness blocks, and these four 4x4 brightness blocks together occupy an 8x8 brightness region; a second encoder for encoding, after the four arrays of residual luminance coefficients are encoded, two arrays of residual chromaticity coefficients for the first chroma channel, wherein each of these two arrays of residual chroma coefficients corresponds to a 4x4 chroma block and each 4x4 chroma block for the first chroma channel located with two of the four 4x4 brightness units mentioned; and a third encoder for encoding, after said two arrays of residual chroma coefficients for the first chroma channel are encoded, two arrays of residual chroma coefficients for the second chroma channel, wherein each of these two arrays of residual chroma coefficients corresponds to a 4x4 chroma block and each chroma block for the second The color channel is co-located with two of the four 4x4 luma blocks. 5. A long-term computer-readable storage medium with a program recorded on it, while the program, when executed by the processor, performs the method of claim 1. 6. A long-term computer-readable storage medium with a program recorded on it, while the program, when executed by the processor, performs the method of claim 2.

Images