JP2023154047A - Method, apparatus and system for encoding and decoding tree or blocks of video samples - Google Patents

Abstract

To provide a system and method of decoding a transform block for a color channel of an image frame from a video bitstream.SOLUTION: A method comprises: determining a chroma format of an image frame, the chroma format having chroma channels of the image frame being subsampled relative to a luma channel of the image frame; determining a coefficient group size of the transform block, the coefficient group size being determined based only on a transform block size of a largest area of the transform block of up to 16 samples, and independent of a color plane of the transform block and color plane subsampling due to the determined chroma format; and decoding the transform block using coefficient groups of the determined size from the video bitstream.SELECTED DRAWING: None

Description

REFERENCE TO RELATED APPLICATIONS This application is filed under Australian patent application no. S. C § 119 and is incorporated herein by reference as if fully set forth herein.

The present invention relates generally to digital video signal processing and, more particularly, to methods, apparatus, and systems for encoding and decoding trees or blocks of video samples. The invention also relates to a computer program product comprising a computer readable medium having recorded thereon a computer program for encoding and decoding trees or blocks of video samples.

Many applications currently exist for video encoding, including applications for the transmission and storage of video data. A number of video coding standards have also been developed, and others are currently under development. Recent developments in video coding standardization have led to the formation of a group called the "Joint Video Experts Team" (JVET). The Joint Video Experts Team (JVET) is a member of Study Group 16, Question 6 (SG16/Q6) of the Telecommunications Standardization Sector (ITU-T) of the International Telecommunication Union (ITU), also known as the "Video Coding Experts Group" (VCEG). Members of the International Organization for Standardization/International Electrotechnical Commission Joint Technical Committee 1/Subcommittee 29/Working Group 11 (ISO/IEC JTC1/SC29/WG11), also known as the "Moving Picture Experts group" (MPEG) Including members of.

The Joint Video Experts Team (JVET) analyzed the responses and issued a Call for Proposals (CfP) at its 10th meeting in San Diego, USA. The submitted responses demonstrated video compression capabilities that significantly exceed those of the current state-of-the-art video compression standard, namely "High Efficiency Video Coding" (HEVC). Based on this outperformance, it was decided to launch a project to develop a new video compression standard named "versatile video coding" (VVC). VVC in particular continues to grow as video formats increase in capacity (e.g., with higher resolutions and higher frame rates) and address increasing market demand for service delivery over WANs where bandwidth costs are relatively high. It is expected to address the continuing demand for high compression performance. At the same time, VVC must be implementable in modern silicon processes and have an acceptable balance between achieved performance versus implementation cost (e.g. in terms of silicon area, CPU processor load, memory usage, and bandwidth). A trade-off must be offered.

Video data includes a sequence of frames of image data, each frame including one or more color channels. Generally, one primary color channel and two secondary color channels are required. Primary color channels are commonly referred to as "luma" channels, and secondary color channels are commonly referred to as "chroma" channels. Video data is typically displayed in an RGB (red-green-blue) color space, which has a high degree of correlation between its three respective components. Video data representations seen by encoders or decoders often use a color space such as YCbCr. YCbCr concentrates luminance mapped into "luma" according to a transfer function into the Y (primary) channel and chroma into the Cb and Cr (secondary) channels. Additionally, the Cb and Cr channels are spatially sampled (subsampled) at a lower rate, e.g. half horizontally and half vertically, compared to the luma channel, known as the "4:2:0 chroma format". may be done. The 4:2:0 chroma format is commonly used in "consumer" applications such as Internet video streaming, broadcast television, and storage on Blu-Ray ^TM discs. Subsampling the Cb and Cr channels at half the rate horizontally and not vertically is known as the "4:2:2 chroma format." The 4:2:2 chroma format is typically used in professional applications including capturing video for film production and the like. The higher sampling rate of the 4:2:2 chroma format makes the resulting video more resilient to editing operations such as color grading. Prior to distribution to consumers, 4:2:2 chroma format material is often converted to 4:2:0 chroma format and then encoded for distribution to consumers. In addition to chroma format, videos are also characterized by resolution and frame rate. Example resolutions are Ultra High Definition (UHD) with a resolution of 3840x2160, or "8K" with a resolution of 7680x4320, and example frame rates are 60 or 120Hz. The luma sample rate may range from about 500 megasamples/second to several gigasamples/second. For the 4:2:0 chroma format, the sample rate of each chroma channel is one-fourth of the luma sample rate, and for the 4:2:2 chroma format, the sample rate of each chroma channel is the luma sample rate. It is half of

The VVC standard is a "block-based" codec in which a frame is first divided into square arrays of regions known as "coding tree units" (CTUs). A CTU typically occupies a relatively large area, such as 128x128 luma samples. However, the right and bottom CTUs of each frame may have a smaller area. Associated with each CTU is a "coding tree" for the luma channel and an additional coding tree for the chroma channel. A coding tree defines a decomposition of a region of a CTU into a series of blocks, also called "coding blocks" (CBs). It is also possible that a single coding tree specifies blocks for both luma and chroma channels, in which case the set of co-located coding blocks is called a "coding unit" (CU); That is, each CU has a coded block for each color channel. CBs are processed for encoding or decoding in a particular order. As a result of the use of the 4:2:0 chroma format, a CTU with a luma encoding tree for a 128x128 luma sample area has a 64x64 chroma sample area co-located with a 128x128 luma sample area. has a corresponding chroma encoding tree for . When a single coding tree is used for the luma and chroma channels, the collection of collocated blocks for a given area is generally referred to as a "unit", e.g., the CU mentioned above, and a "prediction unit". ” (PU), and “transformation unit” (TU). If separate coding trees are used for a given area, the CBs mentioned above are used, as well as "prediction blocks" (PBs) and "transform blocks" (TBs).

Despite the above distinction between "unit" and "block", the term "block" may also be used as a general term for an area or region of a frame in which an operation is applied to all color channels. good.

For each CU, a prediction unit (PU) of the content (sample values) of the corresponding region of frame data is generated (“prediction unit”). Furthermore, a representation of the difference (or "residual" in the spatial domain) between the prediction and the content of the domain seen at the input to the encoder is formed. The differences for each color channel may be transformed and encoded as a sequence of residual coefficients to form one or more TUs for a given CU. The applied transform may be a discrete cosine transform (DCT) or other transform applied to each block of residual values. This transformation is applied separably, ie the two-dimensional transformation is performed in two passes. The block is first transformed by applying a one-dimensional transformation to each row of samples within the block. The partial results are then transformed by applying a one-dimensional transform to each column of the partial results to produce a final block of transform coefficients that substantially decorrelate the residual samples. Conversions of various sizes are supported by the VVC standard, including conversion of rectangular shaped blocks, each side dimension being a power of two. The transform coefficients are quantized for entropy encoding into the bitstream.

When spatial prediction (“intra prediction”) is used to generate a PB, a set of reference samples is used to generate predicted samples for the current PB. The reference samples include samples adjacent to the PB that have already been "reconstructed" (addition of residual samples to intra-predicted samples). These adjacent samples form rows above the PB and columns to the left of the PB. Rows and columns also extend beyond the PB boundaries and include additional neighboring samples. Due to the scanning of blocks in Z-scanning, some of the reference samples have been reconstructed in the previous block. The use of samples from the previous block introduces feedback dependencies that limit the throughput of the block through the video encoder or decoder. Furthermore, when relatively small blocks are predicted from other frames ("inter prediction"), the Memory bandwidth can become excessive.

It is an object of the invention to substantially overcome or at least ameliorate one or more disadvantages of existing arrangements.

One aspect of the present disclosure is a method for decoding a transform block of color channels of an image frame from a video bitstream, the method comprising: determining a chroma format of the image frame; Having the chroma channels of the image frame being sampled and determining the coefficient group size of the transform block, the coefficient group size is the maximum area of the transform block up to 16 samples, and the coefficient group size is the transform block size only. coefficient groups of the determined size from the video bitstream, independent of both (i) the color plane of the transform block and (ii) color plane subsampling according to the determined chroma format. decoding a transform block using a transform block.

According to other aspects, a single table is used for transform blocks belonging to luma and chroma color planes of image frames of a bitstream.

According to other aspects, a coefficient group size is selected to have an aspect ratio closest to 1:1 within the transform block width and height constraints.

Another aspect of the disclosure is a non-transitory computer-readable medium having a computer program stored thereon for implementing a method of decoding transform blocks of color channels of image frames from a video bitstream, the program comprising: Code for determining a chroma format, where the chroma format has the chroma channels of the image frame subsampled with respect to the luma channels of the image frame, and a code for determining the coefficient group size of the transform block, The size is the maximum area of the transform block up to 16 samples, and the coefficient group size is determined based only on the transform block size, and (i) the color plane of the transform block and (ii) the color plane according to the determined chroma format. and a code for decoding a transform block from a video bitstream using coefficient groups of a determined size. provide.

Another aspect of the disclosure is a video decoder that receives a transform block for a color channel of an image frame from a video bitstream, and determines a chroma format of the image frame, the chroma format for a luma channel of the image frame. Have the chroma channel of the image frame to be subsampled and determine the coefficient group size of the transform block, where the coefficient group size is the maximum area of the transform block up to 16 samples, and the coefficient group size is based only on the transform block size. using coefficient groups of the determined size from the video bitstream, independent of both (i) the color plane of the transform block and (ii) color plane subsampling according to the determined chroma format. A video decoder is provided, wherein the video decoder is configured to decode a transform block.

Another aspect of the disclosure is a system having a memory and a processor, wherein the processor is configured to perform a method for decoding a transform block of color channels of an image frame from a video bitstream. The method is configured to execute code stored in the memory of the image frame, the method comprising: determining a chroma format of an image frame, the chroma format being subsampled relative to a luma channel of the image frame; chroma channel and determining the coefficient group size of the transform block, the coefficient group size is the maximum area of the transform block up to 16 samples, the coefficient group size is determined based only on the transform block size, and ( from the video bitstream, using coefficient groups of the determined size, independent of both i) the color plane of the transform block and (ii) color plane subsampling with the determined chroma format. and decoding.

Other aspects are also disclosed.

At least one exemplary embodiment of the invention is described with reference to the following drawings and appendices.
FIG. 1 is a schematic block diagram illustrating a video encoding and decoding system. FIG. 2A forms a schematic block diagram of a general purpose computer system capable of implementing one or both of the video encoding and decoding systems of FIG. FIG. 2B forms a schematic block diagram of a general purpose computer system capable of implementing one or both of the video encoding and decoding systems of FIG. FIG. 3 is a schematic block diagram showing functional modules of a video encoder. FIG. 4 is a schematic block diagram showing functional modules of a video decoder. FIG. 5 is a schematic block diagram illustrating possible partitioning of blocks into one or more blocks in a tree structure of general purpose video encoding. FIG. 6 is a schematic diagram of the data flow to achieve the permitted partitioning of blocks into one or more blocks in a tree structure of general purpose video encoding. FIG. 7A shows an example of dividing a coding tree unit (CTU) into several coding units (CU). FIG. 7B shows an example of dividing a coding tree unit (CTU) into several coding units (CU). FIG. 8A shows an example of dividing a coding tree unit (CTU) into several coding blocks (CBs) in a luma channel and a chroma channel. FIG. 8B shows an example of dividing a coding tree unit (CTU) into several coding blocks (CB) in a luma channel and a chroma channel. FIG. 8C shows an example of dividing a coding tree unit (CTU) into several coding blocks (CB) in a luma channel and a chroma channel. FIG. 9 shows a set of transform block sizes and associated scan patterns. FIG. 10 shows a set of rules for generating a list of allowed splits in the luma and chroma encoding trees. FIG. 11 shows a method for encoding a coding tree of image frames into a video bitstream. FIG. 12 shows a method for decoding a coding tree of image frames from a video bitstream. FIG. 13 shows a method of encoding an encoding tree of image frames into a video bitstream. FIG. 14 shows a method for decoding a coding tree of image frames from a video bitstream. FIG. 15 shows a set of transform block partitions of an intra predictive coding unit. FIG. 16 shows a method of encoding encoding units of image frames into a video bitstream. FIG. 17 shows a method for decoding coded units of an image frame from a video bitstream.

References to steps and/or features having the same reference numerals in any one or more of the accompanying drawings refer to those steps and/or features for the purposes of this specification unless a contrary intention appears. , have the same function or operation.

As mentioned above, the use of samples from the previous block introduces feedback dependencies that can limit the throughput of the block at the video encoder or decoder. To ensure that high rate processing blocks can be maintained, as is required for typical real-time encoding and decoding applications, methods that reduce the severity of the resulting feedback dependency loops are desirable. Feedback dependent loops are particularly problematic for the high sample rates of modern video formats, for example from 500-4000 samples per second, whereas ASIC (Application Specific Integrated Circuit) clock frequencies are typically hundreds of MHz. .

FIG. 1 is a schematic block diagram illustrating functional modules of a video encoding and decoding system 100. System 100 may utilize different rules for permissible subdivision of regions in luma and chroma encoding trees to reduce the worst-case block processing rate encountered. For example, system 100 may operate such that blocks are always sized as multiples of sixteen (16) samples, regardless of the block's aspect ratio. Additionally, if the coding tree includes a split that indicates the presence of small luma coded blocks, splitting may be prohibited in the chroma channel, resulting in a single chroma CB being juxtaposed with multiple luma CBs. A chroma CB can use a single prediction mode, such as one intra-prediction mode, independently of the prediction mode of each collocated luma CB (including when one or more luma CBs use inter-prediction). can be used. Residual coefficient encoding can also utilize multiples of 16 block sizes, including for blocks with a width or height of two samples.

System 100 includes a source device 110 and a destination device 130. Communication channel 120 is used to communicate encoded video information from source device 110 to destination device 130. In some configurations, source device 110 and destination device 130 may comprise either or both respective mobile telephone handsets or "smartphones," in which case communication channel 120 is a wireless channel. In other configurations, source device 110 and destination device 130 may include video conferencing equipment, in which case communication channel 120 is typically a wired channel, such as an Internet connection. In addition, source device 110 and destination device 130 may be configured to receive encoded video data on some computer-readable storage medium, such as over-the-air television broadcasts, cable television applications, Internet video applications (including streaming), and hard disk drives within file servers. Any of a wide variety of devices may be included, including devices that support applications that are supported.

As shown in FIG. 1, source device 110 includes a video source 112, a video encoder 114, and a transmitter 116. Video source 112 is typically a source of imaged video frame data (shown as 113), such as an imaging sensor, a previously imaged video sequence stored on a non-transitory storage medium, or video from a remote imaging sensor. Video source 112 may also be the output of a computer graphics card, for example, displaying the video output of an operating system and various applications running on a computing device, such as a tablet computer. Examples of source devices 110 that may include an imaging sensor as a video source 112 include smartphones, video cameras, professional video cameras, and network video cameras.

Video encoder 114 converts (or " "encoding"). Bitstream 115 is transmitted by transmitter 116 over communication channel 120 as encoded video data (or “encoded video information”). Bitstream 115 may also be stored in non-transitory storage 122, such as "flash" memory or a hard disk drive, until or instead of being transmitted over communication channel 120 at a later time. It is.

Destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. Receiver 132 receives encoded video data from communication channel 120 and passes the received video data as a bitstream to video decoder 134 (indicated by arrow 133). Video decoder 134 then outputs decoded frame data (indicated by arrow 135) to display device 136. Decoded frame data 135 has the same chroma format as frame data 113. Examples of display devices 136 include a cathode ray tube, a liquid crystal display such as a smartphone, a tablet computer, a computer monitor, or a stand-alone television set. It is also possible that the functionality of each of source device 110 and destination device 130 is implemented in a single device, examples of which include a mobile telephone handset and a tablet computer.

Notwithstanding the example devices described above, each of source device 110 and destination device 130 may be configured within a general-purpose computing system, generally through a combination of hardware and software components. FIG. 2A shows a computer module 201 configured with input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227, which can be configured as a video source 112, and a microphone 280, a printer 215, and a display device 136. 2 shows such a computer system 200 including a display device 214 that can be used, and an output device that includes a speaker 217. External modem transceiver device 216 may be used by computer module 201 to communicate with communications network 220 via connection 221 . Communication network 220, which may represent communication channel 120, may be a wide area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. If connection 221 is a telephone line, modem 216 may be a conventional "dial-up" modem. Alternatively, if connection 221 is a high capacity (eg, cable or optical) connection, modem 216 may be a broadband modem. A wireless modem may also be used for wireless connection to communications network 220. Transceiver device 216 may provide transmitter 116 and receiver 132 functionality, and communication channel 120 may be embodied in connection 221.

Computer module 201 typically includes at least one processor unit 205 and memory unit 206. For example, memory unit 206 can include semiconductor random access memory (RAM) and semiconductor read only memory (ROM). Computer module 201 also includes an audio video interface 207 that couples to a video display 214, speakers 217, and microphone 280, a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown). ), and an interface 208 for an external modem 216 and printer 215. The signal from audio video interface 207 to computer monitor 214 is typically the output of a computer graphics card. In some implementations, modem 216 may be incorporated within computer module 201 within interface 208, for example. Computer module 201 also has a local network interface 211, which enables coupling of computer system 200 via connection 223 to a local area communications network 222, known as a local area network (LAN). As shown in FIG. 2A, local communication network 222 may also be coupled to wide network 220 via a connection 224, which typically includes a so-called "firewall" device or a device of similar functionality. Local network interface 211 may include an Ethernet™ circuit card, a Bluetooth™ wireless configuration, or an IEEE 802.11 wireless configuration, although many other types of interfaces may be implemented for interface 211. Local network interface 211 may also provide the functionality of transmitter 116, and receiver 132 and communication channel 120 may also be embodied in local communication network 222.

I/O interfaces 208 and 213 may provide either or both serial and parallel connectivity, with the former typically implemented according to the Universal Serial Bus (USB) standard and with a corresponding USB connector (Fig. (not shown). Storage 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices such as floppy disk drives and magnetic tape drives (not shown) may also be used. Optical disk drive 212 is typically provided to serve as a non-volatile source of data. For example, portable memory devices such as optical disks (e.g., CD-ROM, DVD, Blu-ray DiscTM), USB-RAM, portable, external hard drives, and floppy disks may be used as suitable sources of data for computer system 200. I can do it. Typically, HDD 210, optical drive 212, and any of networks 220 and 222 are configured to operate as video source 112 or as a destination for decoded video data to be stored for playback via display 214. may be configured. Source device 110 and destination device 130 of system 100 may be embodied in computer system 200.

Components 205-213 of computer module 201 typically communicate via interconnect bus 204 in a manner that provides conventional modes of operation of computer system 200 as known to those skilled in the art. For example, processor 205 is coupled to system bus 204 using connection 218. Similarly, memory 206 and optical disk drive 212 are coupled to system bus 204 by connection 219. Examples of computers on which the above configuration can be performed include IBM-PCs and compatibles, Sun SPARC stations, Apple MacTM or similar computer systems.

Where appropriate or necessary, video encoder 114 and video decoder 134 and the methods described below may be implemented using computer system 200. In particular, video encoder 114, video decoder 134, and the described methods may be implemented as one or more software application programs 233 executable within computer system 200. Specifically, video encoder 114, video decoder 134, and the steps of the described method are performed by instructions 231 (see FIG. 2B) within software 233 executing within computer system 200. Software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate parts, where the code module corresponding to the first part performs the described method and the code module corresponding to the second part performs the method described. Manages the user interface between parts of the system and the user.

The software may be stored on computer readable media, including, for example, the storage devices described below. The software is loaded into computer system 200 from a computer readable medium and then executed by computer system 200. A computer readable medium having such software or a computer program recorded thereon is a computer program product. Preferably, use of the computer program product in computer system 200 provides video encoder 114, video decoder 134, and advantageous apparatus for implementing the described methods.

Software 233 is typically stored on HDD 210 or memory 206. The software is loaded onto computer system 200 from a computer readable medium and executed by computer system 200. Thus, for example, software 233 may be stored on an optically readable disk storage medium (eg, CD-ROM) 225 that is read by optical disk drive 212.

In some cases, the application program 233 may be provided to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212, or read by the user from the network 220 or 222. may be done. Additionally, software can also be loaded onto computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory, tangible storage media that provides recorded instructions and/or data to computer system 200 for execution and/or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray DiscTM, hard disk drives, ROMs or integrated circuits, USB memories, magneto-optical disks, or computer readable cards such as PCMCIA cards. , whether such devices are internal or external to computer module 201. Examples of transitory or non-tangible computer-readable transmission media that may also participate in providing software, application programs, instructions and/or video data or encoded video data to computer module 401 include wireless or infrared transmissions. Channels and network connections to other computers or networked devices, as well as the Internet or intranets, including e-mail transmissions and information recorded on websites and the like.

The second portion of application program 233 and the corresponding code modules described above are executed to implement one or more graphical user interfaces (GUIs) that are rendered or otherwise represented on display 214. You can. Typically through operation of a keyboard 202 and a mouse 203, applications and users of computer system 200 may manipulate the interface in a functionally adaptive manner to provide control commands and/or input to applications associated with the GUI. Can be done. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface that utilizes speech prompts output via speaker 217 and user voice commands input via microphone 280.

FIG. 2B is a detailed schematic block diagram of processor 205 and “memory” 234. Memory 234 represents a logical collection of all memory modules (including HDD 209 and semiconductor memory 206) that are accessible to computer module 201 of FIG. 2A.

When computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. POST program 250 is typically stored in ROM 249 of semiconductor memory 206 in FIG. 2A. A hardware device such as ROM 249 that stores software is sometimes referred to as firmware. The POST program 250 checks the hardware within the computer module 201 to ensure proper functioning and typically checks the processor 205, memory 234 (209, 206), and basic input/output system for correct operation. Check the software (BIOS) module 251 (usually also stored in ROM 249). When the POST program 250 is successfully executed, the BIOS 251 boots the hard disk drive 210 of FIG. 2A. When the hard disk drive 210 is started, the bootstrap loader program 252 resident on the hard disk drive 210 is executed via the processor 205. As a result, the operating system 253 is loaded into the RAM memory 206, and the operating system 253 starts operating thereon. Operating system 253 is a system-level application executable by processor 205 that performs various high-level functions including processor management, memory management, device management, storage management, software application interfaces, and general purpose user interfaces.

Operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on computer module 201 has enough memory to run without conflicting with memory allocated to another process. We guarantee that we have the following. Furthermore, the different types of memory available in the computer system 200 of FIG. 2A must be used appropriately so that each process can execute effectively. Accordingly, aggregate memory 234 is not intended to indicate how particular segments of memory are allocated (unless otherwise specified), but rather a general representation of memory accessible by computer system 200. It is intended to provide a comprehensive view of how such segments are used.

As shown in FIG. 2B, processor 205 includes a number of functional modules, including a controller 239, an arithmetic logic unit (ALU) 240, and local or internal memory 248, sometimes referred to as cache memory. Cache memory 248 typically includes a number of storage registers 244-246 within a register section. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using connections 218 . Memory 234 is coupled to bus 204 using connection 219.

Application program 233 includes a sequence of instructions 231 that may include conditional branch and loop instructions. Program 233 may also include data 232 used to execute program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of instructions 231 and memory locations 228-230, particular instructions may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, the instructions may be segmented into several parts, each stored in a separate memory location, as illustrated by the instruction segments shown in memory locations 228 and 229.

Generally, processor 205 is provided with a set of instructions to execute therein. Processor 205 waits for subsequent input, to which processor 205 reacts by executing another set of instructions. Each input may include data generated by one or more of input devices 202, 203, data received from an external source via one of networks 220, 202, data received from an external source via one of networks 220, 202, or one of storage devices 206, 209. 2A, or data retrieved from a storage medium 225 inserted into a corresponding reader 212, all of which are shown in FIG. 2A. has been done. Executing a set of instructions may output data. Execution may also include storing data or variables in memory 234.

Video encoder 114, video decoder 134, and the described method may use input variables 254 stored in corresponding memory locations 255, 256, 257 within memory 234. Video encoder 114, video decoder 134, and the described method produce output variables 261, which are stored in corresponding memory locations 262, 263, 264 within memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266 and 267.

Referring to processor 205 in FIG. 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 perform "fetch, decode, and execute the sequence of micro-operations necessary to execute the "run" cycle. Each fetch, decode, and execute cycle consists of: a fetch operation in which the instruction 231 is fetched or read from a memory location 228, 229, 230, a decode operation in which the controller 239 determines which instruction was fetched, a decode operation in which the controller 239 and/or the ALU 240 It has a , which performs the action of executing an instruction.

Further fetch, decode, and execute cycles for the next instruction can then be performed. Similarly, controller 239 may perform a store cycle in which a value is stored or written to memory location 232 .

Each step or subprocess in the method of FIGS. 10 and 11 described below is associated with one or more segments of program 233 and typically includes register sections 244, 245, 247, ALU 240, and control within processor 205. This is accomplished by units 239 working together to perform fetch, decode, and execute cycles for all instructions in the instruction set for the noted segment of program 233.

FIG. 3 is a schematic block diagram illustrating functional modules of video encoder 114. FIG. 4 is a schematic block diagram illustrating functional modules of video decoder 134. Generally, data is passed between the functional modules of video decoder 134 and video encoder 114 in groups or as an array of samples or coefficients, such as a division of a block into fixed-sized subblocks. Video encoder 114 and video decoder 134 may be implemented using a general purpose computer system 200, as shown in FIGS. 2A and 2B, with various functional modules residing on a hard disk drive 205 and being executed by a This may be implemented by specialized hardware within computer system 200, by software executable within computer system 200, such as one or more software code modules of software application program 233 controlled during its execution. Alternatively, video encoder 114 and video decoder 134 may be implemented by a combination of software and specialized hardware executable within computer system 200. Video encoder 114, video decoder 134, and the described methods may alternatively be implemented in dedicated hardware such as one or more integrated circuits that perform the functions or sub-functions of the described methods. Such specialized hardware may be a graphics processing unit (GPU), digital signal processor (DSP), application specific standard product (ASSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or one It may include one or more microprocessors and associated memory. In particular, video encoder 114 includes modules 310-386, and video decoder 134 includes modules 420-496, each of which may be implemented as one or more software code modules of software application program 233.

Video encoder 114 of FIG. 3 is an example of a general purpose video coding (VVC) video encoding pipeline, although other video codecs may be used to perform the processing stages described herein. Video encoder 114 receives imaged frame data 113, such as a series of frames, each frame including one or more color channels. Frame data 113 may be in 4:2:0 chroma format or 4:2:2 chroma format. Block partitioner 310 initially partitions frame data 113 into CTUs, which are generally square in shape and configured such that a particular size for the CTU is used. The size of the CTU can be, for example, 64x64, 128x128, or 256x256 luma samples. Block partitioner 310 further partitions each CTU into one or more CBs according to a luma encoding tree and a chroma encoding tree. CBs have a variety of sizes and may include both square and non-square aspect ratios. The operation of block partitioner 310 will be further described with reference to FIG. However, in the VVC standard, CB, CU, PU, and TU always have side lengths that are powers of two. Accordingly, the current CB, denoted as 312, is output from the block partitioner 310 and progresses according to the CTU's chroma encoding tree and luma encoding tree as it repeats over one or more blocks of the CTU. Options for dividing a CTU into CBs are further explained below with reference to FIGS. 5 and 6.

The CTUs obtained from the first partition of frame data 113 may be scanned in raster scan order and grouped into one or more "slices." A slice may be an "intra" (or "I") slice, where an intra slice (I slice) indicates that all CUs within the slice are intra-predicted. Alternatively, the slices may be uni- or bi-predictive ("P" or "B" slices, respectively), indicating the further availability of uni- and bi-prediction in the slice, respectively.

For each CTU, video encoder 114 operates in two stages. In the first stage (referred to as the "search" stage), block partitioner 310 tests various potential configurations of the encoding tree. Each potential configuration of the coding tree has an associated "candidate" CB. The first stage involves testing various candidate CBs to select the CB that provides high compression efficiency with low distortion. This test typically involves a Lagrangian optimization whereby candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error with respect to input frame data 113). The “best” candidate CB (the CB with the lowest estimated rate/distortion) is selected for subsequent encoding into bitstream 115. Candidate CB evaluation involves using a CB for a given area, or subdividing the area according to various partitioning options and encoding each resulting smaller area with a further CB, or Includes options to further divide the area. As a result, both the CB and the coding tree itself are selected in the search stage.

Video encoder 114 generates a predictive block (PB), indicated by arrow 320, for each CB, e.g., CB 312. PB 320 is a prediction of the content of the associated CB 312. Subtractor module 322 generates a difference between PB 320 and CB 312, denoted as 324 (or "residual", referring to the difference that is in the spatial domain). Difference 324 is the block size difference between corresponding samples in PB 320 and CB 312. Difference 324 is transformed, quantized, and represented as a transform block (TB) indicated by arrow 336. PB 320 and associated TB 336 are typically selected from one of many possible candidate CBs based on, for example, estimated cost or distortion.

A candidate coded block (CB) is a CB resulting from one of the prediction modes available to video encoder 114 for the associated PB and the resulting residual. Each candidate CB yields one or more corresponding TBs, as described below with reference to FIG. TB 336 is a quantified and transformed representation of difference 324. When combined with the predicted PB at video decoder 114, TB 336 reduces the difference between the decoded CB and original CB 312 at the expense of additional signal in the bitstream.

Therefore, each candidate coded block (CB), i.e. a prediction block (PB) in combination with a transform block (TB), has an associated coding cost (or "rate") and an associated difference (or "distortion"). have Rate is typically measured in bits. CB distortion is typically estimated as a difference in sample values, such as a sum of absolute differences (SAD) or a sum of squared differences (SSD). The estimate obtained from each candidate PB is determined by mode selector 386 using difference 324 to determine the intra prediction mode (represented by arrow 388). Estimating the coding cost associated with each candidate prediction mode and corresponding residual encoding can be performed at a significantly lower cost than entropy encoding of the residual. Therefore, a large number of candidate modes can be evaluated to determine the optimal mode in rate-distortion sensing.

Determining the optimal mode from a rate-distortion perspective is typically accomplished using a variant of Lagrangian optimization. Selection of an intra-prediction mode 388 typically involves determining a coding cost for residual data resulting from application of a particular intra-prediction mode. The encoding cost can be approximated by using the "sum of absolute transform differences" (SATD), whereby a relatively simple transform such as the Hadamard transform is used to reduce the estimated transform residual cost to obtain. In some embodiments that use relatively simple transformations, the cost obtained from the simplified estimation method is monotonically related to the actual cost that would otherwise be determined from a complete evaluation. In embodiments with monotonically related estimated costs, a simplified estimation method can be used to make the same decision (i.e., intra-prediction mode) while reducing the complexity of video encoder 114. To allow for possible non-monotonicity in the relationship between estimated and actual costs, a simplified estimation method can be used to generate the list of best candidates. Non-monotonicity may arise, for example, from further mode decisions available for encoding the residual data. The list of best candidates may be any number. Using the best candidates, a more complete search can be performed to establish the optimal mode selection for encoding the residual data for each of the candidates, and the final intra-prediction mode along with other mode decisions. Allow choice.

Other mode decisions include the ability to skip forward transforms, known as "transform skip." Skipping the transform is suitable for residual data that lacks proper correlation to reduce encoding cost via representation as transform basis functions. Certain types of content, such as relatively simple computer-generated graphics, may exhibit similar behavior. In the case of a "skipped transform", the residual coefficients are still encoded even though the transform itself is not performed.

A Lagrangian or similar optimization process may be employed to select both the optimal partitioning of CTUs into CBs (by block partitioner 310) and the selection of the best prediction mode from multiple possibilities. Through application of a Lagrangian optimization process of candidate modes in mode selection module 386, the intra-prediction mode with the lowest cost measure is selected as the "best" mode. The lowest cost mode is the selected intra prediction mode 388 and is encoded into the bitstream 115 by the entropy encoder 338. The selection of intra-prediction mode 388 by the operation of mode selection module 386 extends to the operation of block partitioner 310. For example, candidates for selection of intra-prediction modes 388 include modes applicable to a given block and also modes applicable to multiple smaller blocks that are collectively placed together with the given block. and may include. The process of implicitly selecting candidates is also the process of determining the best hierarchical decomposition of CTUs into CBs, given the modes applicable to a given block and smaller colocated blocks.

In the second stage of operation of video encoder 114 (referred to as the "encoding" stage), the selected luma encoding tree and the selected chroma encoding tree, and thus the iterations for each selected CB, are performed within video encoder 114. executed. At the iteration, the CB is encoded into the bitstream 115, as described further herein.

Entropy encoder 338 supports both variable length encoding of syntax elements and arithmetic encoding of syntax elements. Arithmetic encoding is supported using context-adaptive binary arithmetic encoding processing. An arithmetic encoded syntax element consists of a sequence of one or more 'bins'. A bin, like a bit, has a value of "0" or "1". However, the bins are not encoded as discrete bits within the bitstream 115. A bin has an associated predicted (or "likely" or "most likely") value and an associated probability, known as the "context." A "most probable symbol" (MPS) is encoded when the actual bin encoded matches the predicted value. Encoding the most probable symbols is relatively cheap in terms of bits consumed. If the actual bin to be encoded does not match the likely value, a "least probability symbol" (LPS) is encoded. Encoding the lowest probability symbol has a relatively high cost in terms of bits consumed. Bin encoding techniques allow for efficient encoding of bins in which the probabilities of ``0'' versus ``1'' are skewed. For syntax elements with two possible values (ie, flag), a single bin is sufficient. For syntactic elements with many possible values, a series of bins is required.

The presence of later bins in the sequence may be determined based on the values of earlier bins in the sequence. Furthermore, each bin can be associated with more than one context. The selection of a particular context may depend on the previous bin of the syntax element, the bin values of adjacent syntax elements (ie, from adjacent blocks), etc. Each time a context encoding bin is encoded, the selected context (if any) for that bin is updated in a manner that reflects the new bin value. In this way, the binary arithmetic coding system is said to be adaptive.

Also supported by video encoder 114 are bins lacking context (“bypass bins”). Bypass bins are encoded assuming an equal probability distribution between "0" and "1". Therefore, each bin occupies one bit in bitstream 115. No context saves memory and reduces complexity. Therefore, if the distribution of values in a particular bin is not skewed, a bypass bin is used. An example of an entropy coder that uses context and adaptation is known in the art as CABAC (Context Adaptive Binary Arithmetic Coder), and many variants of this coder are used for video coding.

Entropy encoder 338 encodes intra prediction mode 388 using a combination of context encoding bins and bypass encoding bins. Typically, a list of “most probable modes” is generated at video encoder 114. The list of most probable modes is typically a fixed length, such as 3 or 6 modes, and can include modes encountered in previous blocks. The context encoding bin encodes a flag indicating whether the intra prediction mode is one of the most probable modes. If intra prediction mode 388 is one of the most probable modes, further signaling using bypass encoded bins is encoded. Further encoded signaling indicates which most probable mode corresponds to the intra-prediction mode 388, for example using a truncated unary bin string. Otherwise, intra prediction mode 388 is encoded as a "remaining mode." Encoding as a remaining mode allows intra-predictions other than those present in the most probable mode list using alternative syntax such as fixed-length codes that are also encoded using bypass-encoded bins. express the mode.

Multiplexer module 384 outputs PB 320 according to the determined best intra prediction mode 388 to select from the tested prediction modes of each candidate CB. Candidate prediction modes need not include all possible prediction modes supported by video encoder 114.

Prediction modes are broadly classified into two categories. The first category is "intra frame prediction" (also called "intra prediction"). In intra-frame prediction, a prediction is generated for a block, and the generation method may use other samples obtained from the current frame. For intra-predicted PBs, different intra-prediction modes can be used for luma and chroma, and thus intra-prediction is primarily described in terms of operations on the PB.

The second category of prediction modes is "interframe prediction" (also called "inter prediction"). In interframe prediction, a prediction of a block is generated using samples from one or two frames that precede the current frame in the order of encoding the frames in the bitstream. Furthermore, for interframe prediction, a single coding tree is typically used for both luma and chroma channels. The encoding order of frames within a bitstream may differ from the order of frames when captured or displayed. If one frame is used for prediction, the block is said to be "uni-predictive" and has one associated motion vector. If two frames are used for prediction, a block is said to be "bi-predicted" and has two associated motion vectors. For P slices, each CU may be intra-predicted or single-predicted. For B slices, each CU may be intra-predicted, uni-predicted, or bi-predicted. Frames are typically encoded using a "group of pictures" structure that allows for a temporal hierarchy of frames. The temporal hierarchy of frames allows frames to refer to preceding and following pictures in the order in which they are displayed. Images are encoded in the order necessary to ensure that the dependencies for decoding each frame are met.

A subcategory of inter prediction is called "skip mode." Inter-prediction and skip modes are described as two separate modes. However, both inter-prediction mode and skip mode include motion vectors that refer to blocks of samples from the previous frame. Inter prediction includes a coded motion vector delta and specifies a motion vector to a motion vector predictor. A motion vector predictor is obtained from a list of one or more candidate motion vectors selected by a "merge index." The encoded motion vector delta provides a spatial offset to the selected motion vector prediction. Inter prediction also uses coded residuals within the bitstream 133. Skip mode uses only the index (also called "merge index") to select one of several motion vector candidates. The selected candidate is used without further signaling. Also, skip mode does not support encoding of residual coefficients. When skip mode is used, the absence of coded residual coefficients means that there is no need to perform a transform for skip mode. Therefore, skip mode typically does not create pipelining problems. Pipelining problems may be the case for intra-predicted CUs and inter-predicted CUs. Because of the limited signaling of skip mode, skip mode is useful for achieving very high compression performance when relatively high quality reference frames are available. Bi-predictive CUs at higher temporal layers of the random access picture group structure typically have high quality reference pictures and motion vector candidates that accurately reflect the underlying motion.

Samples are selected according to the motion vector and reference picture index. Motion vectors and reference picture indices apply to all color channels, so inter prediction is primarily described in terms of operation on the PU rather than the PB. Within each category (i.e., intra and interframe prediction), different techniques may be applied to generate PUs. For example, intra-prediction may use values from adjacent rows and columns of previously reconstructed samples in combination with direction to generate PUs according to a predetermined filtering and generation process. Alternatively, a PU may be described using a small number of parameters. Inter prediction methods can vary in the number of motion parameters and their accuracy. The motion parameters typically include a reference frame index indicating which reference frame from the list of reference frames should be used and a spatial transformation for each of the reference frames, but more frames, special frames, or can include complex affine parameters such as scaling and rotation. Additionally, a predetermined motion refinement process may be applied to generate a dense motion estimate based on the reference sample block.

Determining and selecting the PB 320 and subtracting the PB 320 from the original sample block in a subtractor 322 yields the residual, denoted 324, which has the lowest encoding cost and undergoes lossy compression. The lossy compression process includes the steps of transformation, quantization, and entropy encoding. A forward linear transform module 326 applies a forward transform to the difference 324 to transform the difference 324 from the spatial domain to the frequency domain and generate linear transform coefficients represented by arrows 328. The linear transform coefficients 328 are passed to a forward quadratic transform module 330 to generate transform coefficients represented by arrows 332 by performing a non-separable quadratic transform (NSST) operation. Forward linear transforms are typically separable, typically using DCT-2 to transform a set of rows and then a set of columns for each block, but DST-7 and DCT-8 are also used. , for example, horizontally for block widths not exceeding 16 samples and vertically for block heights not exceeding 16 samples. The transformation of each set of rows and columns involves first applying a one-dimensional transformation to each row of the block to produce a partial result, and then applying a one-dimensional transformation to each column of the partial result to produce the final result. executed by The forward quadratic transform is generally a non-separable transform, which is only applied to the intra-predicted CU residuals and may nevertheless be bypassed. The forward quadratic transform consists of 16 samples (arranged as the upper left 4x4 subblock of the linear transform coefficients 328) or 64 samples (arranged as four 4x4 subblocks of the linear transform coefficients 328). (arranged as an 8x8 coefficient in the top left corner). Furthermore, the matrix coefficients of the forward quadratic transform are selected from multiple sets according to the intra-prediction mode of the CU, such that two sets of coefficients are available for use. Using one of the sets of matrix coefficients, or bypassing the forward quadratic transform, is signaled with the "nsst_index" syntax element, using a truncated unary binarisation: It is encoded to represent the values zero (no quadratic transform applied), one (first set of selected matrix coefficients), or two (second set of selected matrix coefficients).

Transform coefficients 332 are passed to quantizer module 334. In module 334, quantization with a "quantization parameter" is performed to generate residual coefficients represented by arrows 336. The quantization parameter is constant for a given TB, thus providing uniform scaling for the generation of residual coefficients for the TB. Non-uniform scaling is also possible by applying a "quantization matrix", whereby the scaling factor applied to each residual coefficient is equal to the quantization parameter and of a size typically equal to the size of TB. is derived from the combination with the corresponding entry in the scaling matrix with . Residual coefficients 336 are provided to entropy encoder 338 for encoding in bitstream 115. Typically, the residual coefficients of each TB having at least one significant residual coefficient of the TU are scanned to generate an ordered list of values according to a scan pattern. The scan pattern typically scans the TB as a sequence of 4x4 "subblocks", providing a regular scanning operation with a granularity of 4x4 sets of residual coefficients, and the arrangement of the subblocks depends on the size of the TB. Dependent. Additionally, the prediction mode 388 and corresponding block partitioning are also encoded into the bitstream 115.

As mentioned above, video encoder 114 needs access to a frame representation that corresponds to the frame representation seen at video decoder 134. Therefore, residual coefficients 336 are also dequantized by dequantizer module 340 to produce inverse transform coefficients represented by arrows 342. The inverse transform coefficients 342 are passed through an inverse quadratic transform module 344 to produce intermediate inverse transform coefficients represented by arrows 346. The intermediate inverse transform coefficients 346 are passed to an inverse linear transform module 348 to generate residual samples represented by TU arrows 350. The type of inverse transform performed by inverse quadratic transform module 344 corresponds to the type of forward transform performed by forward quadratic transform module 330. The type of inverse transform performed by inverse linear transform module 348 corresponds to the type of linear transform performed by linear transform module 326. Summing module 352 sums residual samples 350 and PU 320 to generate reconstructed samples of the CU (indicated by arrow 354).

The reconstructed samples 354 are passed to a reference sample cache 356 and an in-loop filter module 368. Reference sample cache 356 is typically implemented using static RAM on an ASIC (thus avoiding costly off-chip memory accesses) and is used to generate intra-frame PBs for subsequent CUs within a frame. Provide the minimum sample storage needed to satisfy dependencies. The minimum dependency typically includes a "line buffer" of samples along the bottom of the CTU's row, used by the CTU's next row and column buffering, and whose range is set by the CTU's height. Ru. Reference sample cache 356 provides reference samples (indicated by arrow 358) to reference sample filter 360. Sample filter 360 applies a smoothing operation to produce a filtered reference sample (indicated by arrow 362). Filtered reference samples 362 are used by intra-frame prediction module 364 to generate an intra-predicted block of samples represented by arrow 366. For each candidate intra prediction mode, intra frame prediction module 364 generates a block, or 366, of samples.

An in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stage includes a "deblocking filter" (DBF) that applies smoothing aligned to the CU boundaries to reduce artifacts resulting from discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further available filtering stage in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and applying an offset at the sample level according to the assigned category.

Filtered samples, represented by arrow 370, are output from in-loop filter module 368. Filtered samples 370 are stored in frame buffer 372. Frame buffer 372 typically has the capacity to store a number (eg, up to 16) pictures and is therefore stored in memory 206. Frame buffer 372 is typically not stored using on-chip memory because of the large memory consumption required. Therefore, accessing frame buffer 372 is costly in terms of memory bandwidth. Frame buffer 372 provides a reference frame (represented by arrow 374) to motion estimation module 376 and motion compensation module 380.

Motion estimation module 376 estimates a number of “motion vectors” (denoted as 378), each of which is a Cartesian spatial offset from the current CB position and one of the reference frames in frame buffer 372. Refer to blocks within. A filtered block of reference samples (denoted as 382) is generated for each motion vector. Filtered reference samples 382 form further candidate modes available for potential selection by mode selector 386. Furthermore, for a given CU, PU 320 may be formed using one reference block ("uni-prediction") or using two reference blocks ("bi-prediction"). Good too. For the selected motion vector, motion compensation module 380 generates PB 320 according to a filtering process that supports sub-pixel precision within the motion vector. Therefore, motion estimation module 376 (operating on many candidate motion vectors) is compared to that of motion compensation module 380 (operating only on selected candidates) to reduce computational complexity. can perform a simplified filtering process.

Although video encoder 114 of FIG. 3 is described with reference to generic video coding (VVC), other video encoding standards or implementations may also use the processing stages of modules 310-386. Frame data 113 (and bitstream 115) may also be read from (or written to) memory 206, hard disk drive 210, CD-ROM, Blu-ray disc™, or other computer-readable storage medium. Additionally, frame data 113 (and bitstream 115) may be received (or transmitted) from an external source, such as a server connected to communication network 220 or a radio frequency receiver.

Video decoder 134 is shown in FIG. Although video decoder 134 of FIG. 4 is an example of a general purpose video coding (VVC) video decoding pipeline, other video codecs may be used to perform the processing stages described herein. As shown in FIG. 4, bitstream 133 is input to video decoder 134. Bitstream 133 may be read from memory 206, hard disk drive 210, CD-ROM, Blu-ray Disc ^™ , or other non-transitory computer-readable storage medium. Alternatively, bitstream 133 may be received from an external source, such as a server connected to communications network 220 or a radio frequency receiver. Bitstream 133 includes encoded syntax elements representing the imaging frame data to be decoded.

Bitstream 133 is input to entropy decoder module 420. Entropy decoder module 420 extracts syntax elements from bitstream 133 by decoding the sequence of “bins” and passes the values of the syntax elements to other modules in video decoder 134. Entropy decoder module 420 decodes each syntax element as a sequence of one or more bins using a computational decoding engine. Each bin may use one or more "contexts," with a context describing the probability level used to encode the bin's "1" and "0" values. If multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts for decoding the bin. The process of decoding the bins forms a sequential feedback loop. The number of operations in the feedback loop is preferably minimized to enable entropy decoder 420 to achieve high throughput in bins/second. Context modeling relies on context, ie, other properties of the bitstream known to video decoder 134, when selecting the previous property of the current bin. For example, a context may be selected based on the current CU's quadtree depth within the encoding tree. Preferably, the dependence is determined prior to decoding the bins based on well-known characteristics or without requiring lengthy sequential processing. The quadtree depth of the encoding tree is an example of a dependence on context modeling that is readily known. Intra prediction mode is an example of a dependency for context modeling that is relatively difficult or computationally intensive to determine. An intra-prediction mode is encoded as either an index into a list of "most probable modes" (MPM) or an index into a list of "remaining modes", and is an index between the MPM and the remaining modes. The selection between is according to the decoded "intra_luma_mpm_flag". If MPM is used, the "intra_luma_mpm_idx" syntax element is decoded to select which of the most probable modes to use. Generally, there are six MPMs. If the remaining modes are used, the "intra_luma_remainder" syntax element is decoded to select which of the remaining (non-MPM) modes to use. Determining both the most probable mode and the remaining modes requires a significant number of operations and involves dependence on the intra-prediction modes of neighboring blocks. For example, the adjacent block may be the upper left block of the current block. Desirably, the context of each CU's bin can be determined without knowing the intra prediction mode being signaled, allowing parsing by the arithmetic coding engine. Therefore, the feedback loop present in the arithmetic coding engine for sequential bin decoding avoids dependence on intra prediction mode. The intra-prediction mode decision may be deferred to a subsequent processing stage using a separate feedback loop due to the dependence of the MPM list configuration on the intra-prediction modes of neighboring blocks. Accordingly, the computational decode engine of entropy decoder module 420 can parse intra_luma_mpm_flag, intra_luma_mpm_idx, intra_luma_remainder without needing to know the intra prediction mode of previous (eg, adjacent) blocks. Entropy decoder module 420 applies an arithmetic coding algorithm, such as “context adaptive binary arithmetic coding” (CABAC), to decode syntax elements from bitstream 133. The decoded syntax elements are used to reconstruct parameters within video decoder 134. The parameters include residual coefficients (represented by arrow 424) and mode selection information, such as intra prediction mode (represented by arrow 458). The mode selection information also includes information such as motion vectors and the division of each CTU into one or more CBs. The parameters are used to generate the PB, typically in combination with sample data from previously decoded CBs.

Residual coefficients 424 are input to an inverse quantization module 428. Inverse quantization module 428 performs inverse quantization (or "scaling") on residual coefficients 424 to produce reconstructed intermediate transform coefficients represented by arrows 432 according to quantization parameters. The reconstructed intermediate transform coefficients 432 are passed to an inverse quadratic transform module 436 where a quadratic transform is applied or not operated on (bypassed) according to the decoded "nsst_index" syntax element. “nsst_index” is decoded from bitstream 133 by entropy decoder 420 under execution of processor 205. As explained with reference to FIG. 3, "nsst_index" is decoded from the bitstream 133 as a truncated unary syntax element having a value between zero and two. Inverse quadratic transform module 436 produces reconstructed transform coefficients 440. If the use of a non-uniform inverse quantization matrix is indicated in bitstream 133, video decoder 134 reads the quantization matrix from bitstream 133 as a sequence of scaling factors and places the scaling factors into the matrix. Inverse scaling uses a quantization matrix in combination with a quantization parameter to produce reconstructed intermediate transform coefficients 432.

The reconstructed transform coefficients 440 are passed to an inverse linear transform module 444. Module 444 transforms the coefficients from the frequency domain back to the spatial domain. TB is effectively based on valid and non-valid residual coefficient values. The result of the operation of module 444 is a block of residual samples represented by arrow 448. Residual samples 448 are equal in size to the corresponding CU. Residual samples 448 are provided to a summing module 450. At summing module 450, residual samples 448 are added to the decoded PB (represented as 452) to produce a block of reconstructed samples represented by arrow 456. Reconstructed samples 456 are provided to a reconstructed sample cache 460 and an in-loop filtering module 488. In-loop filtering module 488 produces a reconstructed block of frame samples, represented as 492. Frame samples 492 are written to frame buffer 496.

Reconstruction sample cache 460 operates similarly to reconstruction sample cache 356 of video encoder 114. Reconfigured sample cache 460 is needed to intra-predict subsequent CBs without going through memory 206 (e.g., by instead using data 232, which is typically on-chip memory). Provide storage for reconstructed samples. Reference samples represented by arrow 464 are obtained from reconstructed sample cache 460 and provided to reference sample filter 468 to produce filtered reference samples shown by arrow 472. Filtered reference samples 472 are provided to an intra-frame prediction module 476. Module 476 generates a block of intra-prediction samples, represented by arrow 480, according to intra-prediction mode parameters 458 signaled in bitstream 133 and decoded by entropy decoder 420.

If the prediction mode of the CB is indicated to be intra prediction in the bitstream 133, the intra prediction samples 480 form a decoded PB 452 via a multiplexer module 484. Intra-prediction produces predictive blocks (PBs) of samples, ie, blocks within one color component that are derived using "adjacent samples" within the same color component. Neighboring samples are samples that are adjacent to the current block and have already been reconstructed by being ahead in the block decoding order. If the luma and chroma blocks are co-located, the luma and chroma blocks can use different intra prediction modes. However, each of the two chroma channels share the same intra prediction mode. Intra predictions are classified into three types. "DC intra-prediction" involves populating the PB with a single value that represents the average of adjacent samples. "Planar intra prediction" involves populating the PB with samples that follow the plane, with DC offsets and vertical and horizontal gradients derived from neighboring samples. “Angular intra prediction” involves populating a PB with neighboring samples that are filtered and propagated in a particular direction (or “angle”) across the PB. In VVC65, angles are supported in rectangular blocks that allow additional angles not available in square blocks, producing a total of 87 angles. A fourth type of intra-prediction is available for chroma PB, whereby the PB is generated from collocated luma reconstructed samples according to a "cross component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived model is then used to generate blocks of chroma PB samples from the collocated luma samples.

If the prediction mode of the CB is indicated to be inter prediction in the bitstream 133, the motion compensation module 434 selects a block of samples from the frame buffer 496 and uses the motion vector and reference frame index for filtering. to generate a block of inter-predicted samples, denoted as 438. Blocks of samples 498 are obtained from previously decoded frames stored in frame buffer 496. For bidirectional prediction, two blocks of samples are generated and blended together to generate the samples for the decoded PB 452. Frame buffer 496 is populated with filtered block data 492 from in-loop filtering module 488 . Similar to in-loop filtering module 368 of video encoder 114, in-loop filtering module 488 applies any, at least, or all of DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both luma and chroma channels, but the filtering process for subsample interpolated luma and chroma channels is different. If the partitioning in the coding tree results in a relatively small set of luma blocks and the corresponding chroma regions are not partitioned into corresponding small chroma blocks, the blocks are coded as described with reference to FIGS. 13 and 14, respectively. encoded and decrypted. In particular, if any small luma block is predicted using inter prediction, the inter prediction operation is performed only on the luma CB and not on any part of the corresponding chroma CB. In-loop filtering module 368 generates filtered block data 492 from reconstructed samples 456.

FIG. 5 is a schematic block diagram illustrating a set 500 of available divisions or splits of a region into one or more sub-regions within a tree structure for general purpose video encoding. The divisions shown in set 500 divide each CTU into one or more CUs or CBs according to the coding tree, as determined by Lagrangian optimization, as described with reference to FIG. It is available to block partitioner 310 of encoder 114 to do this.

Although set 500 shows that only square regions are partitioned into other, possibly non-square sub-regions, diagram 500 shows potential partitioning but does not require the containing region to be square. I hope you understand that. If the containing area is non-square, the dimensions of the block resulting from the split are scaled according to the aspect ratio of the containing block. When a region is no longer divided, ie at a leaf node of the coding tree, a CU occupies the region. A particular subdivision of a CTU into one or more CUs by block partitioner 310 is referred to as a "coding tree" of the CTU.

The process of subdividing a region into subregions must end when the resulting subregion reaches the minimum CU size. In addition to constraining the CU to prohibit block regions smaller than a predetermined minimum size, eg, 16 samples, the CU is constrained to have a minimum width or height of 4. Other minimum values for both width and height or for width or height are also possible. The process of subdivision may also terminate before the deepest level of decomposition, resulting in a CU larger than the minimum CU size. No splitting occurs, so that a single CU can occupy the entire CTU. A single CU occupying the entire CTU is the largest available coding unit size. Further, a CU in which division does not occur is larger than the processing area size. As a result of the two or three splits at the highest level of the encoding tree, CU sizes such as 64x128, 128x64, 32x128, and 128x32 are possible, each of which is larger than the processing area size. An example of a CUS larger than the processing region size further described with reference to FIGS. 10A-10F. The use of a subsampled chroma format, such as 4:2:0, allows the configuration of video encoder 114 and video decoder 134 to finish dividing regions in the chroma channel sooner than in the luma channel.

At the leaf nodes of the encoding tree, there are CUs with no further subdivisions. For example, leaf node 510 includes one CU. At non-leaf nodes of the encoding tree, there may be a partition into two or more further nodes, each node containing a leaf node and thus one CU, or a further partition into smaller regions. At each leaf node of the coding tree, there is one coding block for each color channel. A split terminating at the same depth for both luma and chroma results in three juxtaposed CBs. A split that terminates at a luma depth deeper than chroma will result in multiple luma CBs being juxtaposed with the chroma channel's CBs.

Quadtree partitioning 512 partitions the containing region into four equally sized regions, as shown in FIG. Compared to HEVC, general purpose video coding (VVC) achieves additional flexibility by adding horizontal halves 514 and vertical halves 516. Each of partitions 514 and 516 divides the containing region into two equally sized regions. The divisions are along horizontal boundaries (514) or vertical boundaries (516) within the containing block.

Additional flexibility is achieved in general purpose video encoding by adding horizontal thirds 518 and vertical thirds 520. Triple divisions 518 and 520 divide the block into three sections bounded either horizontally (518) or vertically (520) along 1/4 and 3/4 of the width or height of the containing region. Divide into regions. The combination of quadtree, binary tree, and tertiary tree is called "QTBTTT." The root of the tree contains zero or more quadtree splits (the "QT" section of the tree). When a QT section ends, zero or more bipartitions or three-partitions ("multi-tree" or "MT" sections of the tree) occur, ultimately terminating in the CB or CU of a leaf node of the tree. If the tree describes all color channels, the tree leaf nodes are CUs. If the tree describes a luma or chroma channel, the tree leaf nodes are CBs.

Compared to HEVC, which supports only quadtrees and thus only square blocks, QTBTT has a larger number of possible yields the CU size. The possibility of unusual (non-square) block sizes constrains the split options to eliminate splits where the block width or height is either less than 4 samples or not a multiple of 4 samples. This can be reduced by Generally, this constraint is applied when considering luma samples. However, in the described configuration, constraints can be applied separately to blocks for chroma channels. Applying constraints to the splitting options for chroma channels may result in different minimum block sizes for luma and chroma, such as when frame data is in 4:2:0 chroma format or 4:2:2 chroma format. Each division generates subregions whose side dimensions are unchanged, halved, or quartered with respect to this included region. Since the CTU size is a power of 2, the side dimensions of all CUs are also powers of 2.

FIG. 6 is a schematic flow diagram illustrating a data flow 600 of a QTBTTT (or "coding tree") structure used in general purpose video encoding. The QTBTT structure is used for each CTU to define the division of the CTU into one or more CUs. The QTBTTT structure of each CTU is determined by block partitioner 310 in video encoder 114 and encoded into bitstream 115 or decoded from bitstream 133 by entropy decoder 420 in video decoder 134. Data flow 600 further characterizes the permissible combinations available to block partitioner 310 to partition a CTU into one or more CUs according to the partitioning shown in FIG.

Starting from the top level of the hierarchy, ie, the CTU, zero or more quadtree splits are first performed. Specifically, a quadtree (QT) partitioning decision 610 is made by block partitioner 310. The decision at 610 to return a "1" symbol indicates a decision to split the current node into four subnodes according to quadtree partitioning 512. As a result, four new nodes are generated, such as 620, and for each new node, the QT split decision 610 is returned. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splits should be performed (returns a "0" symbol), quadtree splits stop and multi-tree (MT) splits are then considered.

First, an MT partitioning decision 612 is made by the block partitioner 310. At 612, a decision to perform MT splitting is indicated. Returning a "0" symbol in decision 612 indicates that no further division of the node into subnodes is performed. If no further splitting of the node is performed, the node is a leaf node of the encoding tree and corresponds to a CU. Leaf nodes are output at 622. Alternatively, if MT partitioning 612 indicates a decision to perform MT partitioning (returning a "1" symbol), block partitioner 310 proceeds to direction determination 614.

Direction determination 614 indicates the direction of the MT split as either horizontal (“H” or “0”) or vertical (“V” or “1”). Block partitioner 310 proceeds to decision 616 if decision 614 returns a "0" indicating horizontal direction. Block partitioner 310 proceeds to decision 618 if decision 614 returns a "1" indicating vertical direction.

In each of decisions 616 and 618, the number of partitions for the MT split is indicated as either two (two splits or "BT" nodes) or three (three splits or "TT") for the BT/TT split. That is, a BT/TT split decision 616 is made by the block partitioner 310 when the indicated direction from 614 is horizontal, and a BT/TT split decision 618 is made when the indicated direction from 614 is vertical. This is sometimes done by block partitioner 310.

The BT/TT split decision 616 indicates whether the horizontal split is a split of two 514, indicated by returning a "0", or a split of three 518, indicated by returning a "1". If the BT/TT split decision 616 indicates a two-part split, two nodes are generated by the block partitioner 310 according to the horizontal two-part split 514 in an HBT CTU node generation step 625 . If the BT/TT partition 616 indicates a partition into three, three nodes are generated by the block partitioner 310 according to the horizontal partition into thirds 518 in an HTT CTU node generation step 626 .

The BT/TT split decision 618 indicates whether the vertical split is a split of two 516, indicated by returning a "0", or a split of three 520, indicated by returning a "1". If the BT/TT split 618 indicates a two-part split, then in the VBT CTU node generation step 627, two nodes are generated by the block partitioner 310 according to the vertical two-part split 516. If the BT/TT partition 618 indicates a partition into three, then three nodes are generated by the block partitioner 310 according to the vertical partition into thirds 520 in a VTT CTU node generation step 628 . For each node resulting from steps 625-628, the recursion of data flow 600 back to MT splitting decision 612 is applied in left-to-right or top-to-bottom order, depending on direction 614. As a result, binary tree and tertiary tree partitioning can be applied to generate CUs with various sizes.

The set of allowed and disallowed splits at each node of the encoding tree is further explained with reference to FIG.

7A and 7B provide an example partition 700 of a CTU 710 into several CUs or CBs. An example of the CU 712 is shown in FIG. 7A. FIG. 7A shows the spatial arrangement of CUs in CTU 710. Example partitioning 700 is also shown as encoding tree 720 in FIG. 7B.

For each non-leaf node in CTU 710 of FIG. 7A, e.g. nodes 714, 716, and 718, the accommodated nodes (which may be further divided and may be CUs) are used to create a list of nodes. are scanned or traversed in “Z order” and represented as columns in encoding tree 720. In the case of quadtree partitioning, the Z-order scan is from top left to right, then bottom left to right. For horizontal and vertical splits, the Z-order scan (traversal) simplifies to top-to-bottom and left-to-right scans, respectively. The encoding tree 720 of FIG. 7B lists all nodes and CUs according to the applied scan order. Each split generates a list of 2, 3, or 4 new nodes at the next level of the tree until a leaf node (CU) is reached.

A block partitioner 310 decomposes the image into CTUs, further decomposes the image into CUs, and uses the CUs to generate each residual block (324), as described with reference to FIG. is forward transformed and quantized by video encoder 114. The resulting TB 336 is then scanned as part of the operation of the entropy encoding module 338 to form a sequential list of residual coefficients. An equivalent process is executed within video decoder 134 to obtain the TB from bitstream 133.

The examples of FIGS. 7A and 7B illustrate coding trees applicable to both luma and chroma channels. However, the examples of FIGS. 7A and 7B also illustrate behavior with respect to traversal of a coding tree that is applicable only to luma channels or only applicable to chroma channels. For coding trees with many nested partitions, the partitioning options available at deeper levels are constrained by the limits of the available block size of the corresponding small region. The limit on the available block size for small regions is imposed to prevent the worst case of block processing rates so high as to impose an unreasonable burden on the implementation. In particular, the constraint that the block size be a multiple of sixteen (16) samples in chroma allows the implementation to process samples at a granularity of sixteen (16) samples. Limiting the block size to multiples of 16 samples creates an "intra-reconstruction" feedback loop, i.e., a path within video decoder 134 of FIG. Particularly relevant to equivalent paths within. In particular, limiting block sizes to multiples of sixteen (16) samples helps maintain throughput in intra-prediction mode. For example, "Simultaneous Data Multiple Instruction" (SIMD) microprocessor architectures typically operate on wide words that can contain 16 samples. Also, the hardware architecture may use a wide bus, such as a bus with a width of 16 samples, to transfer samples along the intra-reconstruction feedback loop. If a smaller block size is used, eg 4 samples, the bus will be underutilized, eg by one quarter of the bus width containing the sample data. An underutilized bus can handle smaller blocks (i.e. less than 16 samples), but in the worst case scenario, such as many blocks or all blocks of relatively small size, the underutilized bus will cause the encoder ( 114) or the real-time operation of the decoder (134). For inter prediction, each block depends on reference samples obtained from a frame buffer (such as buffer 372 or 496). Since the frame buffer is populated with reference samples when processing the previous frame, there are no feedback dependent loops that affect block-by-block operations to generate inter-predicted blocks. In addition to the feedback dependent loops associated with intra frame reconstruction, there is an additional simultaneous feedback loop associated with determining intra prediction mode 458. Intra prediction mode 458 is determined by selecting a mode from the most probable mode list or by selecting a mode from the remaining mode list. Determining the most probable mode list and the remaining mode list requires the intra-prediction modes of neighboring blocks. If a relatively small block size is used, the most probable mode list and the remaining mode list need to be determined more frequently, i.e. at a frequency governed by the sample block size and the channel sampling rate. .

8A, 8B, and 8C show an exemplary partitioning of CTU 800 (8A) by encoding tree 820 (Fig. 8B) with chroma partitioning terminated before luma partitioning and using a 4:2:0 chroma format. provide. When chroma splitting is finished, a pair of CBs are used, one for each chroma channel. For convenience of explanation, CTU800 with size 64x64 luma sample. CTU 800 is equal to a CTU size of 128x128 and a coding tree that includes one additional quadtree split. A quadtree decomposition is applied to the 8x8 luma region 814. The 8x8 luma region 814 is divided into four 4x4 luma CBs, but no division occurs in the chroma channels. Instead, chroma CB pairs of a predetermined minimum size (16 in the described example) are used, one corresponding to each chroma channel. A pair of chroma CBs is typically of a minimum size corresponding to the minimum granularity for the number of samples desired to be processed simultaneously. For example, many implementations of video encoder 114 and video encoder 134 operate on sets of 16 samples, eg, due to the use of correspondingly wider internal buses in the hardware implementation. Furthermore, each luma CB resulting from the splitting at least partially overlaps a pair of chroma CBs, and the aggregate luma CB completely overlaps a pair of chroma CBs. In the example of region 814, 4×4 chroma CB pairs are generated. FIG. 8C shows an example of how the resulting luma CB and chroma CB are related.

Referring again to 8A, a vertical bisection is applied to the 16×4 luma region 810. The 16x4 luma region 810 is divided into two 8x4 luma CBs, but no splitting occurs in the chroma channels, resulting in a pair of 8x2 chroma CBs. Vertical division into thirds is applied to the 16×4 luma region 812. The 16x4 luma region 812 is divided into 4x4, 4x8, and 4x4 luma CBs, but no splitting occurs in the chroma channels, resulting in a pair of 8x2 chroma CBs. Horizontal bisection is applied to an 8x16 luma region 816. The 8x16 luma region 816 is divided into 8x4, 8x8, and 8x4 luma CBs, but no splitting occurs in the chroma channels, resulting in a pair of 4x8 chroma CBs. Therefore, Chroma CB is at least 16 samples in area.

FIG. 8C shows a portion of a CTU 800 with three color planes shown in an "exploded" (or separated) manner to illustrate different block structures in different planes. A luma sample plane 850, a first chroma sample plane 852, and a second chroma sample plane 854 are shown. When the "YCbCr" color space is in use, the luma sample plane 850 contains the Y samples of the image frame, the first chroma sample plane 852 contains the Cb samples of the image frame, and the second chroma sample plane 854 contains the image frame's Cb samples. Contains Cr samples of the frame. Using a 4:2:0 chroma format, first chroma sample plane 852 and second chroma sample plane 854 will have half the sample density horizontally and vertically to luma sample plane 850. As a result, the CB dimensions of the chroma blocks in the sample are typically half the dimensions of the corresponding luma CBs. That is, for a 4:2:0 chroma format, the width and height of the chroma CB are half the width and height of the collocated luma CB, respectively. For the 4:2:2 chroma format, the height of the chroma CB is half the height of the collocated luma CB, and the width is the same as the width of the collocated luma CB. For clarity, only the parent split in the encoding tree of the 8x16 luma region 816 is shown, and the split is only shown in the luma sample plane 850. Once the chroma division is complete, multiple luma CBs are juxtaposed with a pair of chroma CBs. For example, the coding tree for CTU 800 includes horizontal third divisions applied to an 8x16 luma region 816. Horizontal division into thirds results in 8x4 luma CB 860, 8x8 luma CB 862, and 8x4 luma CB 864 that reside in luma sample plane 850. Since the 8x16 luma region 816 corresponds to an area of 4x8 chroma samples in the chroma sample planes (852 and 854), the 3 division of the encoding tree is not applied to the chroma sample planes (852 and 854). Therefore, the 4x8 chroma sample region forms a leaf node for chroma, resulting in a pair of chroma CBs: chroma CB 866 for the first chroma sample plane 852 and chroma CB 866 for the second chroma sample plane 854. Chroma CB868 is obtained. In the example of horizontal third division applied only to the luma plane, a minimum chroma CB size of 32 samples is achieved. Other example luma regions (810, 812, and 814) yield a minimum chroma CB size of 16, corresponding to the minimum luma block size and desired granularity of sample processing.

FIG. 9 shows a set 900 of transform block sizes and associated scan patterns for chroma channels resulting from the use of a 4:2:0 chroma format. Set 900 can also be used for 4:2:2 chroma formats. The described configuration is suitable for use with image frames having a chroma format in which the chroma channels of the image frame are subsampled relative to the luma channels of the image frame, particularly for 4:2:0 and 4:2:2 formats. suitable for Set 900 does not include all possible chroma transform block sizes. In FIG. 9, only chroma conversion blocks with a width of 16 or less or a height of 8 or less are shown. Chroma blocks with larger widths and heights may occur, but are not shown in FIG. 9 for ease of reference.

The set of prohibited transform sizes 910 includes transform block sizes 2x2, 2x4, and 4x2, all of which have areas less than 16 samples. In other words, in the example of FIG. 9, a minimum transform size of sixteen (16) chroma samples results from the operation of the described configuration, especially for the intra-predicted CB. Instances of prohibited transform sizes 910 are avoided by determining the splitting options as described with reference to FIG. The residual coefficients in the transform are scanned in a two-layer approach where the transform is divided into "subblocks" (or "coefficient groups"). The scan is performed along the scan path from the last valid (non-zero) coefficient to the DC (top left) coefficient. The scan path is defined as the progression within each subblock (the "lower layer") and from one subblock to the next (the "upper layer"). In set 900, 8x2 TB 920 uses 8x2 subblocks, ie, subblocks containing 16 residual coefficients. The 2x8 TB 922 uses 2x8 subblocks, i.e. also includes 16 residual coefficients.

A TB with a width or height of 2 and other dimensions that are multiples of 8 uses multiple 2x8 or 8x2 subblocks. Thus, in some examples a chroma block with a width of two samples is encoded using dividing the block into subblocks, each of size 2x8 samples and a height of two samples. A chroma block having a size of 8×2 samples each is encoded using dividing the block into subblocks in some examples. For example, a 16x2 TB 916 has two 8x2 subblocks, and each subblock is scanned as shown for TB 920. The progression of the scan from one subblock to the next, as shown in subblock progression 917.

2x32 TB (not shown in Figure 9) uses four 2x8 subblocks arranged as a 1x4 array. The residual coefficients within each subblock are scanned as shown for 2×8 TB 922, with the subblocks proceeding from the lowest to the highest subblock of the 1×4 array.

The larger the TB, the more similar scans will follow. For all TBs whose width and height are each greater than or equal to 4, a 4x4 subblock scan is used. For example, a 4x8 TB 923 uses a 4x4 subblock scan 924 with progression from the lower subblock to the upper subblock. A 4×4 TB925 can be scanned in a similar manner. The 8x8 TB 929 uses progression 930 for four 4x4 subblocks. In all cases, the scan within a subblock and the progression from subblock to subblock follows a backward diagonal scan, i.e. the scan starts from the "last" significant residual coefficient to Proceed backwards towards the upper left residual coefficients. FIG. 9 also shows the scan order across, for example, 8x4 TB 932, 16x4 TB 934, and 16x8 TB 936. Furthermore, depending on the position of the last significant coefficient along the scan path, only the part of the subblock that contains the last significant residual coefficient from the last significant coefficient position of the subblock back to the top left residual coefficient is scanned. There is a need to. Further sub-blocks along the scan path in the forward direction (ie closer to the bottom right of the block) do not need to be scanned. Set 900, and in particular prohibited transform sizes 910, impose limits on the ability to partition regions (or nodes) of the encoding tree in chroma into sub-regions (or sub-nodes), as explained with reference to FIG. .

In VVC systems using 2x2, 2x4, and 4x2 TBs (sets of TB910), 2x2 subblocks are used for TBs that are two samples wide and/or high. obtain. As mentioned above, the use of TB 910 increases throughput constraints in intra-reconfiguration feedback dependency loops. Furthermore, the use of subblocks with only four coefficients increases the difficulty of parsing the residual coefficients with higher throughput. In particular, for each subblock, a "significance map" indicates the significance of each residual coefficient contained therein. Encoding a one-valued significance flag establishes the magnitude of the residual coefficient as at least one, and encoding a zero-valued flag establishes the magnitude of the residual coefficient as zero. The magnitude and sign of the residual coefficients (from one step forward) are encoded only for those residual coefficients that are "significant." Significant bits are not encoded, and magnitude (from zero) is always encoded for DC coefficients. High-throughput encoders and decoders may need to encode or decode multiple significance map bins per clock cycle to maintain real-time operation. The difficulty of multi-bin encoding and decoding per cycle increases when there are more inter-bin dependencies, eg, when smaller subblock sizes are used. In system 100, the subblock size is 16 regardless of block size (with the exception of the subblock containing the last significant coefficient).

FIG. 10 shows a set of rules 1000 for generating a list of allowed splits in a chroma encoding tree. Other frames may allow for a mix of inter-predicted blocks and intra-predicted blocks. Although the full set of available partitions of the encoding tree has been described with reference to FIG. 6, the limitations on the available transform sizes impose constraints on the particular partitioning options for a given region size. As explained below, the splitting options for each chroma channel are determined according to the dimensions of the region of the corresponding coding tree unit.

Rules 1020 for chroma regions indicate the allowed division of different regions. The allowed divisions of rule 1020 are expressed in units of luma samples even though chroma channels are being considered since different chroma formats may be used.

Obtain the list of allowed splits for chroma by checking the availability of a set of split options with the region size of the coding tree when traversing the nodes of the coding tree. Split options that result in regions that may be encoded using CB are added to the list of allowed splits. For a region to be encoded using CB, the region size must allow encoding with an integer number of transforms of a particular size from set 900. The particular size is chosen to be the largest size (considering both width and height) that does not exceed the region size. Therefore, for smaller regions, a single transform is used. If the region size exceeds the size of the largest available transform, the largest available transform is tiled to occupy the entire region.

When considering a node in a coding tree with a given region (represented by luma samples), the ability to perform a given type of segmentation is determined according to the segmentation type and chroma region area. As shown in FIG. 10, the splitting option is tested against the region size to determine whether the splitting option results in a subregion of prohibited size. Splitting options that result in subregions of the allowed size are considered allowed chroma splitting 1070.

For example, if in QT mode (corresponding to decision 610 of FIG. 6), the region is of size 8x8 or 4:2:2 in 4:2:0 format, as shown as rule 1021a for chroma region. For the 8x8 format, quadtree splitting is not allowed because splitting results in a transform size of 2x2 or 2x4 for the chroma channels, respectively. Allowable area sizes are indicated by arrows 1021. Similarly, other acceptable divisions for chroma rule set 1020 are indicated by arrows 1022, 1023, 1024, 1025, and 1026 and are discussed below in connection with FIGS. 13 and 14. Arrows 1021, 1022, 1023, 1024, 1025 and 1026 each refer to the allowed chroma split list 1070.

The region size of the chroma channel is described in terms of the luma sample grid. For example, an 8x4 region corresponds to a 4x2 transformation of the chroma channels if a 4:2:0 chroma format is used. If a 4:2:2 chroma format is used, the 8x4 region corresponds to a 4x4 transformation of the chroma. When the 4:4:4 chroma format is used, chroma is not subsampled with respect to luma, so the transform size in chroma corresponds to the region size.

Acceptable splitting options are further described in connection with FIGS. 13 and 14 below.

FIG. 11 shows a method 1100 for encoding an encoding tree of image frames into a video bitstream. Method 1100 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Further, method 1100 may be performed by video decoder 114 under execution of processor 205. Accordingly, method 1100 may be stored on computer readable storage medium and/or memory 206. Method 1100 begins with step 1105 of determining a chroma format.

In determining chroma format step 1105, processor 205 determines the chroma format of frame data 113 as one of a 4:2:0 chroma format or a 4:2:2 chroma format. Chroma format is a property of the frame data and does not change during operation of method 1100. The method 1100 continues from step 1105 with step 1110 of dividing the frame into CTUs under the control of processor 205.

In step 1110 of partitioning the frame into CTUs, block partitioner 310, under execution of processor 205, partitions the current frame of frame data 113 into an array of CTUs. The progression of encoding over the CTUs resulting from the split begins. Control within the processor passes from step 1110 to step 1120 where an encoding tree is determined.

In determining a coding tree step 1120, video encoder 114, under execution of processor 205, tests various prediction modes and splitting options in combination to arrive at a coding tree of CTUs. Furthermore, the prediction mode and residual coefficients for each CU of the coding tree for the CTU are derived. Generally, Lagrangian optimization is performed to select the optimal coding tree and CU for the CTU. When evaluating the use of inter prediction, a motion vector is selected from a set of candidate motion vectors. Candidate motion vectors are generated according to the search pattern. When evaluating the distortion test of fetched reference blocks against candidate motion vectors, the application of forbidden chroma splitting in the encoding tree is taken into account. If segmentation is prohibited in chroma and allowed in luma, the resulting luma CB may use inter-prediction. Since motion compensation is applied only to the luma channel, the distortion calculation takes into account luma distortion and not chroma distortion. If chroma division was prohibited, no motion compensation would be performed on the chroma channel, so chroma distortion would not be taken into account. For chroma, the distortion resulting from the considered intra-prediction mode and the encoded chroma TB (if any) is taken into account. When considering both luma and chroma, an inter-predictive search may first select a motion vector based on luma distortion and then "refine" the motion vector by also considering chroma distortion. Refinement generally takes into account small variations in motion vector values, such as sub-pixel displacements. If chroma segmentation is prohibited and inter-prediction evaluation is performed on small luma blocks, chroma refinement is not required. Control within processor 205 passes from step 1120 to step 1130, which encodes the encoding tree.

In step 1130 of encoding a coding tree, video encoder 114, under execution of processor 205, performs method 1300 described in connection with FIG. encode. Step 1130 is performed to encode the current CTU into a bitstream. Control in processor 205 passes from step 1130 to a final CTU test step 1140.

In a last CTU test step 1140, processor 205 tests whether the current CTU is the last CTU in the slice or frame. If not (“NO” in step 1140), video encoder 114 advances to the next CTU in the frame and control within processor 205 returns from step 1140 to step 1120 to continue processing the remaining CTUs in the frame. do. If the CTU is the last CTU in the frame or slice, step 1140 returns "YES" and method 1100 ends. As a result of method 1100, an entire image frame is encoded into a bitstream as a sequence of CTUs.

FIG. 12 shows a method 1200 for decoding an encoding tree of image frames from a video bitstream. Method 1200 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Further, method 1200 may be performed by video decoder 134 under execution of processor 205. Accordingly, method 1200 may be stored on computer readable storage medium and/or memory 206. Method 1200 begins with step 1205 of determining a chroma format.

In determining chroma format step 1205, processor 205 determines the chroma format of frame data 113 as one of a 4:2:0 chroma format or a 4:2:2 chroma format. Chroma format is a property of the frame data and does not change during operation of method 1200. Video decoder 134 may determine the chroma format according to the profile of bitstream 133. A profile defines the set of encoding tools that may be used by a particular bitstream 133 and may constrain the chroma format to particular values, such as 4:2:0. The profile is determined, for example, by decoding a "profile_idc" syntax element from bitstream 133 or by decoding one or more constraint flags from bitstream 133, each constraint flag being a specific restrict the use of tools. If the chroma format is not fully specified by the profile, further syntax such as "chroma_format_idc" may be decoded to determine the chroma format. Method 1200 continues under execution of processor 205 from step 1205 to step 1210 of dividing the frame into CTUs.

In step 1210 of dividing the frame into CTUs, video decoder 134, under execution of processor 205, determines the division of the current frame of frame data 133 to be decoded into an array of CTUs. The decoding process begins over the CTUs resulting from the determined partition. Control within the processor passes from step 1210 to step 1220, which decodes the encoding tree.

In step 1220 of decoding the coding tree, video decoder 134 under execution of processor 205 performs method 1400 on the current CTU to decode the coding tree of the current CTU from bitstream 133. The current CTU is the selected one of the CTUs resulting from the execution of step 1210. Control in processor 205 passes from step 1220 to a final CTU test step 1240 .

In a last CTU test step 1240, processor 205 tests whether the current CTU is the last CTU in the slice or frame. If not (“NO” in step 1240), video decoder 134 advances to the next CTU in the frame and control within processor 205 returns from step 1240 to step 1220 to continue decoding CTUs from the bitstream. If the CTU is the last CTU in the frame or slice, step 1240 returns "YES" and method 1300 ends.

FIG. 13 shows a method 1300 for encoding an encoding tree of image frames into a video bitstream. Method 1300 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Further, method 1300 may be performed by video encoder 114 under execution of processor 205. Accordingly, method 1300 may be stored on computer readable storage medium and/or memory 206. Method 1300 encodes blocks into bitstream 115 such that each block is in a minimal area. The described configuration uses a predetermined minimum size sample. The minimum size used in the example described is 16 samples, which is preferred from some hardware and software implementation perspectives. However, different minimum sizes can nevertheless be used. For example, a processing granularity of 32 or 64 and a corresponding minimum block area of 32 or 64 samples, respectively, are possible. A coded block with minimal area is advantageous for implementation feasibility, both in hardware and software implementation. For a software implementation, a minimum area of 16 samples aligns with typical single instruction multiple data (SIMD) instruction sets such as AVX-2 and SSE4. The method 1300, first invoked at the root node of the current CTU's encoding tree, begins with step 1310 of encoding a split mode.

In step 1310 of encoding the splitting mode, entropy encoder 338 encodes the splitting mode at the current node of the encoding tree into bitstream 115 under execution of processor 205 . The split mode is one of the splits as explained with reference to FIG. 5, and the step of encoding the split mode allows only possible splits to be encoded. For example, quadtree split 512 is only possible at the root node of the coding tree or below other quadtree splits in the coding tree. As shown with respect to set 910, splits that result in luma CBs having a width or height of less than 4 samples are prohibited. For example, based on rule set 1010, other constraints regarding the maximum depth of bisection and/or triplication may also be valid. Control in processor 205 passes from step 1310 to test no split step 1320 .

At test no split step 1320, processor 205 tests whether the current split is "no split" (ie, 510). If the current partition is no partition 510 (“YES” in step 1320), control of processor 205 proceeds from step 1320 to step 1330 where the CU is encoded. Otherwise, if the current division is not 510 (“NO” in step 1320), control of processor 205 continues to chroma division inhibit test step 1340.

In step 1330 of encoding the CU, entropy encoder 338 encodes the CU's prediction mode and the CU's residual into bitstream 115 under execution of processor 205 . Once step 1330 is reached at each leaf node of the encoding tree, method 1300 ends with a completion step 1330 and returns to the parent call in the encoding tree traversal. Once all nodes of the encoding tree have been traversed, the entire CTU is encoded into bitstream 115 and control returns to method 1100 to proceed to the next CTU in the image frame.

In a chroma splitting prohibition test step 1340, the processor 205 determines whether splitting for the current node in the encoding tree is allowed to be applied to the chroma channel, as in step 1310, according to the chroma region 1020 splitting rule set of FIG. Determine whether there is. If the current node in the encoding tree covers a luma region of 128 luma samples (32x4 or 4x32 or 16x8 or 8x16), then the corresponding chroma region (16x2, 2 x16, 8x4, 4x8 chroma samples) is prohibited as shown in rule set 1020. If splitting into three is allowed, the resulting block size will include the prohibited block size (eg, 2x4 or 4x2). If the current node in the encoding tree covers a luma region of 64 luma samples, 2-split, 3-split, and quadtree splits are prohibited, as shown in rule set 1020. If a luma region of 64 luma samples is divided into two, divided into three, or quadtree divided, the chroma block size becomes prohibited (2×2, 2×4, 4×2). If the split is not prohibited (i.e., if the split is a permitted chroma split in list 1070), step 1340 returns "NO" and control of processor 205 continues from step 1340 to perform luma and chroma splits. Proceed to 1350. If not, and if division is prohibited (“YES” in 1340), control of processor 205 proceeds to step 13100 where luma division is performed.

In step 1350 of performing luma and chroma partitioning, processor 205 applies partitioning to partition the current region associated with the current node of the encoding tree into subregions associated with subnodes of the encoding tree. The splitting is applied according to the description of FIGS. 5 and 6. Control within processor 205 passes from step 1350 to step 1360 where a region is selected.

In select region step 1360, the processor selects one of the sub-regions resulting from step 1350. Sub-regions are selected according to a Z-order scan of the region. The selection progresses through the subregions in subsequent iterations of step 1360. Control within processor 205 passes from step 1360 to step 1370, which encodes the encoding tree.

In step 1370 of encoding the encoding tree, processor 205 recursively invokes method 1300 for the selected region resulting from step 1360. Step 1370 further operates to encode luma and chroma blocks and associated prediction modes and residual coefficients for each region of the bitstream. Control in processor 205 passes from step 1370 to a final area test step 1380.

In a final region test step 1380, processor 205 tests whether the selected region selected in step 1360 is the last one of the regions obtained from the split mode segmentation to be performed in step 1350. If the region is not the last region (“NO” in step 1380), control in processor 205 passes from step 1380 to step 1360 to continue advancing through the region of division; otherwise, step 1380 returns “YES” and the method 1300 ends and control in processor 205 passes to the parent invocation of method 1300.

In step 13100 of performing luma splitting, the splitting mode as encoded in step 1310 is performed on the luma channel by processor 205 only. As a result, the current node of the coding tree is split into multiple luma CBs according to the splitting mode. Only pairs of chroma CBs are generated, ie, one chroma CB per chroma channel. Each resulting luma CB partially overlaps (juxtaposes) the pair of chroma CBs and the collectively resulting luma CB. The aggregate luma CB exactly covers the area of the chroma CB pair. Chroma CB pair area. Further, the minimum area of each luma CB and chroma CB is the minimum size, for example, 16 samples.

Steps 13100 and 1350 operate to determine the size of chroma encoded blocks for chroma channels Cb and Cr, respectively. In step 1350, the chroma encoding block size of the chroma channel is determined based on the division mode determined in step 1310. At step 13100, a chroma encoding block size for a chroma channel is determined based on a predetermined minimum chroma block size. As mentioned above, step 1350 is performed based on prohibited chroma splitting for the coding tree unit. As shown in rule set 1020 of FIG. 10, the allowable partitioning, and thus the size of the chroma encoded block, is determined based on the chroma format determined in step 1105.

Control within processor 205 passes from step 13100 to step 13110 where a luma CB is selected.

In step 13110 of selecting a luma CB, processor 205 selects the luma CB next to the CB obtained from step 13100. The method 13100 first selects the first CB, ie, the upper left luma CB of the CB resulting from the luma split. Upon subsequent activation of step 13110, each "next" luma CB is selected according to the Z-ordered scan across the luma CB resulting from step 13100. Control in processor 205 passes from step 13110 to step 13120, which encodes the luma CB.

In a step of encoding luma CB 13120 , entropy encoder 338 encodes the selected luma CB into bitstream 115 under execution of processor 205 . Generally, the prediction mode and residual coefficients are encoded for the selected luma CB. The encoded prediction mode for luma CB may use inter prediction or intra prediction. For example, "cu_skip_flag" is encoded to indicate the use of inter-prediction without residuals, otherwise "pred_mode_flag" and optionally "pred_mode_ibc_flag" are respectively coded to indicate the use of inter-prediction with optional residual coefficients. , inter-prediction, or encoded to indicate the use of intra-block copy. If residuals may be present, the "cu_cbf" flag indicates the presence of at least one significant (non-zero) residual coefficient in any TB of the CB. If the CB is instructed to use inter-prediction, the associated motion vectors are only applicable to the luma CB. That is, motion vectors are also not applied to generate PBs associated with partially collocated chroma CBs. When a CB is instructed to use intrablock copying, the associated block vectors are associated only with luma CBs and not with partially collocated chroma CBs. Control in processor 205 passes from step 13120 to a final luma CB test step 13130.

Last Luma CB Test In step 13130, processor 205 tests whether the luma CB selected in step 13110 is the last luma CB according to the Z-ordered iteration of the luma CB of the split performed in step 13100. If the selected luma CB is not the last one (“NO” in step 13130), control of processor 205 proceeds from step 13130 to step 13120. If not, step 13130 returns "YES" and control of processor 205 proceeds to step 13140, which determines the chroma intra prediction mode.

In determining chroma intra prediction mode 13140, video encoder 114, under execution of processor 205, determines an intra prediction mode for the luma CB of step 13100 and the co-located chroma CB pair. Step 13140 effectively determines that the chroma block is encoded using intra prediction. A determination is made whether the area occupied by the chroma CB is further divided into multiple luma CBs in the luma channel. The size of the chroma block for the channel is a predetermined minimum value (eg, 16 samples) determined by the operation of step 1350. In step 13120, an intra-prediction mode is determined for the pair of chroma CBs even if the corresponding luma CBs were encoded using inter-prediction. In one configuration, a single prediction mode, such as DC intra prediction, is applied to each chroma CB. The use of a single prediction mode allows the mode to be determined by prohibiting chroma splitting (a "YES" result in step 1340) and determines which one of multiple possible modes should be used. No additional exploration is required to determine which. Furthermore, the bitstream 115 does not require any additional signaling in this case, ie no additional "intra_chroma_pred_mode" syntax element needs to be encoded. However, when chroma splitting is prohibited (“YES” in step 1340), the configuration allows intra prediction of one of several possible intra prediction modes by including an “intra_chroma_pred_mode” syntax element in the bitstream 115. Higher compression performance can be achieved by signaling the prediction mode. Video encoder 114 determines which intra prediction mode to use. Intra prediction modes are generally determined according to considerations of encoding cost compared to distortion. However, higher compression performance is generally obtained compared to using a single intra-prediction mode for such chroma CBs. Control in processor 205 passes from step 13140 to step 13150, which encodes the chroma CB.

In step 13150 of encoding the chroma CB, the entropy encoder 338, under execution of the processor 205, uses the "intra_chroma_pred_mode" syntax element to perform intra prediction of the chroma CB when multiple intra prediction modes are available. The mode is encoded into the bitstream 115. When one intra-prediction mode is possible, for example DC intra-prediction, “intra_chroma_pred_mode” is not encoded into the bitstream 115. Available intra prediction modes for chroma intra prediction may include DC, planar, and the following angular prediction modes: horizontal, vertical, top right diagonal. The available intra-prediction modes may also include a “direct mode” (DM_CHROMA), whereby the chroma intra-prediction mode extracts the most of the resulting luma CB from step 13100, generally from the co-located luma CB. Retrieved from the bottom and the rightmost. If "cross-component linear model" intra-prediction is available, chroma CB may be predicted from samples from luma CB. As described with reference to step 14150 of FIG. 14, the residual coefficients of chroma TB associated with chroma CB may also be encoded into bitstream 115. Once step 13150 is executed by processor 205, method 1300 ends and control within processor 205 returns to the parent invocation of method 1300.

FIG. 14 shows a method 1400 of decoding an encoding tree of image frames from a video bitstream, performed in step 1220 of method 1200. Method 1400 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Further, method 1400 may be performed by video decoder 134 under execution of processor 205. As such, method 1400 may be stored on computer readable storage medium and/or memory 206. Method 1400 decodes blocks from bitstream 133 such that each block is no smaller than a minimum area, such as 16 samples, which is advantageous for implementation feasibility in both hardware and software cases. become. For software, a minimum area of 16 samples aligns with typical single instruction multiple data (SIMD) instruction sets such as AVX-2 and SSE4. The method 1400, which is first activated at the root node of the encoding tree of the current CTU, begins with step 1410 of decoding the split mode.

In step 1410 of decoding split modes, entropy decoder 420 , under execution of processor 205 , decodes the split modes at the current node of the encoding tree into bitstream 133 . The split mode is one of the splits as described with reference to FIG. 5, and the method of encoding the split mode is allowed even if split is prohibited in the chroma channel. That is, only the encoding of the divisions allowed in the luma channel is allowed. For example, quadtree split 512 is only possible at the root node of the coding tree or below other quadtree splits in the coding tree. Splitting that results in a luma CB with a width or height of less than 4 samples is prohibited. Therefore, the minimum luma CB size is 16 samples. Other constraints on the maximum depth of bisection and/or triplication may also be valid. Control in processor 205 passes from step 1410 to test no split step 1420 .

In a no split test step 1420, processor 205 tests whether the current split is "no split" (ie, 510). If the current partition is no partition 510 (“YES” at 1420), control of the processor 205 proceeds from step 1420 to step 1430 where the CU is decoded. If not, step 1420 returns "NO" and processor 205 control passes to chroma splitting inhibit test step 1440.

In step 1430 of decoding a CU, entropy decoder 420 , under execution of processor 205 , decodes the prediction mode of the CU and the residual coefficients of the CU of bitstream 115 . Step 1430 operates to decode the coding unit using the residual coefficients and prediction mode determined from the bitstream by entropy decoder 420. Once step 1430 is reached at each leaf node of the encoding tree, method 1400 ends upon completion of step 1430 and returns to the parent call in the encoding tree search. Once all nodes of the encoding tree have been traversed, the entire CTU is decoded from bitstream 133 and control returns to method 1200 to proceed to the next CTU in the image frame.

In a chroma splitting prohibition test step 1440, the processor 205 determines whether splitting for the current node in the encoding tree is allowed to be applied to the chroma channels, as in step 1410, according to the chroma region 1020 splitting rule set of FIG. Determine whether there is. Step 1440, similar to step 1340 of method 1300, determines whether split testing is prohibited. The act of step 1440 prevents the occurrence of prohibited block sizes. If the chroma region is already at a minimum size, for example 16 chroma samples, any type of further division is not allowed as the resulting region is smaller than the allowed minimum. If the chroma region size is 32 samples and the corresponding division is 3 divisions (whether horizontal or vertical 3 divisions), further divisions are also not allowed to avoid chroma blocks of 8 chroma samples in the region. . If splitting is not prohibited (ie, splitting is allowed), step 1450 returns "NO" and control of processor 205 passes from step 1440 to step 1450, which performs luma and chroma splitting. Otherwise, if segmentation is prohibited (“YES” in step 1450), control of the processor 205 proceeds to step 14100 in which a chroma intra prediction mode is determined.

In step 1450 of performing luma and chroma partitioning, processor 205 applies partitioning to partition the current region associated with the current node of the encoding tree into subregions associated with subnodes of the encoding tree. The splitting is applied as described in connection with FIGS. 5 and 6.

Steps 14100 and 1450 operate to determine the size of chroma encoded blocks for chroma channels Cb and Cr, respectively. In step 1450, the chroma encoding block size of the chroma channel is determined based on the division mode decoded in step 1410. At step 14100, a chroma encoding block size for a chroma channel is determined based on a predetermined minimum chroma block size. As mentioned above, step 1450 is based on the chroma splitting that is prohibited for the encoding tree unit, corresponding to a minimum chroma CB size of 16 (and 32 in the case of 3 splits of the luma region 128 samples). Implemented. As shown in rule set 1020 of FIG. 10, the allowable partitioning, and thus the size of the chroma encoded block, is determined based on the chroma format determined in step 1205.

Control within processor 205 passes from step 1450 to region selection step 1460.

In a region selection step 1460, processor 205 selects one of the sub-regions resulting from step 1450 according to the Z-order scan of the region. Step 1460 manipulates the progression selection through the subregions in subsequent iterations. Control within processor 205 passes from step 1460 to step 1470, which decodes the encoding tree.

In step 1470 of decoding the encoding tree, processor 205 recursively invokes method 1400 for the selected region resulting from the operation of step 1460. Step 1470 further operates to decode each region of the encoding tree using the residual coefficients and prediction mode determined from the bitstream. Control in processor 205 passes from step 1470 to a final area test step 1480.

In a final region test step 1480, the processor 205 determines whether the selected region is the last one of the regions resulting from the split mode partitioning performed in step 1450, as preselected in the last iteration of step 1460. test. If the region is not the last region (“NO” in step 1480), control of processor 205 passes from step 1480 to step 1460 and continues to advance through the regions of the division. If not, step 1480 returns "YES" and method 1400 ends and processor 205 control passes to the parent invocation of method 1400.

In step 14100 of performing luma splitting, the splitting mode as encoded in step 1410 is performed on the luma channel by processor 205 only. As a result, the current node of the coding tree is split into multiple luma CBs according to the splitting mode. Step 14100 operates to generate only chroma CB pairs, ie, one chroma CB per chroma channel. Each resulting luma CB partially overlaps (at least partially co-locates with) a pair of chroma CBs, and collectively, luma CBs completely overlap a pair of chroma CBs. Further, the minimum area of each luma CB and chroma CB is 16 samples. Control within processor 205 passes from step 14100 to step 14110, which selects a luma CB.

In step 14110 of selecting a luma CB, processor 205 selects the luma CB next to the CB obtained from step 14100. The selection of the next luma CB starts from the top left luma CB of the first CB, ie the CB resulting from the luma split. On subsequent invocations of step 14110, each "next" luma CB is selected according to the Z-order scan over the luma CB resulting from step 14100. Control within processor 205 passes from step 14110 to step 14120, which decodes the luma CB.

In step 14120 of decoding luma CB, entropy decoder 420 decodes the selected luma CB into bitstream 115 under execution of processor 205 . Generally, the prediction mode and residual are decoded for the selected luma CB. For example, "cu_skip_flag" is decoded to indicate the use of inter-prediction without residuals, otherwise "pred_mode_flag" and optionally "pred_mode_ibc_flag" are used for intra-prediction, inter-prediction with optional residual coefficients, respectively. , or decoded to indicate the use of intrablock copy. If residuals may exist, the "cu_cbf" flag indicates the presence of at least one significant (non-zero) residual coefficient in any TB of the CB. When a CB is instructed to use inter-prediction, the associated motion vectors are only applicable to luma CBs, i.e. the motion vectors are used to generate PBs that are associated with partially collocated chroma CBs. It also does not apply to When a CB is instructed to use intrablock copying, the associated block vectors are associated only with luma CBs and not with partially collocated chroma CBs. Control in processor 205 passes from step 14120 to step 14130, where the last luma CB is tested.

In step 14130 of testing the last luma CB, the processor 205 tests whether the luma CB selected in step 14110 is the last luma CB according to the Z-ordered iteration of the luma CB of the split performed in step 14100. . If the selected luma CB is not the last one, control within processor 205 proceeds from step 14130 to step 14110. If not, control in processor 205 proceeds to step 14140 where a chroma intra prediction mode is determined.

In determining chroma intra-prediction mode 14140, video decoder 134, under execution of processor 205, determines an intra-prediction mode for the luma CB of step 14100 and the co-located chroma CB pair. Step 14140 includes stopping the splitting of the coding tree for chroma while the chroma block is splitting the coding tree for luma, as determined by the operation of step 1440. If so, it effectively determines that the chroma block was encoded using intra-prediction and therefore should be decoded using intra-prediction. In step 14120, an intra prediction mode for the pair of chroma CBs is determined even if the corresponding luma CB was decoded using inter prediction. In one configuration, a single prediction mode, such as DC intra prediction, is applied to each chroma CB. The use of a single prediction mode allows the mode to be determined by prohibiting chroma splitting (a "YES" result in step 1440) and determines which one of multiple possible modes should be used. No additional exploration is required to determine which. Furthermore, the bitstream 134 does not require any additional signaling in this case, ie, no additional "intra_chroma_pred_mode" syntax element needs to be encoded. However, when chroma splitting is prohibited (“YES” in step 1440), the configuration can be configured to enable one of several possible intra prediction modes by including an “intra_chroma_pred_mode” syntax element in the bitstream 134. Higher compression performance can be achieved by signaling the prediction mode. Video decoder 134, using entropy decoder 420, must determine the intra prediction mode to be used to decode the “intra_chroma_pred_mode” syntax element from bitstream 134. Control in processor 205 proceeds from step 14140 to step 14150, where the chroma CB is decoded.

In step 14150 of decoding a chroma CB, entropy decoder 420, under execution of processor 205, determines the intra prediction mode of the chroma CB from bitstream 420, generally according to the decoded "intra_chroma_pred_mode" syntax element. Decoding of "intra_chroma_pred_mode" is performed when multiple intra prediction modes are available. If only one intra-prediction mode is available, e.g. DC intra-prediction, the mode is inferred from the bitstream 133 without decoding additional syntax elements. Intra prediction modes available for chroma intra prediction may include DC, planar, and the following angular prediction modes: horizontal, vertical, top right diagonal. The available intra-prediction modes may also include a “direct mode” (DM_CHROMA), whereby the chroma intra-prediction mode is configured to perform a Obtained from juxtaposed luma CB. If "cross-component linear model" intra-prediction is available, chroma CB may be predicted from samples from luma CB. For a pair of chroma CBs, the "cu_cbf" flag signals the presence of at least one significant residual coefficient in any one of the pair of chroma CBs. “tu_cbf_cb” and “tu_cbf_cr” indicate the presence of at least one significant coefficient in the chroma CB of the Cb and Cr channels, respectively, if at least one significant residual coefficient is present in any one of the pair of chroma CBs. signal. For chroma CBs that have at least one significant residual coefficient, a "residual_coding" sequence of syntax elements is decoded to determine the residual coefficient for each chroma CB. The residual encoding syntax encodes the residual coefficients as a sequence of values that populates the transform block from the last significant coefficient position to the top left ("DC") coefficient position according to a backward diagonal scan. A backward diagonal scan scans "subblocks" (or "coefficient groups"), typically of size 4x4, but also of size 2x2, 2x4, 2x8, 8x2, 4x2. Perform scanning of transform blocks as a sequence. The scans within each coefficient group are back diagonal, and the scans from one subblock to the next are also back diagonal. Once step 14150 is executed by processor 205, method 1400 ends and control within processor 205 returns to the parent invocation of method 1400.

The coding tree approach of methods 1300 and 1400 maintains a minimum block area of 16 samples for 4:2:0 chroma format video data, facilitating high-throughput implementation in both software and hardware. Additionally, the limitation of inter-prediction for luma CBs for small CB sizes reduces this worst-case scenario to motion-compensated memory bandwidth by avoiding the need to also fetch samples to generate motion-compensated chroma CBs. Reduce memory bandwidth. In particular, if the minimum chroma CB size is 2x2 and additional samples are required to provide filter support for subsample interpolation of the chroma CB, inter-prediction in the luma channel for small block sizes is There is a substantial increase in memory bandwidth compared to just running. The coding gain of motion compensation appears substantially within the luma channel; therefore, omitting small blocks from being motion compensated also achieves a reduction in memory bandwidth for a relatively small coding performance impact. do. Furthermore, the reduction in memory bandwidth contributes to the feasibility of performing motion compensation on 4x4 luma CBs and achieving the resulting coding gain.

In one configuration of video encoder 114 and video decoder 134, two or more luma splits may occur within the encoding tree from the point where the chroma split of the encoding tree ends. For example, an 8x16 luma region is not divided in chroma channels, resulting in a 4x8 chroma CB pair. In the luma channel, the 8x16 luma region is first divided into horizontal thirds, and then one of the resulting luma CBs is further divided. For example, the resulting 8x4 luma CB is vertically bisected into two 4x4 luma CBs. Configurations with more than one luma split in the coding tree from the point where the chroma split of the coding tree ends restarts methods 1300 and 1400 in video encoder 114 and video decoder 134, respectively, in the chroma splitting prohibited region. , with the modification that no further chroma CBs are required on subsequent calls. In calls to methods 1300 and 1400 in which a pair of chroma CBs are created, the entire chroma region is covered by the created chroma CBs, so recursive calls to methods 1300 and 1400 do not need to create additional chroma CBs. .

FIG. 15 shows a set 1500 of transform block partitions of an intra predictive coding unit. The luma CB may be divided into one luma TB (“ISP_NO_SPLIT”) of the same size. A luma CB with a size of 4×4 has an area of 16 samples, and is not further divided into one luma TB with a size of 4×4. A luma CB with an area of 32 samples may be divided into two sections. For example, an 8×4 Luma CB1510 may be split horizontally (“ISP_HOR_SPLIT”) into two 8×2 Luma TB1520 or vertically (“ISP_VER_SPLIT”) into two 4×4 Luma TB1530. good. If the luma CB 1510 is a 4x8 luma CB, the block may be divided horizontally into two 4x4 luma TBs at 1520 or vertically into two 2x8 luma TBs at 1530.

The luma CB in the area of 64 samples or more is divided into one section into four sections. A Luma CB1550 with a larger area of 64 samples of width W and height H may be horizontally divided into four Luma TB1560 of size Wx(H/4), or four (W/4)xH The luma TB may be divided vertically. As shown in set 1500, dividing luma CB into multiple sections makes luma TB smaller and smaller. Intra prediction is performed to generate a PB for each luma TB, and an intra reconstruction process is performed within the luma CB from one partition to the next.

FIG. 16 shows a method 1600 for encoding encoded units of image frames into a video bitstream 115. Method 1600 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Additionally, method 1600 may be performed by video encoder 114 under execution of processor 205. As such, method 1600 may be stored on computer readable storage medium and/or memory 206. Method 1600 will encode blocks into bitstream 115 such that coefficient group sizes are determined based solely on transform block sizes and do not further differentiate between luma and chroma channels. Since entropy encoding is a critical feedback loop in video encoder 114, it is advantageous to reduce the memory accesses or operations required for coefficient group size determination. The method 1600, called for each coding unit in the coding tree, ie, called in step 1330 of FIG. 13, begins with step 1610 of encoding pred_mode. As mentioned above, step 1330 is performed if step 1320 determines that the current split is no split 510.

In step 1610 of encoding pred mode, entropy encoder 338 encodes the prediction mode of the CU into bitstream 115 under execution of processor 205 . Control in processor 205 passes from step 1610 to intra-prediction test step 1620 .

In an intra-prediction test step 1620, the processor 205 tests the prediction mode of the CU. If the prediction mode is intra prediction (“YES” in step 1620), control of processor 205 proceeds from step 1620 to step 1650, where the intra subdivision mode is encoded. Otherwise, if the prediction mode is not intra-prediction (“No” in step 1620), control of processor 205 passes from step 1620 to step 1630, where the merge flag and index are encoded.

In step 1630 of encoding merge flags and indices, entropy encoder 338, under execution of processor 205, inserts a merge flag into bitstream 115 that signals the use (or non-use) of "merge mode" for inter prediction. encode. The merge mode causes the motion vector of a CU to be obtained from a spatially (or temporally) adjacent block among a candidate set of spatially (or temporally) adjacent blocks. If merge mode is used, one candidate is selected with the corresponding "merge index". The merge index is encoded within the bitstream 115 along with the merge flag. When "motion vector prediction" is used, a similar encoding is performed whereby one of several possible candidate motion vectors is signaled as a predictor using a flag. Control in processor 205 passes from step 1630 to step 1640, where the motion vector delta is encoded.

In step 1640 of encoding motion vector deltas, entropy encoder 338 encodes the motion vector deltas into bitstream 115 under execution of processor 205 . Step 1640 is performed if motion vector prediction is used for the CU. Motion vector delta specifies the delta between the motion vector predictor encoded in step 1630 and the motion vector used for motion compensation. Control in processor 205 passes from step 1640 to encoded residual test step 1660. If motion vector prediction is not used for the CU, step 1640 is not performed and method 1600 proceeds directly to step 1660.

In step 1650 of encoding intra-subpartitioning mode, entropy encoder 338, under execution of processor 205, uses the context encoding "Intra_subpartitions_mode_flag" syntax element to determine whether intra-subpartitioning is to be used for bitstream 115. encode. Intra subdivision is available for luma channels when the luma CB size is larger than the minimum luma transform block size, ie larger than 16 luma samples. Intra subpartitioning partitions the coding unit into multiple luma transform blocks, as shown in set 1500. If a luma CB is split into multiple TBs, "intra_subpartitions_split_flag" signals whether the split of the luma CB into multiple luma TBs occurs horizontally or vertically. Collectively, "intra_subpartitions_mode_flag" and "intra_subpartitions_split_flag" are divided into three possible values, listed as "ISP_NO_SPLIT", "ISP_HOR_SPLIT" and "ISP_VER_SPLIT". Encode the split. Control in processor 205 passes from step 1650 to encoded residual test step 1660.

In a coding residual testing step 1660, processor 205 determines whether at least one residual coefficient in any transform block of the coding block is significant. This determination includes all luma TBs resulting from the application of intra-subdivision and the chroma TB pairs associated with the two chroma channels. If at least one residual coefficient in either luma TB and chroma TB is significant, entropy encoder 338, under execution of processor 205, arithmetically encodes a “1” for the “cu_cbf” syntax element; Step 1660 returns "YES" and processor 205 proceeds to step 1670 where it determines the luma coefficient group size. If there are no significant residual coefficients in any TB of the CU, step 1660 returns "NO", a "0" is arithmetic encoded for cu_cbf, method 1600 ends, and processor 205 Proceed to the next CU within.

In determining luma coefficient group size step 1670, the processor determines the coefficient group size of one or more luma TBs (transform blocks) associated with the CU. If intra-subpartitioning is not used, one luma TB exists. If intra-subpartitioning is used, there are two or four luma TBs. The size of the luma TB depends on the intra subdivisions being performed horizontally or vertically, and the number of luma TBs, and thus on the luma CU size, as shown in set 1500.

The coefficient group size is determined using the luma TB width and height as shown in Table 1 below. Table 1 shows the transform block (TB) sizes for coefficient group mapping tables for luma and chroma channels with coefficient groups of the same size for the TB, regardless of whether the TB is for a luma channel or a chroma channel. TB width and height are powers of 2, so Table 1 is the log2 of TB width and height, i.e. "log2TBwidth" and "log2TBheight" form the first two indexes into the third dimension of Table 1 Take that into consideration. The final dimensions of the table distinguish between the width and height of the coefficient groups. The dimensions of the coefficient group are stored as log2 width and log2 height. For example, a TB of size 16x16 yields index (4,4) of table 1, which returns (2,2) indicating a coefficient group size of 4x4. A TB of size (2x32) yields index (1,5) of table 1, which returns (1,3) indicating a coefficient group size of 2x8. Since the minimum area of the luma TB is 16 samples, if log2width+log2height in Table 1 is less than 4, it will not be accessed. If intra-subpartitioning is used for CU, each luma TB has the same size, so the coefficient group sizing for luma TB is performed once for CU.

Table 2 below shows that compared to chroma, luma maps transform block (TB) size to coefficient group size for luma and chroma channels with different coefficient group sizes for the same size TB. If Table 2 is used, an additional dimension is required, namely to distinguish between luma and chroma, and the table size is twice as large compared to Table 1. The coefficient group size defined in Table 1 fits within the TB width and height, but results in a size that is the largest possible size that does not exceed 16 samples in area. Table 1 provides a set of coefficient group sizes from which the coefficient group size is selected. The selected coefficient group aspect ratio of width to height is kept as close to 1:1 as possible within the TB width and height constraints. Control within processor 205 passes from step 1670 to step 1680, which encodes the luma TB.

In a step of encoding luma TBs 1680, entropy encoder 338, under execution of processor 205, encodes the residual coefficients of one or more luma TBs of the CU into bitstream 115. The determined coefficient group size of step 1670 is used for each luma TB. For each luma TB, a coded block flag is encoded into the bitstream 115 indicating the presence of at least one significant coefficient within the luma TB. If there is at least one significant coefficient in the luma TB, the last significant position is encoded into the bitstream. The last significant position is defined as the last significant coefficient along the scan path going from the DC (top left) coefficient of TB to the bottom right coefficient. A scan path is defined as a diagonal scan in dividing the TB into an array of non-overlapping sub-blocks, each sized as the coefficient group size and occupying the entire TB. The progression from one subblock to the next in the scan order also follows a diagonal scan. Entropy encoder 338 encodes an "encoding subblock flag" for each coefficient group other than the upper left coefficient group and the coefficient group containing the last significant coefficient. The encoded subblock flag indicates the presence of at least one significant residual coefficient within the subblock. If there are no significant residual coefficients in a subblock, the diagonal scan of residual coefficients in TB skips that subblock. If there is at least one significant residual coefficient in a subblock, all positions in that subblock are scanned, the magnitude of each residual coefficient is encoded, and the sign of each significant residual coefficient is encoded. Ru. Control in processor 205 passes from step 1680 to step 1690, where the chroma coefficient group size is determined.

In determining chroma coefficient group size step 1690, processor 205 determines the coefficient group size for the pair of chroma transform blocks associated with the CU. One chroma CB for each chroma channel is associated with a CU, regardless of whether the luma CB is divided into multiple luma TBs. The coefficient group size is determined using the chroma TB width and height, as shown in Table 1. TB width and height are powers of 2, so Table 1 is the log2 of TB width and height, i.e. "log2TBwidth" and "log2TBheight" form the first two indexes into the third dimension of Table 1 Take that into account. The final dimensions of the table distinguish between the width and height of the coefficient groups. The dimensions of the coefficient group are stored as log2 width and log2 height. For example, a TB of size 16x16 yields index (4,4) of table 1, which returns (2,2) indicating a coefficient group size of 4x4. A TB of size (2x32) yields index (1,5) of table 1, which returns (1,3) indicating a coefficient group size of 2x8. Since each chroma TB has the same size, coefficient group size determination for a pair of chroma TBs is performed once for a CU. If table 2 is used, an additional dimension is required, ie a dimension to distinguish between luma and chroma, and the table size is twice compared to that of table 1.

As described in connection with steps 1670 and 1690, the coefficient group size is determined based solely on the transform block size and no further differentiation is made between luma and chroma channels. Therefore, the coefficient group size is determined regardless of whether the chroma format is 4:2:2 or 4:2:0. As explained in connection with Table 1, the coefficient group size is based on the maximum area of the coefficient group up to 16 samples. Step 1690 determines the coefficient group size for TB due to the chroma format, independent of the transform block or subsampling color plane (Y or Cb or Cr) in the color plane (applicable to Cb and Cr channels). Works for. Table 1 is used in both step 1670 and step 1690. Therefore, a single table is used for each transform block belonging to the luma plane and the chroma color plane. Control in processor 205 passes from step 1690 to step 16100, which encodes the chroma TB.

In step 16100 of encoding chroma TBs, entropy encoder 338 encodes the residual coefficients of the CU's chroma TB pairs into bitstream 115 under execution of processor 205 . The determined coefficient group size of step 1690 is used for the pair of chroma TBs. For each chroma TB, a coded block flag indicating the presence of at least one significant coefficient in the chroma TB is encoded into the bitstream 115. The remainder of the encoding steps for each chroma TB correspond to the encoding process for the luma TB, as described with reference to step 1680. Method 1600 ends upon execution of step 16100, and control in processor 205 proceeds to the next CU of the CTU.

FIG. 17 shows a method 1700 for decoding encoded units of image frames from a video bitstream 133. Method 1700 may be implemented by a device such as a configured FPGA, ASIC, or ASSP. Further, method 1700 may be performed by video decoder 134 under execution of processor 205. As such, method 1700 may be stored on computer readable storage medium and/or memory 206. Method 1700 will decode blocks from bitstream 133 such that coefficient group sizes are determined based solely on transform block sizes and do not further differentiate between luma and chroma channels. Since entropy decoding is a critical feedback loop in video encoder 134, it is advantageous to reduce the memory accesses or operations required for coefficient group size determination. Method 1700 is called for each coding unit in the coding tree, ie, in step 1430 of FIG. As mentioned above, step 1430 is performed if the current split is no split 510. Method 1700 begins with step 1710 of decoding pred_mode.

In step 1710 of decoding pred_mode, entropy decoder 420 , under execution of processor 205 , decodes the prediction mode of the CU from bitstream 133 . Control in processor 205 passes from step 1710 to intra-prediction test step 1720 .

In an intra-prediction test step 1720, the processor 205 tests the prediction mode of the CU decoded in step 1710. If the prediction mode is intra prediction, step 1720 returns "YES" and control in processor 205 passes from step 1720 to step 1750, which decodes the intra subdivision mode. Otherwise, if it is not an intra prediction, step 1720 returns "NO" and control in processor 205 passes from step 1720 to step 1730 where the merge flag and index are decoded.

In step 1730 of decoding the merge flag and index, the entropy decoder 420, under execution of the processor 205, signals from the bitstream 133 whether "merge mode" is used for inter prediction in the bitstream. Decode the merge flag to be used. The merge mode causes the motion vector of a CU to be obtained from spatially (or temporally) adjacent blocks from a set of spatially or temporally adjacent candidate blocks. If merge mode is used, one candidate is selected by the “merge index” and is also decoded from the bitstream 133. If "motion vector prediction" is used, a similar decoding is performed whereby one of several possible candidate motion vectors is signaled as a predictor by a flag in the bitstream. Control in processor 205 passes from step 1730 to step 1740, where the motion vector delta is decoded.

In step 1740 of decoding motion vector deltas, entropy decoder 420 , under execution of processor 205 , decodes motion vector deltas from bitstream 133 . Step 1740 is performed if motion vector prediction is used for the CU. Motion vector delta specifies the delta between the motion vector predictor encoded in step 1730 and the motion vector used for motion compensation. Control in processor 205 passes from step 1740 to encoded residual test step 1760. If motion vector prediction is not used for the CU, step 1740 is not performed and control in processor 205 proceeds directly to step 1760.

In step 1750 of decoding intra-subpartition modes, entropy decoder 420, under execution of processor 205, determines whether to use intra-subpartitions from bitstream 133 using a context-encoded "intra_subpartitions_mode_flag" syntax element. Decoding whether to decide. Intra subdivision is available for luma channels when the luma CB size is larger than the minimum luma transform block size, ie, 16 luma samples. As shown in set 1500, intra subpartitioning partitions a coding unit into multiple luma transform blocks. If the luma CB is split into multiple TBs, the "intra_subpartitions_split_flag" signals whether the split of the luma CB into multiple luma TBs occurs horizontally or vertically. Collectively, "intra_subpartitions_mode_flag" and "intra_subpartitions_split_flag" are listed as "ISP_NO_SPLIT", "ISP_HOR_SPLIT", and "ISP_VER_SPLIT". encode two possible partitions. Control in processor 205 passes from step 1750 to encoded residual test step 1760.

In a coding residual testing step 1760, processor 205 determines whether at least one residual coefficient in any transform block of the coding block is significant. This determination includes all luma TBs resulting from the application of intra-subdivision and the chroma TB pairs associated with the two chroma channels. Entropy encoder 420, under execution of processor 205, arithmetically decodes the "cu_cbf" syntax element, and processor 205 determines whether at least one residual coefficient in any TB of the CU is significant. do. If at least one residual coefficient in either luma TB and chroma TB is significant, step 1760 returns "YES" and control in processor 205 proceeds to step 1770, which determines the luma coefficient group size. If there are no significant residual coefficients in any TB of the CU, as indicated by the arithmetic decoded "zero" for cu_cbf, step 1760 returns "no" and the method 1700 ends and the processor 205 proceeds to the next CU within the CTU.

In determining luma coefficient group size step 1770, processor 205 determines the coefficient group size of one or more luma transform blocks associated with the CU. The determination at step 1770 operates similarly to the determination at step 1670. Control within processor 205 passes from step 1770 to step 1780, which decodes the luma TB.

In step 1780 of decoding luma TB, entropy decoder 420 , under execution of processor 205 , decodes the residual coefficients of one or more luma TBs of the CU from bitstream 133 . The determined coefficient group size of step 1770 is used for each luma TB. For each luma TB, a coded block flag is decoded from the bitstream 133 indicating the presence of at least one significant coefficient within the luma TB. If there is at least one significant coefficient in the luma TB, the last significant position is decoded from the bitstream. The last significant position is defined as the last significant coefficient along the scan path going from the DC (top left) coefficient of TB to the bottom right coefficient. A scan path is defined as a diagonal scan in dividing the TB into an array of non-overlapping sub-blocks, each sized as the coefficient group size and occupying the entire TB. The progression from one subblock to the next in the scan order also follows a diagonal scan. Entropy encoder 338 encodes an "encoding subblock flag" for each coefficient group other than the upper left coefficient group and the coefficient group containing the last significant coefficient. The encoded subblock flag indicates the presence of at least one significant residual coefficient within the subblock. If there are no significant residual coefficients in a subblock, the diagonal scan of residual coefficients in TB skips that subblock. If there is at least one significant residual coefficient in a subblock, all positions in that subblock are scanned, the magnitude of each residual coefficient is encoded, and the sign of each significant residual coefficient is encoded. Ru. From step 1780, control in processor 205 passes to step 1790, where the chroma coefficient group size is determined.

In determining chroma coefficient group size step 1790, processor 205 determines the coefficient group size for the pair of chroma transform blocks associated with the CU. The determination made in step 1790 operates similarly to the determination made in step 1690.

Similar to step 1690, the coefficient group size is determined in step 1790 based on the transform block size and does not further differentiate between luma and chroma channels. Therefore, the coefficient group size is determined regardless of whether the chroma format is 4:2:2 or 4:2:0, or the corresponding subsampling in each color plane. As explained in connection with Table 1, the coefficient group size is based on the maximum area of the TB, which is up to 16 samples. Step 1690 operates to determine the coefficient group size of TB regardless of the color plane (Cb or Cr) of the transform block. Control within processor 205 passes from step 1790 to step 17100, which decodes the chroma TB.

In step 17100 of decoding chroma TB, entropy decoder 420, under execution of processor 205, decodes the residual coefficients of the CU's chroma TB pair from bitstream 133. The determined coefficient group size of step 1790 is used for the pair of chroma TBs. For each chroma TB, a coded block flag is decoded from the bitstream 133 indicating the presence of at least one significant coefficient within the chroma TB. The remainder of the decoding process for each chroma TB operates in the same manner as for luma TBs, as described with reference to step 1780. Method 1700 ends upon execution of step 17100 and control in processor 205 proceeds to the next CU of the CTU.

Table 3 shows the encoding performance results obtained under the JVET "Common Test Conditions" (CTC) - "All Intra Main 10" configuration when Table 1 is used. The results in Table 3 were obtained using the "VVC Test Model" (VTM) software implementing methods 1600 and 1700 compared to baseline VTM-4.0 without implementing methods 1600 and 1700. Overall, there was no coding impact from the change and even a small gain in the chroma channel, indicating that simplifying the mapping table from transform block size to coefficient group size is not detrimental to coding performance. Demonstrate.

Video encoder 115 and video decoder 134 use methods 1600 and 1700, respectively, to achieve memory reduction in the residual encoding/decoding process by matching coefficient group sizes of luma TB and chroma TB. As a result, Chroma TB has access to coefficient group sizes such as 2x8. Not just 2x2 and 4x4, but 4x2, 2x4, 8x2. For luma TB, sizes of 16x1 and 1x16 are possible using intra subdivision. Although sizes 16x1 and 1x16 are available to chroma due to their presence in Table 1, the minimum width and height of a chroma block is two samples, so sizes 16x1 and 1x16 are not used in chroma TB. Since residual encoding and decoding are part of a feedback loop in the design, memory reduction corresponds to, for example, improving cache performance in a software implementation or critical path reduction in a hardware implementation.

uint32_t g_log2SbbSize[MAX_CU_DEPTH + 1][MAX_CU_DEPTH + 1][2] =
//===== Luma/Chroma =====
{
{ { 0,0 },{ 0,1 },{ 0,2 },{ 0,3 },{ 0,4 },{ 0,4 },{ 0,4 },{ 0,4 } },
{ { 1,0 },{ 1,1 },{ 1,2 },{ 1,3 },{ 1,3 },{ 1,3 },{ 1,3 },{ 1,3 } },
{ { 2,0 },{ 2,1 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 } },
{ { 3,0 },{ 3,1 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 } },
{ { 4,0 },{ 3,1 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 } },
{ { 4,0 },{ 3,1 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 } },
{ { 4,0 },{ 3,1 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 } },
{ { 4,0 },{ 3,1 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 },{ 2,2 } }
};
Table 1: Conversion of block size to coefficient group mapping table for luma channel and chroma channel (with same size coefficient group for TB regardless of whether TB is for luma channel or chroma channel)
uint32_t g_log2SbbSize[2][MAX_CU_DEPTH+1][MAX_CU_DEPTH+1][2] =
{
//===== Luma =====
{
{ {0,0}, {0,1}, {0,2}, {0,3}, {0,4}, {0,4}, {0,4}, {0,4} },
{ {1,0}, {1,1}, {1,2}, {1,3}, {1,3}, {1,3}, {1,3}, {1,3} },
{ {2,0}, {2,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} },
{ {3,0}, {3,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} },
{ {4,0}, {3,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} },
{ {4,0}, {3,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} },
{ {4,0}, {3,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} },
{ {4,0}, {3,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} }
},
//===== Chroma =====
{
{ {0,0}, {0,0}, {0,0}, {0,0}, {0,0}, {0,0}, {0,0}, {0,0} },
{ {0,0}, {1,1}, {1,1}, {1,1}, {1,1}, {1,1}, {1,1}, {1,1} },
{ {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} },
{ {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} },
{ {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} },
{ {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} },
{ {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} },
{ {0,0}, {1,1}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2}, {2,2} }
},
};
Table 2: Conventional mapping of transform block size to coefficient group size for luma and chroma channels (with different coefficient group sizes for the same size TB for luma vs. chroma)

Table 3: Coding performance resulting from having coefficient groups of the same size for a TB, regardless of whether the TB is for a luma channel or a chroma channel.

Industrial Applicability The described configuration is applicable to the computer and data processing industry, in particular to digital signal processing for encoding, decoding signals such as video and image signals, achieving high compression efficiency. do.

In contrast to HEVC, VVC systems allow the use of separate coding trees for luma and chroma channels for increased flexibility. However, as mentioned above, resulting problems can arise from the use of smaller chroma blocks, which affects throughput. The configuration described herein determines appropriate rules as each encoding tree unit is processed to help avoid throughput problems. Furthermore, as mentioned above, the above configuration improves the efficiency of the arithmetic coding of the context coding bins used to describe each coding tree, given the rules to avoid throughput problems. and can assist in providing accuracy.

The foregoing describes only some embodiments of the invention; modifications and/or changes may be made thereto without departing from the scope and spirit of the invention, and the embodiments are exemplary and limiting. Not the point.

Claims (4)

A method for decoding a transform block of an image frame from a bitstream, the method comprising:
determining a chroma format of the image frame from a plurality of chroma formats including a 4:2:0 chroma format and a 4:2:2 chroma format;
dividing the coding tree unit into one or more coding units each having a luma coded block and a chroma coded block;
determining a sub-block of the transform block that is a luma transform block for a luma encoded block or a chroma transform block for a chroma encoded block;
decoding the transform block from the bitstream using the sub-blocks;
If the block size of the chroma coded block of a certain coding unit having a luma coded block and a chroma coded block in the coding tree unit of 4:2:0 chroma format is 8x2, the certain code Even if vertical ternary division is performed on the luma encoded block of a coding unit, vertical ternary division of the chroma encoded block of the certain encoding unit is not permitted;
If the luma encoding block of the certain encoding unit is vertically ternary divided and the chroma encoding block of the certain encoding unit is not vertically ternary dividing, the chroma encoding of the certain encoding unit The block is arranged at a position corresponding to three luma encoded blocks obtained by vertical ternary division of the luma encoded block of the certain encoding unit,
The size ratio of each of the three luma encoded blocks is 1:2:1,
The transform block is decoded based on (i) the size of the transform block; and (ii) whether the transform block is a luma transform block or a chroma transform block;
The size of the sub-blocks of the transform block is determined by the size of the sub-block of the transform block without using both (i) whether the transform block is a luma transform block or a chroma transform block; and (ii) the chroma format of the image frame. A method characterized in that the size of the transformation block is determined from the size of the transformation block. A video decoder for decoding transform blocks of image frames from a bitstream, the video decoder comprising:
means for determining a chroma format of the image frame from a plurality of chroma formats including a 4:2:0 chroma format and a 4:2:2 chroma format;
means for dividing the coding tree unit into one or more coding units each having a luma coded block and a chroma coded block;
means for determining a subblock of the transform block that is a luma transform block for a luma encoded block or a chroma transform block for a chroma encoded block;
means for decoding the transform block from the bitstream using the sub-blocks;
If the block size of the chroma coded block of a certain coding unit having a luma coded block and a chroma coded block in the coding tree unit of 4:2:0 chroma format is 8x2, the certain code Even if vertical ternary division is performed on the luma encoded block of a coding unit, vertical ternary division of the chroma encoded block of the certain encoding unit is not permitted;
If the luma encoding block of the certain encoding unit is vertically ternary divided and the chroma encoding block of the certain encoding unit is not vertically ternary dividing, the chroma encoding of the certain encoding unit The block is arranged at a position corresponding to three luma encoded blocks obtained by vertical ternary division of the luma encoded block of the certain encoding unit,
The size ratio of each of the three luma encoded blocks is 1:2:1,
The transform block is decoded based on (i) the size of the transform block; and (ii) whether the transform block is a luma transform block or a chroma transform block;
The size of the sub-blocks of the transform block is determined by the size of the sub-block of the transform block without using both (i) whether the transform block is a luma transform block or a chroma transform block; and (ii) the chroma format of the image frame. A video decoder characterized in that the size of a transform block is determined based on the size of a transform block. A method for encoding transform blocks of an image frame into a bitstream, the method comprising:
determining a chroma format of the image frame from a plurality of chroma formats including a 4:2:0 chroma format and a 4:2:2 chroma format;
dividing the coding tree unit into one or more coding units each having a luma coded block and a chroma coded block;
determining a sub-block of the transform block that is a luma transform block for a luma encoded block or a chroma transform block for a chroma encoded block;
encoding the transform block into the bitstream using the sub-blocks;
If the block size of the chroma coded block of a certain coding unit having a luma coded block and a chroma coded block in the coding tree unit of 4:2:0 chroma format is 8x2, the certain code Even if vertical ternary division is performed on the luma encoded block of a coding unit, vertical ternary division of the chroma encoded block of the certain encoding unit is not permitted;
If the luma encoding block of the certain encoding unit is vertically ternary divided and the chroma encoding block of the certain encoding unit is not vertically ternary dividing, the chroma encoding of the certain encoding unit The block is arranged at a position corresponding to three luma encoded blocks obtained by vertical ternary division of the luma encoded block of the certain encoding unit,
The size ratio of each of the three luma encoded blocks is 1:2:1,
The transform block is encoded based on (i) the size of the transform block; and (ii) whether the transform block is a luma transform block or a chroma transform block;
The size of the sub-blocks of the transform block is determined by the size of the sub-block of the transform block without using both (i) whether the transform block is a luma transform block or a chroma transform block; and (ii) the chroma format of the image frame. A method characterized in that the size of the transformation block is determined from the size of the transformation block. A video encoder for encoding transform blocks of image frames into a bitstream, the video encoder comprising:
means for determining a chroma format of the image frame from a plurality of chroma formats including a 4:2:0 chroma format and a 4:2:2 chroma format;
means for dividing the coding tree unit into one or more coding units each having a luma coded block and a chroma coded block;
means for determining a subblock of the transform block that is a luma transform block for a luma encoded block or a chroma transform block for a chroma encoded block;
means for encoding the transform block into the bitstream using the sub-blocks;
If the block size of the chroma coded block of a certain coding unit having a luma coded block and a chroma coded block in the coding tree unit of 4:2:0 chroma format is 8x2, the certain code Even if vertical ternary division is performed on the luma encoded block of a coding unit, vertical ternary division of the chroma encoded block of the certain encoding unit is not permitted;
If the luma encoding block of the certain encoding unit is vertically ternary divided and the chroma encoding block of the certain encoding unit is not vertically ternary dividing, the chroma encoding of the certain encoding unit The block is arranged at a position corresponding to three luma encoded blocks obtained by vertical ternary division of the luma encoded block of the certain encoding unit,
The size ratio of each of the three luma encoded blocks is 1:2:1,
The transform block is encoded based on (i) the size of the transform block; and (ii) whether the transform block is a luma transform block or a chroma transform block;
The size of the sub-blocks of the transform block is determined by the size of the sub-block of the transform block without using both (i) whether the transform block is a luma transform block or a chroma transform block; and (ii) the chroma format of the image frame. A video encoder characterized in that the size of the transform block is determined based on the size of the transform block.

Images