JP2024501465A - Adaptive loop filter with fixed filter - Google Patents

Abstract

The video decoder determines a first class index for the reconstructed samples, selects a filter from the first set of filters based on the first class index, and determines a first intermediate sample value. applying a first stage adaptive loop filter (ALF) to the reconstructed sample by applying a filter from the first set of filters to the reconstructed sample; determine a class index, select a second filter from a second set of filters based on the second class index, and add a second filter to the reconstructed sample to determine a first sample modification value. applying a second stage ALF to the reconstructed sample by applying a filter and determining a second sample change value based on the first intermediate sample value; and determining a filtered reconstructed sample based on the first and second sample modification values.

Description

[0001] This application is filed in U.S. Patent Application No. 17/557,706, filed December 21, 2021, filed February 11, 2021, each of which is incorporated herein by reference in its entirety. Claims priority to U.S. Provisional Patent Application No. 63/148,538, filed December 23, 2020, and U.S. Provisional Patent Application No. 63/130,275, filed December 23, 2020. U.S. Patent Application No. 17/557,706, filed on December 21, 2021, and U.S. Provisional Patent Application No. 63/148,538, filed on February 11, 2021, and December 23, 2020 claims the benefit of U.S. Provisional Patent Application Ser.

[0002] The present disclosure relates to video encoding and video decoding.

[0003] Digital video capabilities include digital television, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, and digital media. It can be incorporated into a wide range of devices, including players, video gaming devices, video game consoles, cellular or satellite radio telephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Digital video devices include MPEG-2, MPEG-4, ITU-T H. 263, ITU-T H. H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H. Implement video coding techniques, such as those described in the standards defined by H.265/High Efficiency Video Coding (HEVC), and extensions of such standards. By implementing such video coding techniques, video devices may more efficiently transmit, receive, encode, decode, and/or store digital video information.

[0004] Video coding techniques include spatial (intra-picture) and/or temporal (inter-picture) prediction to reduce or eliminate redundancy inherent in video sequences. In block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) is partitioned into video blocks, sometimes referred to as coding tree units (CTUs), coding units (CUs), and/or coding nodes. obtain. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction on reference samples in adjacent blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in adjacent blocks in the same picture, or temporal prediction with respect to reference samples in other reference pictures. A picture is sometimes called a frame, and a reference picture is sometimes called a reference frame.

[0005] According to the techniques of this disclosure, a video decoder may be configured to perform two-stage adaptive loop filtering. For the first stage of ALF, the video decoder may be configured to determine an activity value and a direction value for the reconstructed sample to be filtered. Activity values may generally indicate the distribution of sample values in a neighborhood of samples around the sample being filtered. Direction values generally determine whether the sample value is increasing horizontally, vertically, increasing 45 degrees, increasing 135 degrees, or not increasing at all. It may indicate the direction in which the sample value is changing, such as whether the sample value is changing or not. Based on the activity value and the direction value, the video decoder may determine a class index for the sample to be filtered. A class index may be associated with a filter from a first set of filters, each filter being defined by a shape and a set of coefficient values. A video decoder may determine a first set of filters and which classifiers correspond to which filters based on syntax contained within the bitstream. A video decoder may apply a filter to the reconstructed samples to determine a first intermediate sample value.

[0006] For the second stage of ALF, the video decoder determines a first sample modification value by applying a second filter to the reconstructed samples. , determining a second sample modification value based on the first intermediate sample value. The video decoder then determines a filtered reconstructed sample based on the reconstructed sample, the first sample modification value, and the second sample modification value. As described herein, a first stage ALF is applied to the reconstructed samples of the reconstructed block, and a second stage ALF is applied to the reconstructed samples. By applying the techniques of this disclosure, the techniques of this disclosure can better estimate the entire decoded video data by considering local features of the video data than can be done by a one-stage ALF. can improve the quality of the material.

[0007] According to one example, a method of decoding video data includes applying a first stage adaptive loop filter (ALF) to reconstructed samples of a reconstructed block; Here, applying the first stage ALF includes determining a first class index for the reconstructed samples and determining a first class index of the filter based on the first class index. selecting a filter from the set; and applying a filter from the first set of filters to the reconstructed sample to determine a first intermediate sample value; applying a second stage ALF, where applying the second stage ALF includes determining a second class index for the reconstructed sample; selecting a second filter from the second set of filters based on the index; and applying the second filter to the reconstructed sample to determine a first sample modification value; determining a second sample modification value based on the intermediate sample value of the reconstructed sample, the first sample modification value, and the second sample modification value; and determining a construction sample.

[0008] According to one example, a device for decoding video data is implemented with a memory configured to store video data and a circuit, wherein the reconstructed samples of the reconstructed block are applying an adaptive loop filter (ALF); and in order to apply the first stage ALF, one or more processors determine a first class index for the reconstructed samples. selecting a filter from the first set of filters based on a first class index; and selecting a filter from the first set of filters on the reconstructed sample to determine a first intermediate sample value. and applying a second stage ALF to the reconstructed sample; or determining a second class index for the reconstructed sample; and selecting a second filter from the second set of filters based on the second class index; further comprising: applying a second filter to the reconstructed sample to determine a sample modification value of; and determining a second sample modification value based on the first intermediate sample value. one or more configured and configured to: determine a filtered reconstructed sample based on the reconstructed sample, the first sample modification value, and the second sample modification value; processor.

[0009] According to one example, a computer-readable storage medium is configured to store a reconstructed sample of a reconstructed block in one or more processors when executed by one or more processors. applying a first stage adaptive loop filter (ALF); determining a filter from a first set of filters based on a first class index; and applying a second stage ALF to the reconstructed sample; the one or more processors determining a second class index for the reconstructed sample and selecting a second filter from the second set of filters based on the second class index; applying a second filter to the reconstructed sample to determine a first sample modification value; and determining a second sample modification value based on the first intermediate sample value. further configured to store instructions for determining a filtered reconstructed sample based on the reconstructed sample, the first sample modification value, and the second sample modification value.

[0010] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

[0011] FIG. 1 is a block diagram illustrating an example video encoding and decoding system that may implement techniques of this disclosure. [0012] FIG. 2 is a conceptual diagram illustrating an example adaptive loop filter shape that may be used in accordance with the techniques of this disclosure. FIG. 2 is a conceptual diagram illustrating an example adaptive loop filter shape that may be used in accordance with the techniques of this disclosure. [0013] FIG. 2 is a conceptual diagram illustrating example subsampled Laplacian values for 4×4 subblock adaptive loop filter classification that may be used in accordance with the techniques of this disclosure. [0014] FIG. 2 is a conceptual diagram illustrating example Laplacian values for luma samples that may be used in accordance with the techniques of this disclosure. [0015] FIG. 2 is a conceptual diagram illustrating an example of adaptive loop filter class merging that may be used in accordance with the techniques of this disclosure. [0016] FIG. 2 is a conceptual diagram illustrating an example geometric transformation of a 7x7 diamond filter shape that may be used in accordance with the techniques of this disclosure. [0017] FIG. 7 illustrates an example of symmetric sample padding in an ALF that may be used in accordance with the techniques of this disclosure. FIG. 3 is a diagram illustrating an example of symmetric sample padding in an ALF that may be used in accordance with the techniques of this disclosure. FIG. 3 is a diagram illustrating an example of symmetric sample padding in an ALF that may be used in accordance with the techniques of this disclosure. [0018] FIG. 4 is a diagram illustrating an example of ALF 4x4 subblock classification that may be used in accordance with the techniques of this disclosure. [0019] An example framework for filtering reconstructed samples of video data using a plurality of fixed filter sets and a plurality of signaled filter sets in accordance with the techniques of this disclosure. Diagram showing. [0020] FIG. 4 illustrates an example of a 5×5 filter that may be used in accordance with the techniques of this disclosure. FIG. 7 is a diagram illustrating an example of a 7×7 filter that may be used in accordance with the techniques of this disclosure. FIG. 7 is a diagram illustrating an example of a 9×9 filter that may be used in accordance with the techniques of this disclosure. FIG. 3 is a diagram illustrating an example of an 11×11 filter that may be used in accordance with the techniques of this disclosure. FIG. 3 is a diagram illustrating an example of a 13×13 filter that may be used in accordance with the techniques of this disclosure. [0021] FIG. 4 illustrates an example for implementing multiple fixed filters with one signaled filter in accordance with the techniques of this disclosure. [0022] FIG. 4 illustrates an example of a two-part filter that may be used in accordance with the techniques of this disclosure. FIG. 3 is a diagram illustrating an example of a two-part filter that may be used in accordance with the techniques of this disclosure. FIG. 3 is a diagram illustrating an example of a two-part filter that may be used in accordance with the techniques of this disclosure. FIG. 3 is a diagram illustrating an example of a two-part filter that may be used in accordance with the techniques of this disclosure. FIG. 3 is a diagram illustrating an example of a two-part filter that may be used in accordance with the techniques of this disclosure. [0023] FIG. 2 is a conceptual diagram illustrating an example ALF framework in accordance with the techniques of this disclosure. 1 is a conceptual diagram illustrating an example ALF framework in accordance with the techniques of this disclosure. [0024] FIG. 2 is a block diagram illustrating an example video encoder that may implement the techniques of this disclosure. [0025] FIG. 2 is a block diagram illustrating an example video decoder that may implement techniques of this disclosure. [0026] FIG. 2 is a block diagram illustrating an example filter unit for implementing the techniques of this disclosure. [0027] FIG. 7 is a flowchart illustrating an example process for encoding a current block in accordance with the techniques of this disclosure. [0028] FIG. 7 is a flowchart illustrating an example process for decoding a current block in accordance with the techniques of this disclosure. [0029] A flowchart illustrating an example process for decoding a current block in accordance with the techniques of this disclosure.

[0030] Video coding (e.g., video encoding and/or video decoding) generally involves predicting blocks of video data from already coded blocks of video data in the same picture (i.e., intra prediction) or different It either involves predicting blocks of video data from already coded blocks of video data in a picture (i.e., inter prediction). In some cases, the video encoder also calculates residual data by comparing the predictive block to the original block. Therefore, the residual data represents the difference between the predictive block and the original block. A video encoder transforms and quantizes the residual data and signals the transformed and quantized residual data in the encoded bitstream. A video decoder adds residual data to the predictive block to generate a reconstructed video block that more closely matches the original video block than the predictive block alone. To further improve the quality of the decoded video, the video decoder may perform one or more filtering operations on the reconstructed video blocks. Examples of these filtering operations include deblock filtering, sample adaptive offset (SAO) filtering, and adaptive loop filtering (ALF). The parameters for these filtering operations are either determined by the video encoder and explicitly signaled in the encoded video bitstream, or the parameters do not need to be explicitly signaled in the encoded video bitstream. may be either implicitly determined by the video decoder.

[0031] This disclosure describes techniques associated with filtering reconstructed video data in a video encoding and/or video decoding process, and more particularly, this disclosure describes techniques associated with ALF. explain. However, the techniques described can potentially be applied to other filtering schemes as well.

[0032] According to the techniques of this disclosure, a video decoder may be configured to perform two-stage adaptive loop filtering. For the first stage of ALF, the video decoder may be configured to determine activity values and direction values for the reconstructed samples to be filtered. Activity values may generally indicate the distribution of sample values in a neighborhood of samples around the sample being filtered. Direction values generally determine whether the sample value is increasing horizontally, vertically, increasing 45 degrees, increasing 135 degrees, or not increasing at all. It can indicate the direction in which the sample value is changing, such as whether the sample value is changing or not. Based on the activity value and the direction value, the video decoder may determine a class index for the sample to be filtered. The class index may be associated with filters from a first set of filters, each filter being defined by a shape and a set of coefficient values. The video decoder may determine the first set of filters and which classifier corresponds to which filter based on syntax contained within the bitstream. A video decoder may apply a filter to the reconstructed samples to determine a first intermediate sample value.

[0033] For the second stage of ALF, the video decoder determines a first sample modification value by applying a second filter to the reconstructed samples, and determines a first sample modification value based on the first intermediate sample value. , determining a second sample modification value. The video decoder then determines filtered reconstructed samples based on the reconstructed samples, the first sample modification value, and the second sample modification value. By applying the first stage ALF to the reconstructed samples of the reconstructed block and the second stage ALF to the reconstructed samples, as described herein, the techniques of the present disclosure may improve the overall quality of decoded video data better than can be done by a one-stage ALF by considering local features of the video data.

[0034] FIG. 1 is a block diagram illustrating an example video encoding and decoding system 100 that may implement the techniques of this disclosure. The techniques of this disclosure are generally directed to coding (encoding and/or decoding) video data. Generally, video data includes some data for processing the video. Accordingly, video data may include raw unencoded video, encoded video, decoded (eg, reconstructed) video, and video metadata such as signaling data.

[0035] As shown in FIG. 1, system 100 includes, in this example, a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. In particular, source device 102 provides video data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 may include desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, etc. , video gaming consoles, video streaming devices, broadcast receiver devices, and the like. In some cases, source device 102 and destination device 116 may be equipped for wireless communication and may therefore be referred to as wireless communication devices.

[0036] In the example of FIG. 1, source device 102 includes a video source 104, a memory 106, a video encoder 200, and an output interface 108. Destination device 116 includes an input interface 122, a video decoder 300, memory 120, and display device 118. According to this disclosure, a video encoder 200 of source device 102 and a video decoder 300 of destination device 116 are configured for adaptive loop filtering using first and second stages in a manner described herein. may be configured to apply techniques. Accordingly, source device 102 represents an example of a video encoding device, and destination device 116 represents an example of a video decoding device. In other examples, the source device and destination device may include other components or arrangements. For example, source device 102 may receive video data from an external video source, such as an external camera. Similarly, destination device 116 may interface with an external display device rather than including an integrated display device.

[0037] The system 100 shown in FIG. 1 is only one example. Generally, any digital video encoding and/or decoding device may implement the techniques for adaptive loop filtering described herein. Source device 102 and destination device 116 are only examples of coding devices such that source device 102 generates coded video data for transmission to destination device 116. This disclosure refers to a "coding" device as a device that performs coding (encoding and/or decoding) of data. Accordingly, video encoder 200 and video decoder 300 represent examples of coding devices, particularly video encoders and video decoders, respectively. In some examples, source device 102 and destination device 116 may operate substantially symmetrically, such that each of source device 102 and destination device 116 includes a video encoding component and a video decoding component. Thus, system 100 may support one-way or two-way video transmission between source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0038] In general, video source 104 represents a source of video data (i.e., raw, unencoded video data) that transmits a continuous series of pictures (also referred to as "frames") of video data to video encoder 200. A video encoder 200 encodes data for a picture. Video source 104 of source device 102 includes a video capture device, such as a video camera, a video archive containing previously captured raw video, and/or a video feed interface for receiving video from a video content provider. obtain. As a further alternative, video source 104 may produce computer graphics-based data or a combination of live, archived, and computer-generated video as the source video. In each case, video encoder 200 encodes captured, pre-captured, or computer-generated video data. Video encoder 200 may reorder pictures from a reception order (sometimes referred to as a "display order") to a coding order for coding. Video encoder 200 may generate a bitstream that includes encoded video data. Source device 102 may then output the encoded video data onto computer-readable medium 110 via output interface 108, for example, for reception and/or retrieval by input interface 122 of destination device 116.

[0039] Memory 106 of source device 102 and memory 120 of destination device 116 represent general purpose memory. In some examples, memories 106, 120 may store raw video data, eg, raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106, 120 may store software instructions executable by video encoder 200 and video decoder 300, respectively, for example. Although memory 106 and memory 120 are shown separately from video encoder 200 and video decoder 300 in this example, video encoder 200 and video decoder 300 may also have internal memory for functionally similar or equivalent purposes. It should be understood that this may include Additionally, memories 106, 120 may store encoded video data, eg, output from video encoder 200 and input to video decoder 300. In some examples, portions of memory 106, 120 may be allocated as one or more video buffers, eg, to store raw decoded and/or encoded video data.

[0040] Computer-readable medium 110 may represent any type of medium or device that is capable of transporting encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 communicates to enable source device 102 to transmit encoded video data directly to destination device 116 in real time, e.g., over a radio frequency network or a computer-based network. Represents a medium. In accordance with a communication standard, such as a wireless communication protocol, output interface 108 may modulate a transmitted signal containing encoded video data, and input interface 122 may demodulate a received transmitted signal. A communication medium may comprise any wireless or wired communication medium, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, wide area network, or global network such as the Internet. Communication media may include routers, switches, base stations, or any other equipment that may be useful for facilitating communication from source device 102 to destination device 116.

[0041] In some examples, source device 102 may output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access encoded data from storage device 112 via input interface 122. Storage device 112 can be a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital device for storing encoded video data. may include any of a variety of distributed or locally accessed data storage media, such as storage media.

[0042] In some examples, source device 102 may output encoded video data to file server 114 or another intermediate storage device that may store encoded video data generated by source device 102. Destination device 116 may access stored video data from file server 114 via streaming or downloading.

[0043] File server 114 may be any type of server device capable of storing encoded video data and transmitting the encoded video data to destination device 116. File server 114 may be a web server (e.g., for a website), a file transfer protocol service (such as File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol). a server configured to provide content delivery network (CDN) devices, hypertext transfer protocol (HTTP) servers, multimedia broadcast multicast services (MBMS) or enhanced MBMS (eMBMS) servers, and/or network attached storage (NAS); ) may represent a device. File server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, etc. It is possible.

[0044] Destination device 116 may access encoded video data from file server 114 through any standard data connection, including an Internet connection. This includes wireless channels (e.g., Wi-Fi connections), wired connections (e.g., digital subscriber line (DSL), (such as a cable modem), or a combination of both. Input interface 122 may be configured in accordance with any one or more of the various protocols described above for retrieving or receiving media data from file server 114 or other such protocols for retrieving media data. may be configured to operate.

[0045] Output interface 108 and input interface 122 are wireless transmitters/receivers, modems, wired networking components (e.g., Ethernet cards), wireless devices that operate in accordance with any of the various IEEE 802.11 standards. May represent a communications component or other physical component. In examples where the output interface 108 and the input interface 122 include wireless components, the output interface 108 and the input interface 122 may be connected to cellular communications, such as 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, etc. It may be configured to transfer data, such as encoded video data, in accordance with a standard. In some examples, the output interface 108 comprises a wireless transmitter, the output interface 108 and the input interface 122 are compatible with the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee®), the Bluetooth® standard. may be configured to transfer data, such as encoded video data, in accordance with other wireless standards, such as, for example. In some examples, source device 102 and/or destination device 116 may include respective system-on-chip (SoC) devices. For example, source device 102 may include an SoC device for implementing functionality attributable to video encoder 200 and/or output interface 108, and destination device 116 may include an SoC device for implementing functionality attributable to video decoder 300 and/or input interface 122. may include an SoC device for implementing the implementation.

[0046] The techniques of this disclosure can be applied to over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions such as Dynamic Adaptive Streaming over HTTP (DASH), encoded onto a data storage medium, etc. The present invention may be applied to video coding to support any of a variety of multimedia applications, such as digital video stored on a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0047] Input interface 122 of destination device 116 receives an encoded video bitstream from computer-readable medium 110 (eg, a communication medium, storage device 112, file server 114, etc.). The encoded video bitstream is encoded by video decoder 300, such as syntax elements having values that describe the characteristics and/or processing of video blocks or other coded units (e.g., slices, pictures, groups of pictures, sequences, etc.). may also include signaling information defined by video encoder 200 that is also used. Display device 118 displays a decoded picture of the decoded video data to the user. Display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0048] Although not shown in FIG. 1, in some examples, video encoder 200 and video decoder 300 may each be integrated with an audio encoder and/or audio decoder to output audio in a common data stream. A suitable MUX-DEMUX unit or other hardware and/or software may be included to handle multiplexed streams containing both video and video. Where applicable, the MUX-DEMUX unit shall comply with the ITU H. 223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP).

[0049] Video encoder 200 and video decoder 300 each include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software , hardware, firmware, etc., or any combination thereof. When the present techniques are partially implemented in software, the device stores instructions for the software on a suitable non-transitory computer-readable medium and executes one or more processors to implement the techniques of the present disclosure. can be used to execute the instructions in hardware. Each of video encoder 200 and video decoder 300 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODEC) in the respective device. . Devices including video encoder 200 and/or video decoder 300 may include integrated circuits, microprocessors, and/or wireless communication devices such as cellular telephones.

[0050] Video encoder 200 and video decoder 300 are based on ITU-T H.V., also called High Efficiency Video Coding (HEVC). The video coding standard may operate in accordance with a video coding standard, such as H.265 or extensions thereof, such as multi-view and/or scalable video coding extensions. Alternatively, video encoder 200 and video decoder 300 may be configured using ITU-T H.280, also referred to as Versatile Video Coding (VVC). It may operate according to other proprietary or industry standards, such as H.266. The draft VVC standard was published by Bross et al., “Versatile Video Coding (Draft 10),” ITU-T SG16 WP3 and ISO/IEC JTC 1/SC29/WG11 Joint Video Expert Team (JVET), 18th Meeting, Video Conference. , June 22, 2020 to July 1, 2020, JVET-S2001-v17 (hereinafter referred to as "VVC Draft 10"). However, the techniques of this disclosure are not limited to any particular coding standard, and further, it is expressly contemplated that the techniques described herein may be used in conjunction with successor standards to VVC. There is.

[0051] Generally, video encoder 200 and video decoder 300 may perform block-based coding of pictures. The term "block" generally refers to a structure containing data to be processed (e.g., to be encoded, decoded, or otherwise used in an encoding and/or decoding process). Point. For example, a block may include a two-dimensional matrix of samples of luminance and/or chrominance data. Generally, video encoder 200 and video decoder 300 may code video data represented in YUV (eg, Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for samples of a picture, video encoder 200 and video decoder 300 may code luminance and chrominance components, where chrominance The components may include both red and blue hue chrominance components. In some examples, video encoder 200 converts received RGB formatted data to a YUV representation prior to encoding, and video decoder 300 converts the YUV representation to RGB format. Alternatively, pre- and post-processing units (not shown) may perform these conversions.

[0052] This disclosure may generally refer to coding (eg, encoding and decoding) pictures to include the process of encoding or decoding data for a picture. Similarly, this disclosure may refer to coding blocks of pictures to include processes of encoding or decoding data for the blocks, such as predictive and/or residual coding. An encoded video bitstream generally includes a series of values for syntax elements that represent coding decisions (eg, coding modes) and partitioning of pictures into blocks. Accordingly, references to coding a picture or block should generally be understood as coding values for the syntax elements forming the picture or block.

[0053] HEVC defines various blocks, including coding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video coder (such as video encoder 200) partitions coding tree units (CTUs) into CUs according to a quadtree structure. That is, the video coder partitions the CTU and CU into four equal non-overlapping squares, where each node of the quadtree has either zero or four child nodes. A node without child nodes may be referred to as a "leaf node," and the CU of such a leaf node may include one or more PUs and/or one or more TUs. A video coder may further differentiate between PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents a partition of a TU. In HEVC, PU represents inter-prediction data, while TU represents residual data. A CU that is intra-predicted includes intra-prediction information such as an intra-mode indication.

[0054] As another example, video encoder 200 and video decoder 300 may be configured to operate according to VVC. According to VVC, a video coder (such as video encoder 200) partitions a picture into multiple coding tree units (CTUs). Video encoder 200 may partition CTUs according to a tree structure, such as a quad tree binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple partition types, such as the separation between CU, PU, and TU in HEVC. The QTBT structure includes two levels: a first level partitioned according to quadtree partitioning and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to the CTU. Leaf nodes of the binary tree correspond to coding units (CUs).

[0055] In an MTT partition structure, a block includes a quad tree (QT) partition, a binary tree (BT) partition, and one or more types of triple tree (TT) (also called ternary tree (TT)) partitions. It can be classified using A triple tree or ternary tree partition is a partition in which a block is split into three subblocks. In some examples, triple tree or ternary tree partitioning divides a block into three subblocks without splitting the original block through the center. Partition types in MTT (eg, QT, BT, and TT) can be symmetric or asymmetric.

[0056] In some examples, video encoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent each of the luminance and chrominance components; in other examples, the video encoder 200 and video decoder 300 include one QTBT/MTT structure for the luminance component and another QTBT/MTT structure for both chrominance components (or two QTBT/MTT structures for each chrominance component). Two or more QTBT or MTT structures may be used, such as.

[0057] Video encoder 200 and video decoder 300 may be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, or other partitioning structures according to HEVC. For purposes of explanation, the description of the techniques of this disclosure is presented with respect to the QTBT partition. However, it should be understood that the techniques of this disclosure may also be applied to video coders configured to use quadtree partitioning, or other types of partitioning as well.

[0058] In some examples, a CTU is a coding tree block (CTB) of luma samples, two corresponding CTBs of chroma samples of a picture with three sample arrays, or a monochrome picture, or for coding samples. Contains the CTB of a sample of a picture coded using three separate color planes and syntax structures used. The CTB may be an N×N block of samples for some value of N such that the division of components into the CTB is piecewise. The components can be arrays from one of the three arrays (luma and two chromas) that organize the picture into a 4:2:0, 4:2:2, or 4:4:4 color format or a single A sample, or an array or a single sample of an array, that composes a picture in monochrome format. In some examples, the coding block is an M×N block of samples, for some value of M and N such that the division of the CTB into coding blocks is partitioned.

[0059] Blocks (eg, CTUs or CUs) may be grouped in various ways in a picture. As an example, a brick may refer to a rectangular area of a CTU row within a particular tile in a picture. A tile may be a rectangular region of CTUs within a particular tile column and a particular tile row in a picture. A tile column refers to a rectangular region of a CTU with a height equal to the height of a picture and a width specified by a syntax element (eg, in a picture parameter set). A tile row refers to a rectangular region of the CTU with a height specified by a syntax element (eg, in a picture parameter set) and a width equal to the width of the picture.

[0060] In some examples, a tile may be partitioned into multiple bricks, each of which may include one or more CTU rows within the tile. Tiles that are not divided into multiple bricks may also be called bricks. However, bricks that are a true subset of tiles may not be called tiles.

[0061] Bricks in a picture may also be arranged in slices. A slice may be an integral number of bricks of pictures that may be contained exclusively in a single network abstraction layer (NAL) unit. In some examples, a slice includes either a number of complete tiles or only a contiguous sequence of complete bricks of one tile.

[0062] This disclosure uses "NxN" and "N by N )'', for example, 16x16 samples or 16 by 16 samples may be used interchangeably. Generally, a 16x16 CU has 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Similarly, an N×N CU generally has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU may be arranged in rows and columns. Moreover, a CU does not necessarily need to have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU may comprise N×M samples, where M is not necessarily equal to N.

[0063] Video encoder 200 encodes video data for CUs representing prediction and/or residual information, as well as other information. Prediction information indicates how a CU should be predicted to form a prediction block for the CU. The residual information generally represents the sample-by-sample difference between the samples of the CU and the predictive block prior to encoding.

[0064] To predict a CU, video encoder 200 may generally form a prediction block for the CU through inter-prediction or intra-prediction. Inter prediction generally refers to predicting a CU from data of a previously coded picture, whereas intra prediction generally refers to predicting a CU from previously coded data of the same picture. To perform inter prediction, video encoder 200 may use one or more motion vectors to generate a predictive block. Video encoder 200 may generally perform a motion search to identify a reference block that closely matches a CU, eg, with respect to the difference between the CU and the reference block. Video encoder 200 uses sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD) to determine whether a reference block exactly matches the current CU. ), or other such difference calculations may be used to calculate the difference metric. In some examples, video encoder 200 may predict the current CU using unidirectional prediction or bidirectional prediction.

[0065] Some examples of VVC also provide an affine motion compensation mode, which can be considered an inter-prediction mode. In affine motion compensation mode, video encoder 200 may determine two or more motion vectors representing non-translational motion, such as zooming in or out, rotation, perspective motion, or other anomalous motion types.

[0066] To perform intra prediction, video encoder 200 may select an intra prediction mode to generate predictive blocks. Some examples of VVC provide 67 intra-prediction modes, including various directional modes as well as planar and DC modes. Generally, video encoder 200 selects an intra prediction mode that describes neighboring samples to a current block from which samples of the current block (eg, a block of a CU) are to be predicted. Such samples are typically located at the top, top left corner of the current block in the same picture as the current block, assuming that video encoder 200 codes CTUs and CUs in raster scan order (left to right, top to bottom). , or can be on the left.

[0067] Video encoder 200 encodes data representing the prediction mode for the current block. For example, in inter-prediction mode, video encoder 200 may encode data representative of which of the various available inter-prediction modes will be used, as well as motion information for the corresponding mode. For example, for unidirectional or bidirectional inter-prediction, video encoder 200 may encode motion vectors using advanced motion vector prediction (AMVP) or merge mode. Video encoder 200 may use a similar mode to encode motion vectors for affine motion compensation mode.

[0068] Following prediction, such as intra-prediction or inter-prediction, of a block, video encoder 200 may calculate residual data for the block. Residual data, such as a residual block, represents the sample-by-sample difference between a block and a predicted block for the block formed using a corresponding prediction mode. Video encoder 200 may apply one or more transforms to the residual block to generate transform data in a transform domain rather than a sample domain. For example, video encoder 200 may apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. Additionally, video encoder 200 may perform a second transform, such as a mode-dependent non-separable secondary transform (MDNSST), a signal-dependent transform, or a Karhunen-Loeve transform (KLT), following the first transform. The following transformation may be applied: Video encoder 200 generates transform coefficients following application of one or more transforms.

[0069] As discussed above, following any transformation to generate transform coefficients, video encoder 200 may perform quantization of the transform coefficients. Quantization generally refers to a process in which transform coefficients are quantized to reduce as much as possible the amount of data used to represent the transform coefficients, providing further compression. By performing a quantization process, video encoder 200 may reduce the bit depth associated with some or all of the transform coefficients. For example, video encoder 200 may round an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, video encoder 200 may perform a bitwise right shift of the value to be quantized.

[0070] Following quantization, video encoder 200 may scan the transform coefficients to generate a one-dimensional vector from the two-dimensional matrix containing the quantized transform coefficients. The scan may be designed to place higher energy (and therefore lower frequency) transform coefficients at the front of the vector and lower energy (and therefore higher frequency) transform coefficients at the rear of the vector. In some examples, video encoder 200 utilizes a predefined scan order to scan the quantized transform coefficients to generate a serialized vector, and then serializes the quantized transform coefficients of the vector. Can be entropy encoded. In other examples, video encoder 200 may implement adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, video encoder 200 may entropy encode the one-dimensional vector, for example, according to context adaptive binary arithmetic coding (CABAC). Video encoder 200 may also entropy encode values for syntax elements that describe metadata associated with encoded video data for use by video decoder 300 in decoding the video data.

[0071] To implement CABAC, video encoder 200 may assign context in a context model to symbols to be transmitted. The context may relate, for example, to whether a symbol's neighboring values are zero values. The probability determination may be based on the context assigned to the symbol.

[0072] Video encoder 200 further converts syntax data such as block-based syntax data, picture-based syntax data, and sequence-based syntax data into, for example, a picture header, a block header, a slice header, or It may be generated for video decoder 300 in other syntax data such as a sequence parameter set (SPS), picture parameter set (PPS), or video parameter set (VPS). Video decoder 300 may similarly decode such syntax data to determine how to decode the corresponding video data.

[0073] In this manner, video encoder 200 generates encoded video data, eg, syntax elements that describe the partitioning of pictures into blocks (eg, CUs) and prediction and/or residual information for the blocks. A bitstream may be generated that includes: Ultimately, video decoder 300 may receive the bitstream and decode the encoded video data.

[0074] Generally, video decoder 300 performs the inverse process of that performed by video encoder 200 to decode encoded video data of a bitstream. For example, video decoder 300 may decode values for syntax elements of a bitstream using CABAC in a manner substantially similar to, but opposite to, the CABAC encoding process of video encoder 200. The syntax element may define partitioning information for partitioning a picture into CTUs and partitioning of each CTU according to a corresponding partitioning structure, such as a QTBT structure, to define a CU of a CTU. The syntax elements may further define prediction and residual information for a block (eg, CU) of video data.

[0075] The residual information may be represented by, for example, quantized transform coefficients. Video decoder 300 may dequantize and inverse transform the quantized transform coefficients of the block to recover a residual block for the block. Video decoder 300 uses the signaled prediction mode (intra or inter prediction) and associated prediction information (eg, motion information for inter prediction) to form a prediction block for the block. Video decoder 300 may then combine the predictive block and the residual block (sample by sample) to reconstruct the original block. Video decoder 300 may perform additional processing, such as performing a deblocking process to reduce visual artifacts along block boundaries.

[0076] This disclosure may generally refer to "signaling" certain information, such as syntax elements. The term "signaling" may generally refer to the communication of values for syntax elements and/or other data used to decode encoded video data. That is, video encoder 200 may signal values for syntax elements in the bitstream. Generally speaking, signaling refers to producing values in a bitstream. As described above, the source device 102 may transport the bitstream to the destination device 116 in substantially real time or store syntax elements in the storage device 112 for later retrieval by the destination device 116. As may be done, the bitstream may be transported to the destination device 116 in non-real time.

[0077]H. In video coding such as the H.266/VVC standard, ALF is applied to minimize the mean squared error between the filtered samples and the original samples. The input samples to the ALF may be, for example, the output samples of the SAO. The output samples of the ALF may be stored in a decoded picture buffer (DPB) or output as a visible picture. The filter shapes for ALF employed in the Joint Investigation Model (JEM) software were 5×5, 7×7, and 9×9 diamond shapes. In JEM, filter shapes can be selected and signaled at the picture level. To obtain a better trade-off between coding efficiency and filter complexity, in VVC, only 7x7 diamond shape and 5x5 diamond shape are used for luma and chroma components, respectively. Supported.

[0078] FIG. 2A shows an exemplary filter 140 that is a 7x7 diamond shaped filter. FIG. 2B shows an exemplary filter 142 that is a 5×5 diamond shaped filter. In each of filters 140 and 142, the integer coefficients ci are represented with 7 bits of fractional precision. The absolute value of c _i is coded by using a zeroth order exponential Golomb code, followed by a sign bit for non-zero coefficients. In FIGS. 2A and 2B, each square corresponds to a luma or chroma sample, with the center square corresponding to the sample currently being filtered. To reduce the overhead of transmitting coefficients and the number of multiplications, the filter shapes in FIGS. 2A and 2B are point symmetric. Additionally, as shown in equation (1), the sum of all filter coefficients is set equal to 128, which is a fixed point representation of 1.0 with 7 bits of fractional precision.

[0079] In equation (1), N is the number of coefficients, and N is equal to 13 and 7 for 7x7 and 5x5 filter shapes, respectively.

[0080] In VVC, nonlinearity is introduced into the ALF. When the difference between neighboring sample values and the currently filtered sample value is too large, a simple clipping function is applied to reduce the influence of neighboring sample values. To filter the samples, the ALF:

where R(x,y) is the sample value after SAO.

[0081] The nonlinear function uses a clipping function to

where j is equal to 0 or 1 and (x _i,j , y _i,j ) is the filter tap position offset of the i-th coefficient c _i .

[0082] In VVC version 1, the clipping parameter b _i for the coefficient c _i is determined by the clipping index d _i as shown in equation (4). BD is the internal bit depth.

[0083] For the filter, the number of signaled coefficients and the number of signaled clipping indices are both N-1. Each coefficient is limited to the range [-128, 127], which is equivalent to [-1.0, 1.0], with 7 bits of fractional precision. Each clipping index di can be 0, 1, 2, or 3 and is signaled by using a 2-bit fixed length code. As in equation (4), to simplify the clipping operation, the value of the clipping parameter b _i may be restricted to powers of two only. Therefore, a bitwise logical operation may be applied as a clipping operation.

[0084] Video encoder 200 and video decoder 300 may be configured to perform sub-block level filter adaptation. In VVC version 1, ALF follows the same luma classification framework as ALF in JEM-7.0. To obtain a better trade-off between coding efficiency and computational complexity, the block size for classification can be increased from 2x2 samples to 4x4 samples. To determine the class index of a 4x4 block, a boxed window containing 8x8 luma samples is utilized to derive orientation and activity information. In this 8×8 luma sample window, the four slope values of every second sample are first calculated, as shown in FIG. FIG. 3 shows subsampled Laplacian values for a 4×4 subblock 150 for ALF classification. The slope values of the samples marked with dots are calculated. The slope values of other samples are set to zero.

[0085] FIG. 4 shows the four slope values for each sample using coordinates (k,l). Dots represent samples for which gradients are being calculated. Block 160 shows the horizontal gradient (H) and block 162 shows the vertical gradient (V). Block 164 shows a 135 degree slope (D1) and block 166 shows a 45 degree slope (D2). H, V, D1, and D2 are derived as follows.

[0086] Variables i and j may indicate the coordinates of the top left sample in the 4x4 block. The sum of the calculated horizontal slopes g _H , the sum of the calculated vertical slopes g _V , the sum of the calculated 135-degree slopes g _D1 , and the sum of the calculated 45-degree slopes g _D2 are calculated as follows: Ru.

[0087] The ratio of the maximum value of the horizontal slope and vertical slope represented by R _H,V to the minimum value of the horizontal slope and vertical slope, and the maximum value of the two diagonal slopes represented by R _D1,D2. , and the minimum value of the two diagonal slopes is calculated as shown in equation (7).

[0088] To derive the directionality D, R _H,V and R _D1,D2 are then compared to each other using two thresholds t ₁ =2 and t ₂ =4.5.

Step 1: If both R _H,V ≦t ₁ and R _D1,D2 ≦t ₁ then D is set to 0 (texture), otherwise continue with step 2.

Step 2: If R _D1,D2 >R _H,V , continue with step 3, otherwise continue with step 4.

Step 3: If R _D1,D2 ≦t ₂ then D is set to 1 (weak diagonal), otherwise D is set to 2 (strong diagonal).

Step 4: If R _H,V ≦t ₂ , D is set to 3 (weak horizontal/vertical);
Otherwise, D is set to 4 (strong horizontal/vertical).

[0089] Activity value A is calculated as follows:

[0090] Figure 5 shows an example of merging 25 luma classes into 7 merged classes (0 to 6), where each rectangle has D and

Represents a class based on the value of . The 5x5 grid 170 represents 25 classes, and the numbers in each box of the 5x5 grid 170 represent merged classes from 0 to 6. Each class, ie, each square of the 5×5 grid 170, can have an index greater than or equal to 0 and less than or equal to 24. A is mapped to a range of 0 to 4, and the quantized value is

It is expressed as Therefore, each 4x4 block is categorized into one of 25 classes as follows.

[0091] The luma filter set includes 25 filters. However, to reduce the number of bits needed to represent filter coefficients while maintaining coding efficiency, different classes may be merged, and the merged classes use the same filter. A merging table is signaled. In the merging table, the filter index for each class is signaled using, for example, a fixed length code. In the exemplary filter set for FIG. 5, seven luma filters are signaled. For each class, the filter index (0 to 6 in this example) is signaled in ALF_APS.

[0092] After determining the filter from the luma filter set based on the class index C of the 4×4 block and the merging table, and before filtering the samples of the 4×4 block, as shown in Table 1, Depending on the gradient values calculated for the 4x4 block, a geometric transformation may be applied to the filter.

[0093] FIG. 6 shows an example of a geometric transformation of filter 140 in FIG. 2A. As can be seen in FIG. 6, filter 180 corresponds to the diagonal inversion of filter 140. Filter 182 corresponds to a vertical inversion of filter 140 and filter 184 corresponds to a clockwise rotation of filter 140.

[0094] Video encoder 200 and video decoder 300 may be configured to perform coding tree block-level adaptation. In JEM-7.0, only one luma filter set is applied to all luma CTBs of a slice, and only one chroma filter set is applied to all chroma CTBs of a slice. However, there are two potential disadvantages. First, when the statistics between CTBs differ to some extent, using the same filter or set of filters for all CTBs of color components is difficult, especially for high-resolution sequences, and mixed-content video sequences. may limit coding efficiency. Second, when deriving a filter for a slice, the filter cannot be computed until statistics for the entire slice have been collected. This multiple pass coding is not suitable for low-latency applications. To address this problem, one solution is to use statistics from previously coded slices. However, this may result in some performance degradation.

[0095] In addition to luma 4x4 block-level filter adaptation, VVC supports CTB-level filter adaptation. In a slice, different luma CTBs are allowed to use different luma filter sets, and different chroma CTBs can use different chroma filters. CTBs with similar statistics may use the same filter. This CTB level filter adaptation improves coding efficiency, especially for low delay applications. In addition, VVC version 1 allows filters from previously coded pictures to be used for CTB. This temporary filter reuse mechanism can reduce filter coefficient signaling overhead. In VVC version 1, up to seven signaled luma filter sets and eight signaled chroma filter sets may be applied to a slice. When no signaled filter is present, one of 16 fixed filter sets may be applied to the luma CTB. When ALF is enabled, a filter set index of either a fixed filter set or a signaled luma filter set is signaled for the luma CTB. The filter index of the signaled chroma filter is signaled for the chroma CTB. By using only the statistics of the current CTU when encoding the current CTU, in low-latency applications, by using filters signaled from previously coded pictures and fixed filters. A CTU level on/off flag and a filter/filter set index may be determined. Thus, the encoded bitstream for each CTU can be generated on the fly, without waiting for the whole picture statistics to become available.

[0096] Video encoder 200 and video decoder 300 may be configured to perform techniques for line buffer reduction. As shown in FIGS. 2A and 2B, in the vertical direction, the filter shape has 7 taps and 5 taps for the luma and chroma components, respectively. As a result, in VVC Test Model 2.0 (VTM-2.0), when decoding a CTU row, seven lumens in the upper CTU row are line and four chroma lines must be stored in the line buffer for the ALF. However, additional line buffers require large chip area, especially for high definition (HD) and ultra high definition (UHD) video sequences.

[0097] To make ALF hardware friendly (e.g., by reducing line buffer requirements), the concept of virtual boundaries (VB) is applied to remove all line buffer overhead for ALF. obtain. Considering the deblocking filter and SAO filter in VVC version 1, the location of VB is 4 luma samples and 2 chroma samples above the horizontal CTU boundary. When one sample on one side of the VB is filtered, samples on the other side of the VB may not be utilized and a modified filtering with symmetric sample padding may be applied.

[0098] FIGS. 7A-7C illustrate examples of symmetric sample padding for luma ALF filtering in ALF VB. In the example of FIGS. 7A-7C, the square in the center of filter 190 is the location of the current filtered sample, and the bold line is the location of VB (VB 192). In FIGS. 7A-7C, filter tap locations with dashed lines are padded. FIG. 7A shows an example where one filter tap position of filter 190 is above or below VB 192. In this example, one filter tap position is padded. FIG. 7B shows an example where the four filter tap positions of filter 190 are above or below VB 192. In this example, four filter tap positions are padded.

[0099] However, if the samples are on the nearest row on each side of the VB 192, as shown in FIG. 7C, the 2D filter is equivalent to a horizontal filter. This may introduce visual artifacts. To address this issue, the filter strength may be compensated when the current filtered sample is placed on the nearest row on each side of the VB, as shown in equation (10). Comparing equation (10) with equation (2), it can be seen that three more bits have been shifted to the right.

[0100] When VB processing is applied, the classification of 4x4 blocks may also be changed. When calculating the class index of a 4x4 block on one side of the VB, the gradients and samples on the other side of the VB cannot be used, as shown in FIG. 8.

[0101] FIG. 8 shows an example of ALF 4×4 subblock classification in ALF VB. When calculating slope values for samples adjacent to VB, samples on the other side of VB cannot be utilized. Therefore, the boundary samples on the current side are expanded iteratively, as shown in FIG. That is, the boundary samples on the current side of the VB are mirrored on the other side of the VB. Since the number of available slope values is reduced, the activity derivation in equation (8) becomes

will be rescaled to

[0102] Video encoder 200 and video decoder 300 may be configured to perform filter coefficient signaling. In VVC version 1, ALF coefficients are signaled in the ALF Adaptive Parameter Set (APS). One APS may include one set of luma filters with up to 25 filters, up to 8 chroma filters, and up to 8 cross-component ALF (CC-ALF) filters. Each set of luma filters supports applying ALF to 25 classes of luma. In VVC version 1, up to 8 ALF_APS are supported.

[0103] Table 2 below shows an example syntax signal table for signaling filter coefficients in accordance with the techniques of this disclosure.

[0104] This disclosure describes techniques that may further improve the performance of ALF. For example, in VVC, only one classifier and one filter may be applied when filtering samples. However, this disclosure has described techniques for applying multiple classifiers and filters to capture more local features. This disclosure describes techniques for a framework that uses multiple filters and classifiers to filter samples.

[0105] FIG. 9 illustrates an example framework for using multiple fixed filter sets and multiple signaled filter sets to filter reconstructed samples of video data. There is. Video encoder 200 and video decoder 300 may be configured to implement fixed filter sets and signaled filter sets according to the techniques of FIG. Although the techniques of FIGS. 10-15 are described with respect to video decoder 300, they may also be performed by video encoder 200.

[0106] In the example of FIG. 9, video decoder 300 applies a first stage ALF 400 to reconstructed samples R(x,y) to determine intermediate filtered signal R'. Video decoder 300 then applies a second stage ALF 410 to R' to determine a value for filtered samples 430.

[0107] Aspects of the first stage ALF 400 will now be described in more detail.
- N _s and N _f represent the number of signaled filter sets and fixed (predefined) filter sets that can be applied to the samples. Rs is the input sample to the ALF.
・F(f,i), i=0. ．． ．． N _f −1 represents the i-th fixed filter set.
・C(f,i), i=0. ．． ．． N _f -1 represents the classifier for the i-th fixed filter set. A filter index is calculated for each sample through C(f,i). Based on the filter index, a filter from the fixed filter set F(f,i) is selected to filter the sample. In addition, C(f,i) determines how to apply geometric transformations to the coefficients.
・F(s,i), i=0. ．． ．． N _s −1 represents the i-th signaled filter set.
・C(s,i), i=0. ．． ．． N _s -1 represents the classifier with the i-th signaled filter set. A filter index is calculated for each sample through C(s,i). Based on the filter index, a filter from the fixed filter set F(s,i) is selected to filter the sample. In addition, C(s,i) determines how to apply geometric transformations to the coefficients.

[0108] For the reconstructed sample R(x,y), by using the reconstructed sample itself and neighboring samples, video decoder 300 may apply multiple filter sets. Given a filter set (which may be a fixed filter set or a signaled filter set), a classifier may be applied. The classifier determines which filters from the filter set may be applied and how to transpose the filter coefficients. After the first stage, for each filter set and corresponding classifier, the intermediate filtered signal R' is, e.g.

It can be calculated as follows.

[0109] i-th filter, i=0. ．． ．． For N _f +N _s −1, c _i,j is the j-th coefficient of the selected filter from the i-th filter set, Ni is the number of coefficients, and f _{i,j,k ,} k =0 or 1 is a function of one neighboring sample and the current sample, for example f _i,j,k can be implemented as a clipping function, as in VVC.

(x _i,j,k , y _i,j,k ) are the coordinate offsets of adjacent samples with respect to the current sample.

[0110] In some examples, the filter shapes of fixed filters or signaled filters may be 5x5, 7x7, 9x9, 11x11, and 13x13. FIG. 10A shows an example filter 510 that is a 5×5 diamond filter shape, and FIG. 10B shows an example filter 520 that is a 7×7 diamond filter shape. FIG. 10C shows an example filter 530 that is a 9×9 diamond filter shape, and FIG. 10D shows an example filter 540 that is an 11×11 diamond filter shape. FIG. 10E shows an example filter 550 that is a 13×13 diamond filter shape.

[0111] In one example, video decoder 300 may be configured to determine classifier C(f/s,i) and activity and direction values based on 2-D Laplacian/gradient values. A classifier may be applied to each sample or block. When a classifier is applied to a block, all samples within the block have the same class index and the same transpose type. For example, w _i can represent the width of the block, h _i can represent the height of the block, and (x,y) can represent the coordinates of the top left sample of the block.

[0112] For a sample with coordinates (k, l), four Laplacian (slope) values are: horizontal slope H, vertical slope V, 135 degree slope D1, and 45 degree slope D2:

can be derived as
(14)
[0113] Video decoder 300 includes:

may be configured to derive the activity value A _i by using the vertical and horizontal gradients as, where a _i and b _i are respectively for the classifier C(f/s,i) is the window size in the horizontal and vertical directions.

[0114] Video decoder 300 may be configured to quantize A _i to a range of 0 to M _A,i -1. The quantized value is

It is expressed as

[0115] Video decoder 300 may be configured to determine direction using horizontal gradient H, vertical gradient V, 135 degree gradient D1, and 45 degree gradient D2.

[0116] Video decoder 300 first calculates the values of a horizontal gradient g _i,H , a vertical gradient g _i,V , and two diagonal gradients g _i,D1 and g _i,D2 as follows. It can be configured as follows.

[0117] To allocate the directionality D _i , video encoder 200 and video decoder 300 calculate the maximum and minimum values of the horizontal gradient and vertical gradient, the maximum and minimum values of the two diagonal gradients, and may be configured to derive.

Then, similar to VVC, the directions D _i can be derived, with the number of directions M _D,i =5.
Step 1.

and

If both, D _i is set to 0 (the block is categorized as "texture").
Step 2.

If so, continue from step 3, otherwise continue from step 4.
Step 3.

, then D _i is set to 2 (the block is categorized as “strong horizontal/vertical”); otherwise, D _i is set to 1 (the block is categorized as “weak horizontal/vertical”). ).
Step 4.

, then D _i is set to 4 (the block is categorized as a "strong diagonal"); otherwise, D _i is set to 3 (the block is categorized as a "weak diagonal"). ).

[0118] In some examples, more directions may be supported.

[0119] Video decoder 300 has a ratio

The horizontal/vertical edge strength (ES _{i,HV ) may be configured to be calculated by comparing the horizontal/vertical edge strength (ES i,HV} ) with an array of threshold values (Th). The size of the array of thresholds is represented by S, and the thresholds are sorted in ascending order. The number of directions is M _D,i = (S+1) x (S+2).

Step 1. Initialize m=0 and ES _i,HV =0.

Step 2. If m is equal to S, stop, otherwise go to step 3.

Step 3.

If so, set m=m+1, ES _i,HV =ES _i,HV +1, and proceed to step 2; otherwise, stop.

[0120] Video decoder 300 may be configured to calculate the diagonal direction (E _i,D ) as follows.

Step 1. Initialize m=0 and E _i,D =0.

Step 2. If m is equal to S, stop, otherwise go to step 3.

Step 3.

If so, proceed to step 2 with m=m+1 and E _i,D =E _i,D +1; otherwise, stop.

[0121] An example of the arrangement of threshold values is Th=[1.25, 1.5, 2, 3, 4.5, 8] and S=6.

[0122] Video decoder 300 may be configured to determine a primary edge strength (ES _M ) and a secondary edge strength (ES _S ) as follows.

, then ES _M is set as ES _HV and ES _S is set as ES _D ; otherwise, ES _M is set as ES _D and ES _S is set as ES _HV .

If ES _S is greater than ES _M , D _i is set to zero. In other cases,

, then D _i =ES _M ×(ES _M +1)/2+ES _S , otherwise D _i =ES _M ×(ES _M +1)/2+ES _S +M _D,i /2.

[0123] Video decoder 300 sets class index C _i to

can be configured to derive as follows.

[0124] Based on C _i , video encoder 200 and video decoder 300 may be configured to select a filter from filter set F(f/s,i).

[0125] Aspects of the second stage ALF 410 will now be described in more detail. In the second stage ALF 410, F' is the filter set and C' is the corresponding classifier. The intermediate filtering result may be further filtered using the current sample and its neighboring samples. C' may be used to determine which filters in F' are applied and how to transpose the coefficients. In one example, F' may be a signaled filter set or a fixed (predefined) filter set. In one example, C' calculates Rs and/or R's to determine the filter index for filter set F' by calculating the activity and direction in the same way as in the first stage. Can be used. When applying F', a transposition may be applied.

[0126] A fixed filter set F(f,i) and a corresponding classifier C(f,i) may be selected from one or more candidate fixed filter sets. In one example, given i, for fixed filter set F(f,i) and classifier C(f,i), there may be several candidate fixed filter sets. In addition, the fixed filter set F (f, i) and the classifier C (f, i) are based on the picture/CTU/CU/PU/TU size, the quantization parameter (QP), and the number of non-zero quantization coefficients. , motion vector, and window size (a _i , b _i ). In some examples, for each F(f, i), to indicate which fixed filter set is used from among the candidate fixed filter sets for the i-th fixed filter set. , a syntax element may be signaled. Syntax elements may be signaled as part of an SPS, PPS, VPS, APS, picture header (PH), slice header (SH), subpicture, CTU, or block. In some examples, all these syntax elements may have the same value and only one syntax element needs to be signaled. In some examples, only one candidate fixed filter set may be available for F(f,i), so the syntax element needs to be signaled. do not have.

[0127] In one example, N _s and N _f are implicitly determined or explicitly signaled at a high level, such as the SPS, VPS, APS, PH, SH, CTU, or sub-block level. obtain.

[0128] Video decoder 300 may be configured to implement multiple fixed filters and one signaled filter. In another example, to reduce bitstream signaling overhead while maintaining adaptability, video encoder 200 and video decoder 300 may be configured using only one filter, along with multiple fixed filters, as shown in FIG. The signaled filter may be configured to filter the current sample.

[0129] FIG. 11 illustrates an example framework for implementing multiple fixed filters with one signaled filter in accordance with the techniques of this disclosure. In the example of FIG. 11, video encoder 200 and video decoder 300 apply a first stage ALF 540 to reconstructed samples R(x,y) of the reconstructed block. To apply the first stage ALF 540, video encoder 200 and video decoder 300 apply a classifier (C) to determine a first class index for the reconstructed samples, and Select a filter (f) from the first set of filters (F) based on the index. Video encoder 200 and video decoder 300 apply filters from the first set of filters to the reconstructed samples to determine a first intermediate sample value (R'). As shown in FIG. 11, in a first stage 540, video encoder 200 and video decoder 300 may use classifiers and filters to determine intermediate sample values.

[0130] Video encoder 200 and video decoder 300 apply second stage ALF 542 to the reconstructed samples. Equation (28) below shows an example of how video encoder 200 and video decoder 300 may apply the second stage ALF.

[0131] N _f represents the number of fixed (predefined) filter sets that may be applied to the samples. Rs is the input to ALF. F(f,i), i=0. ．． ．． N _f −1 represents the i-th fixed filter set. C(f,i), i=0. ．． ．． N _f -1 represents the classifier with the i-th fixed filter set. A filter index is calculated for each sample through C(f,i). Based on the filter index, a filter from the fixed filter set F(f,i) is selected to filter the sample. In addition, C(f,i) determines how to apply geometric transformations to the coefficients.

[0132] The first stage will now be described. In the first stage, for the reconstructed sample R(x,y), the video encoder 200 and the video decoder 300 construct a plurality of fixed filter sets by using the reconstructed sample itself and adjacent samples. may be configured to apply. Given a fixed filter set F(f,i), video encoder 200 and video decoder 300 may apply a classifier C(f,i). Based on the derived class index, video encoder 200 and video decoder 300 may apply filters from filter set F(f,i). Additionally, a geometric transformation may be determined in C(f,i) and the geometric transformation may be applied to the coefficients of the selected filter. After the first stage, for each filter set and corresponding classifier, video encoder 200 and video decoder 300 may calculate an intermediate filtered signal R', for example, as follows.

[0133] i-th filter, i=0. ．． ．． For N _f −1, c _i,j is the j-th coefficient of the filter selected from the i-th filter set F(f,i), N _i is the number of coefficients, and f _{i,j ,k} , k=0 or 1 is a function of one neighboring sample and the current sample, for example f _i,j,k can be implemented as a clipping function, as in VVC.

(x _i,j,k , y _i,j,k ) is the coordinate offset of the adjacent sample with respect to the current sample, and b _i,j is the jth clipping of the selected filter from the i-th filter set. It is a parameter.

[0134] In some examples, the filter shapes of the fixed or signaled filters are 5x5, 7x7, 9x9, 11x11, and 13 as shown in FIG. It can be ×13.

[0135] In one example, video encoder 200 and video decoder 300 may determine activity values and direction values based on 2-D Laplacian values in classifier C(f,i). A classifier may be applied to each sample or block. When a classifier is applied to a block, all samples within the block have the same class index and the same transpose type. For example, w _i can represent the width of the block, h _i can represent the height of the block, and (x,y) can represent the coordinates of the top left sample of the block.

[0136] For a sample with coordinates (k,l), video encoder 200 and video decoder 300 calculate four Laplacian (gradient) values: horizontal gradient H, vertical gradient V, 135 degree gradient D1, as follows: , and a 45 degree slope D2.

[0137] Video encoder 200 and video decoder 300 may derive activity values A _i by using vertical and horizontal gradients as follows.

Here, a _i and b _i are the window sizes in the horizontal and vertical directions for the classifier C(f,i), respectively.

[0138] Video encoder 200 and video decoder 300 may be configured to quantize A _i to a range of 0 to M _A,i -1, and the quantized value is

It is expressed as

[0139] In one example,

bitdepth is the bit depth of R(x,y), mult _i may depend on the window size (a _i , b _i ), and mult _i may be equal to the following values:

[0140] Video encoder 200 and video decoder 300 may be configured to determine direction by using horizontal gradient H, vertical gradient V, 135 degree gradient D1, and 45 degree gradient D2.

[0141] Video encoder 200 and video decoder 300 calculate the values of horizontal gradient g _i,H , vertical gradient g _i,V , and two diagonal gradients g _i,D1 and g _i,D2 as follows. It may be configured to calculate first.

[0142] To allocate the directionality D _i , video encoder 200 and video decoder 300 calculate the maximum and minimum values of the horizontal gradient and vertical gradient, the maximum and minimum values of the two diagonal gradients, and may be configured to derive.

[0143] Video encoder 200 and video decoder 300 may then be configured to derive directions D _i as follows, where the number of directions M _D,i =5.
Step 1.

and

If both, D _i is set to 0 (the block is categorized as "texture").
Step 2.

If so, continue from step 3, otherwise continue from step 4.
Step 3.

, then D _i is set to 2 (the block is categorized as "strong horizontal/vertical"), otherwise D _i is set to 1 (the block is categorized as "weak horizontal/vertical") ).
Step 4.

, then D _i is set to 4 (the block is categorized as a “strong diagonal”); otherwise, D _i is set to 3. (The block is categorized as a "weak diagonal").

[0144] In some examples, more directions may be supported.

[0145] Video encoder 200 and video decoder 300 have a ratio

The horizontal/vertical edge strength (ES _{i,HV ) may be configured to be calculated by comparing the horizontal/vertical edge strength (ES i,HV} ) with an array of threshold values (Th). The size of the array of thresholds may be S, and the thresholds may be sorted in ascending order. The number of directions is M _D,i = (S+1) x (S+2).

Step 1. Initialize m=0 and ES _i,HV =0.

Step 2. If m is equal to S, stop, otherwise go to step 3.

Step 3.

If so, set m=m+1, ES _i,HV =ES _i,HV +1, and proceed to step 2; otherwise, stop.

[0146] Video encoder 200 and video decoder 300 may be configured to calculate the diagonal direction (E _i,D ) as follows.

Step 1. Initialize m=0 and E _i,D =0.

Step 2. If m is equal to S, stop, otherwise go to step 3.

Step 3.

If so, proceed to step 2 with m=m+1 and E _i,D =E _i,D +1; otherwise, stop.

[0147] An example of the arrangement of threshold values is Th=[1.25, 1.5, 2, 3, 4.5, 8] and S=6.

[0148] Video encoder 200 and video decoder 300 may be configured to determine a primary edge strength (ES _M ) and a secondary edge strength (ES _S ) as follows.

, then ES _M is set as ES _HV and ES _S is set as ES _D ; otherwise, ES _M is set as ES _D and ES _S is set as ES _HV .

If ES _S is greater than ES _M , D _i is set to zero. In other cases,

, then D _i =ES _M ×(ES _M +1)/2+ES _S , otherwise D _i =ES _M ×(ES _M +1)/2+ES _S +M _D,i /2.

The class index C _i is

can be derived as

Based on C _i , filters from C(f,i) are selected.

[0149] Video encoder 200 and video decoder 300 select a fixed filter set F(f,i) and a corresponding classifier C(f,i) from one or more candidate fixed filter sets. and can be selected. In one example, given i, for fixed filter set F(f,i) and classifier C(f,i), there may be several candidate fixed filter sets. In addition, the fixed filter set F(f,i) and the classifier C(f,i) are based on the picture/CTU/CU/PU/TU size, QP, the number of non-zero quantization coefficients, the motion vector, and The selection may be based on coding information, such as window size (a _i , b _i ). In some examples, for each F(f, i), to indicate which fixed filter set is used from among the candidate fixed filter sets for the i-th fixed filter set. , a syntax element may be signaled. Syntax elements may be signaled at the SPS, PPS, VPS, APS, PH, SH, subpicture, CTU, or block level. In some examples, all these syntax elements may have the same value and only one syntax element needs to be signaled. In some examples, the candidate fixed filter set may have only one fixed filter set, so no syntax element needs to be signaled.

[0150] In one example, given the window size (a _i , b _i ), the number of all available fixed filters is T. For example, QP may represent the quantization parameter of the current sample, and QP_MIN, QP_MAX, and QP_OFFSET are predefined values. The indices of the two candidate fixed filter sets are:

can be determined as To indicate whether for F(f,i), the _l0th candidate fixed filter set or the _l1th candidate fixed filter set is used. , syntax elements may be signaled. In some examples, all fixed filter sets may share the same syntax elements.

[0151] Aspects of the second stage will now be described. In the second stage, F' is the signaled filter or predefined filter set and C' is the corresponding classifier. The intermediate filtering result may be further filtered using the current sample and/or its neighboring samples. C' may be used to determine which filters in F' are applied and how to transpose the coefficients. As in the first stage, C' may use Rs and/or R's to determine the filter index for filter set F' by calculating the activity and direction. When applying F', a transposition may be applied.

[0152] In one example, for a sample with coordinates (k, l), four Laplacian (slope) values: horizontal slope H, vertical slope V, 135 degree slope D1, and 45 degree slope D2 are:

can be derived as

[0153] Video encoder 200 and video decoder 300 may be configured to derive an activity value A by using vertical and horizontal gradients as follows:

Here, a and b are the window sizes in the horizontal and vertical directions, respectively, for the classifier C', and w and h are the block sizes in which all samples have the same classification index and the same transposed index. width and height.

[0154] Video encoder 200 and video decoder 300 may be configured to quantize A to a range of 0 to M _A -1, and the quantized value is

It is expressed as for example,

bitdepth is the bit depth of R(x,y), mult may depend on the window size a×b, and mult may be equal to the following value:

[0155] Directions and how to perform the transposition can be found in U.S. Patent Publication No. 2017/0238020A1, U.S. Patent Publication No. 2017/0237981A1, or U.S. Pat. It may be calculated in the manner described in Publication No. 2017/0237982A1.

[0156] Further aspects of filtering will now be described. After obtaining the class index of the sample from C', video encoder 200 and video decoder 300 may be configured to select a filter from filter set F' based on the class index. Video encoder 200 and video decoder 300 are

may be configured to apply a filter, such as.

In (28), the filtering is divided into two parts.
・Filter part 1:

: By using neighboring samples filtering, a geometric transformation can be applied, where N ₀ is the number of coefficients of the applied filter shape. The filter may be a 5×5, 7×7, 9×9, 11×11, or 13×13 diamond filter, as shown in FIG.
・Filter part 2:

: By using intermediate filtered samples, filtering and geometric transformations can be applied. N ₀ and N ₁ represent the number of potentially unique coefficients.

The function f _i,j , j=0 or 1 is

can be defined using a clipping function, such as

The function g _i is

can be defined using a clipping function, such as

b _i is a clipping parameter corresponding to coefficient c _i .

[0157] Figures 12A-12E illustrate examples of two-part filters. FIG. 12A shows an example where filter portion 1 (filter 560) is 5×5 N ₁ =7 and for filter portion 2 (filter 562) N ₁ =2. FIG. 12B shows an example where filter portion 1 (filter 564) is 7×7 N ₀ =13 and for filter portion 2 (filter 566) N ₁ =2. FIG. 12C shows an example where filter portion 1 (filter 568) is 9×9 N ₁ =21 and for filter portion 2 (filter 570) N ₁ =2. FIG. 12D shows an example where filter portion 1 (filter 572) is 11×11 N ₁ =31 and for filter portion 2 (filter 574) N ₁ =2. FIG. 12E shows an example where filter portion 1 (filter 576) is 13×13 N ₁ =43 and for filter portion 2 (filter 578) N ₁ =2.

[0158] In some examples, the values of N ₀ and/or N ₁ are explicitly signaled or implicitly signaled at the SPS, VPS, PPS, APS, PH, SH, subpicture header, or block level. can be determined.

[0159] In some examples, a flag is explicitly signaled, or when this flag is equal to one value, all coefficients c _i , i=0 . ．． ．． N ₀ −1 and clipping parameter b _i , i=0. ．． ．． At the SPS, VPS, PPS, APS, PH, SH, sub-picture header, block level, or filter level, N ₀ -1 is not necessarily signaled/used and is inferred to be 0. May be determined implicitly.

[0160] In some examples, a flag is explicitly signaled or when this flag is equal to one value, all coefficients c _i , i=N ₀ . ．． ．． N ₀ +N ₁ -1 and clipping parameter b _i , i=N ₀ . ．． ．． SPS, VPS, PPS, APS, PH, SH, sub-picture header, block level or filter level so that N ₀ +N ₁ -1 is not necessarily signaled/used and is inferred to be 0. can be determined implicitly.

[0161] In one filter set, the value of N ₀ may be different from filter to filter. In one filter set, the value of N ₁ may be different from filter to filter.

[0162] This disclosure recognizes that ALF performance may be improved over the techniques described in JVET-U0100 and US Provisional Patent Application No. 63/130,275. When only filtered samples from the first stage are preferred and no filtering in the second stage is required, the filter coefficients c _i of F', i=0. ．． ．． N ₀ -2 are still signaled and they are all zeros. However, it may not be necessary to signal all these zero coefficients. Therefore, alternative signaling processes may be applied to improve coding efficiency.

[0163] In some examples, video encoder 200 and video decoder 300 include data representing one or more filters F(f,i) in the bitstream, rather than using only fixed filters. may be configured to code. Additionally or alternatively, video encoder 200 and video decoder 300 may be configured to determine whether to perform a second stage of filtering as discussed above. When the second stage is not performed, video encoder 200 and video decoder 300 are configured to code coefficients for one or more filters applied during the second stage. This may reduce signaling overhead in the bitstream and processing performed by both video encoder 200 and video decoder 300.

[0164] FIG. 13 is a conceptual diagram illustrating an example ALF framework in accordance with the techniques of this disclosure. Video decoder 300 may be configured to perform the techniques described with respect to FIG. As shown in FIG. 13, the example ALF framework includes both a first stage 580 and a second stage 582. According to the techniques of this disclosure, video encoder 200 and video decoder 300 may use filter usage information to determine whether the second stage should be performed. When the second stage is not to be performed, video encoder 200 and video decoder 300 may bypass filtering (C', F') in the second stage, as shown in FIG. 13. Therefore, video encoder 200 and video decoder 300 calculate intermediate filtered samples (R'(x,y,0)...R'(x,y, Only N _f −1)) can be used.

[0165] Further, video decoder 300, for example,

Accordingly, a weighted sum of intermediate filtered samples from various filters may be calculated, where w _i are the weights of R'(x,y,i).

[0166] In some examples, filter usage information may be signaled at the sequence, picture, slice, subpicture, CTU, CTB, or subblock level. For example, video encoder 200 and video decoder 300 may use sequence parameter sets (SPS), picture parameter sets (PPS), slice headers, subpicture headers, coding tree unit (CTU) headers, coding tree block (CTB) headers, or Filter usage information may be coded within one or more of the block headers.

[0167] In another example, filter usage information may not be signaled. Instead, video decoder 300 may adaptively derive filter usage information based on coding information, such as quantization parameter (QP), block size, inter/intra mode, or other such data.

[0168] Weight w _i , i=0. ．． ．． N _f −1 may be fixed given the value of the filter usage information, or the video encoder 200 and video decoder 300 may determine the bitstream at the sequence, picture, slice, subpicture, CTU, CTB, subblock level. A value for the weight can be coded within. Several combinations of weights w _i may be used, and filter usage information may be used to determine which combination of weights is used for the current sample.

[0169] In one example, a first flag may be signaled. When this first flag is equal to 1, video encoder 200 and video decoder 300 may bypass the value (C', F'). A second flag may then be signaled. Video encoder 200 and video decoder 300 set the value of the second flag to weight w _i , i=0. ．． ．． It can be associated with N _f −1 combinations. For example, if the second flag is equal to i and i is 0. ．． ．． When N _f -1, R'(x,y,i) can be used as the filtered sample.

[0170] FIG. 14 is a conceptual diagram illustrating another example ALF framework in accordance with the techniques of this disclosure. Video encoder 200 and video decoder 300 may be configured according to the techniques described with respect to FIG. In particular, in FIG. 14, scaling may be applied to the filtering results to obtain filtered samples.

[0171] FIG. 14 illustrates an example framework for using multiple fixed filter sets and multiple signaled filter sets to filter reconstructed samples of video data. There is. Video decoder 300 may be configured to implement fixed filters and signaled filters according to the techniques of FIG. 14. In the example of FIG. 14, video decoder 300 applies a first stage ALF 584 to the reconstructed samples R(x,y) to determine an intermediate filtered signal R'. Video decoder 300 then applies a second ALF stage 586 to R' to determine a value for filtered samples 588.

[0172] FIG. 15 is a block diagram illustrating an example video encoder 200 that may implement the techniques of this disclosure. FIG. 15 is provided for illustrative purposes and should not be considered limiting of the techniques as broadly illustrated and described in this disclosure. For purposes of explanation, this disclosure describes a video encoder 200 that follows the techniques of VVC (ITU-T H.266, under development) and HEVC (ITU-T H.265). However, the techniques of this disclosure may be performed by video encoding devices configured for other video coding standards.

[0173] In the example of FIG. 15, video encoder 200 includes video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, and inverse quantization unit 210. , an inverse transform processing unit 212 , a reconstruction unit 214 , a filter unit 216 , a DPB 218 , and an entropy encoding unit 220 . A video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, and a reconstruction unit 214. , filter unit 216, DPB 218, and entropy encoding unit 220 may be implemented in one or more processors or in processing circuitry. For example, a unit of video encoder 200 may be implemented as one or more circuits or logic elements, as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Moreover, video encoder 200 may include additional or alternative processors or processing circuits to perform these and other functions.

[0174] Video data memory 230 may store video data to be encoded by components of video encoder 200. Video encoder 200 may receive video data stored in video data memory 230, for example, from video source 104 (FIG. 1). DPB 218 may act as a reference picture memory that stores reference video data for use in predicting subsequent video data by video encoder 200. Video data memory 230 and DPB 218 may include DRAM, including synchronous dynamic random access memory (DRAM) (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. , may be formed by any of a variety of memory devices. Video data memory 230 and DPB 218 may be provided by the same memory device or separate memory devices. In various examples, video data memory 230 may be on-chip with other components of video encoder 200, as shown, or off-chip with respect to those components.

[0175] In this disclosure, references to video data memory 230 should not be construed as limited to memory internal to video encoder 200 unless or specifically described as such. It should not be construed as limited to memory external to video encoder 200 unless otherwise specified. Rather, references to video data memory 230 refer to it as a reference memory for storing video data that video encoder 200 receives for encoding (e.g., video data for the current block to be encoded). should be understood. Memory 106 of FIG. 1 may also provide temporary storage of output from various units of video encoder 200.

[0176] The various units in FIG. 15 are shown to assist in understanding the operations performed by video encoder 200. A unit may be implemented as a fixed function circuit, a programmable circuit, or a combination thereof. Fixed function circuit refers to a circuit that provides a specific function and is preset with respect to the operations that may be performed. Programmable circuit refers to a circuit that can be programmed to perform a variety of tasks and to provide flexible functionality in the operations that may be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. Although fixed function circuits may execute software instructions (eg, to receive parameters or output parameters), the types of operations that fixed function circuits perform generally remain unchanged. In some examples, one or more of the units can be discrete circuit blocks (fixed function or programmable), and in some examples, one or more of the units can be integrated circuits. obtain.

[0177] Video encoder 200 may include a programmable core formed from an arithmetic logic unit (ALU), an elementary functional unit (EFU), digital circuitry, analog circuitry, and/or programmable circuitry. In examples where the operations of video encoder 200 are implemented using software executed by programmable circuitry, memory 106 (FIG. 1) stores software instructions (e.g., object code) that video encoder 200 receives and executes. , or another memory (not shown) within video encoder 200 may store such instructions.

[0178] Video data memory 230 is configured to store received video data. Video encoder 200 may retrieve pictures of video data from video data memory 230 and provide video data to residual generation unit 204 and mode selection unit 202. The video data in video data memory 230 may be raw video data that is to be encoded.

[0179] Mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra prediction unit 226. Mode selection unit 202 may include additional functional units for performing video prediction according to other prediction modes. By way of example, mode selection unit 202 may include a palette unit, an intra block copy unit (which may be part of motion estimation unit 222 and/or motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0180] Mode selection unit 202 generally coordinates multiple encoding passes to test combinations of encoding parameters and resulting rate-distortion values for such combinations. The encoding parameters may include partitioning of the CTU into CUs, a prediction mode for the CU, a transform type for the residual data of the CU, a quantization parameter for the residual data of the CU, and so on. Mode selection unit 202 may ultimately select a combination of encoding parameters that has a rate-distortion value that is better than other tested combinations.

[0181] Video encoder 200 may partition a picture retrieved from video data memory 230 into a series of CTUs and encapsulate one or more CTUs within a slice. Mode selection unit 202 may partition the CTUs of a picture according to a tree structure, such as the QTBT structure or quadtree structure of HEVC described above. As explained above, video encoder 200 may form one or more CUs from partitioning the CTUs according to a tree structure. Such a CU may also be generally referred to as a "video block" or "block."

[0182] Generally, the mode selection unit 202 also selects a component thereof (e.g., the current CU, or in HEVC, the overlapping portion of the PU and TU) to generate a predictive block for the current block (e.g., the current CU, or in HEVC, the overlapping portion of the PU and TU). , a motion estimation unit 222, a motion compensation unit 224, and an intra prediction unit 226). For inter-prediction of the current block, motion estimation unit 222 uses one or more exact predictions in one or more reference pictures (e.g., one or more previously coded pictures stored in DPB 218). Motion searching may be performed to identify matching reference blocks. In particular, motion estimation unit 222 determines the potential references to the current block according to, e.g., sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), etc. A value representing how similar the blocks are can be calculated. Motion estimation unit 222 may generally perform these calculations using sample-by-sample differences between the current block and the considered reference block. Motion estimation unit 222 may identify the reference block with the lowest value obtained from these calculations, indicating the reference block that most closely matches the current block.

[0183] Motion estimation unit 222 may form one or more motion vectors (MVs) that define the position of a reference block in a reference picture relative to the position of a current block in a current picture. Motion estimation unit 222 may then provide the motion vector to motion compensation unit 224. For example, for unidirectional inter-prediction, motion estimation unit 222 may provide a single motion vector, whereas for bidirectional inter-prediction, motion estimation unit 222 may provide two motion vectors. Motion compensation unit 224 may then generate a predictive block using the motion vector. For example, motion compensation unit 224 may use motion vectors to retrieve data for reference blocks. As another example, if the motion vector has partial sample accuracy, motion compensation unit 224 may interpolate the values for the predictive block according to one or more interpolation filters. Moreover, in bidirectional inter-prediction, the motion compensation unit 224 retrieves data for the two reference blocks identified by their respective motion vectors and, e.g., through sample-by-sample averaging or weighted averaging, the retrieved data can be combined.

[0184] As another example, for intra prediction, or intra predictive coding, intra prediction unit 226 may generate a predictive block from samples that are adjacent to the current block. For example, in directional mode, intra prediction unit 226 generally mathematically combines the values of adjacent samples and populates these calculated values in a defined direction across the current block to generate a predicted block. obtain. As another example, in DC mode, intra prediction unit 226 may calculate the average of neighboring samples for the current block and generate a predictive block to include this resulting average for each sample of the predictive block.

[0185] Mode selection unit 202 provides the prediction block to residual generation unit 204. Residual generation unit 204 receives the raw, uncoded version of the current block from video data memory 230 and receives the predictive block from mode selection unit 202 . Residual generation unit 204 calculates the sample-by-sample difference between the current block and the predicted block. The resulting sample-by-sample differences define a residual block for the current block. In some examples, residual generation unit 204 may also determine differences between sample values in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). . In some examples, residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

[0186] In an example where mode selection unit 202 partitions CUs into PUs, each PU may be associated with a luma prediction unit and a corresponding chroma prediction unit. Video encoder 200 and video decoder 300 may support PUs with various sizes. As indicated above, the size of a CU may refer to the size of a luma coding block of a CU, and the size of a PU may refer to a size of a luma prediction unit of a PU. Assuming that the size of a particular CU is 2N×2N, video encoder 200 uses a PU size of 2N×2N or N×N for intra prediction and 2N×2N, 2N×N, Symmetric PU sizes of N×2N, N×N, or the like may be supported. Video encoder 200 and video decoder 300 may also support asymmetric partitioning of PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter prediction.

[0187] In examples where mode selection unit 202 does not further partition the CUs into PUs, each CU may be associated with a luma coding block and a corresponding chroma coding block. As mentioned above, the size of a CU may refer to the size of the luma coding block of the CU. Video encoder 200 and video decoder 300 may support CU sizes of 2N×2N, 2N×N, or N×2N.

[0188] For other video coding techniques, such as intra block copy mode coding, affine mode coding, and linear model (LM) mode coding, as some examples, mode selection unit 202 selects each The unit generates a predictive block for the current block being encoded. In some examples, such as palette mode coding, mode selection unit 202 may not generate predictive blocks, but instead generate syntax elements indicating how the blocks should be reconstructed based on the selected palette. . In such a mode, mode selection unit 202 may provide these syntax elements to be encoded to entropy encoding unit 220.

[0189] As explained above, residual generation unit 204 receives video data for the current block and a corresponding predictive block. Residual generation unit 204 then generates a residual block for the current block. To generate a residual block, residual generation unit 204 calculates the sample-by-sample difference between the predicted block and the current block.

[0190] Transform processing unit 206 applies one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form a transform coefficient block. For example, transform processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, transform processing unit 206 may perform multiple transforms on the residual block, e.g., linear transforms and quadratic transforms, such as rotation transforms. In some examples, transform processing unit 206 does not apply a transform to the residual block.

[0191] Quantization unit 208 may quantize the transform coefficients in the transform coefficient block to generate a quantized transform coefficient block. Quantization unit 208 may quantize the transform coefficients of the transform coefficient block according to a quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode selection unit 202) may adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. . Quantization may result in a loss of information, and thus the quantized transform coefficients may have lower accuracy than the original transform coefficients produced by transform processing unit 206.

[0192] Inverse quantization unit 210 and inverse transform processing unit 212 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficient block to reconstruct a residual block from the transform coefficient block. . Reconstruction unit 214 generates a reconstructed block corresponding to the current block (potentially with some distortion) based on the reconstructed residual block and the prediction block generated by mode selection unit 202. Can generate blocks. For example, reconstruction unit 214 may add samples of the reconstructed residual block to corresponding samples from the predictive block generated by mode selection unit 202 to generate a reconstructed block.

[0193] Filter unit 216 may perform one or more filter operations on the reconstructed block. For example, filter unit 216 may perform a deblocking operation to reduce blockiness artifacts along the edges of the CU. Operation of filter unit 216 may be skipped in some examples.

[0194] Video encoder 200 stores the reconstructed blocks in DPB 218. For example, in instances where the operations of filter unit 216 are not performed, reconstruction unit 214 may store reconstructed blocks in DPB 218. In examples where the operations of filter unit 216 are implemented, filter unit 216 may store the filtered reconstructed blocks in DPB 218 . Motion estimation unit 222 and motion compensation unit 224 extract reference pictures formed from the reconstructed (and potentially filtered) blocks from DPB 218 in order to inter-predict blocks of pictures that are subsequently coded. It can be taken out. Additionally, intra prediction unit 226 may use the reconstructed blocks in DPB 218 of the current picture to intra predict other blocks in the current picture.

[0195] Generally, entropy encoding unit 220 may entropy encode syntax elements received from other functional components of video encoder 200. For example, entropy encoding unit 220 may entropy encode the quantized transform coefficient block from quantization unit 208. As another example, entropy encoding unit 220 may entropy encode prediction syntax elements (eg, motion information for inter prediction or intra mode information for intra prediction) from mode selection unit 202. Entropy encoding unit 220 may perform one or more entropy encoding operations on syntax elements, another example of video data, to generate entropy encoded data. For example, entropy encoding unit 220 may perform a context adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable-to-variable (V2V) length coding operation, a syntax-based context adaptive binary arithmetic coding (SBAC) operation, a probability interval partitioning An entropy (PIPE) coding operation, an exponential Golomb coding operation, or another type of entropy coding operation may be performed on the data. In some examples, entropy encoding unit 220 may operate in a bypass mode in which syntax elements are not entropy encoded.

[0196] Video encoder 200 may output a bitstream that includes entropy encoding syntax elements needed to reconstruct a slice or block of a picture. In particular, entropy encoding unit 220 may output a bitstream.

[0197] The operations described above are described in terms of blocks. Such descriptions should be understood as being operations for luma coding blocks and/or chroma coding blocks. As explained above, in some examples, the luma and chroma coding blocks are the luma and chroma components of the CU. In some examples, the luma coding block and chroma coding block are the luma and chroma components of the PU.

[0198] In some examples, the operations performed for luma coding blocks need not be repeated for chroma coding blocks. As an example, the operations for identifying motion vectors (MVs) and reference pictures for luma coding blocks do not need to be repeated to identify MVs and reference pictures for chroma blocks. Rather, the MV for the luma coding block may be scaled to determine the MV for the chroma block, and the reference picture may be the same. As another example, the intra prediction process may be the same for luma coding blocks and chroma coding blocks.

[0199] FIG. 16 is a block diagram illustrating an example video decoder 300 that may implement the techniques of this disclosure. FIG. 16 is provided for illustrative purposes and is not limiting of the techniques broadly illustrated and described in this disclosure. For purposes of explanation, this disclosure describes a video decoder 300 that follows the techniques of VVC (ITU-T H.266, under development) and HEVC (ITU-T H.265). However, the techniques of this disclosure may be implemented by video coding devices configured to other video coding standards.

[0200] In the example of FIG. 16, video decoder 300 includes a coded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, and an inverse transform processing unit 308. , a reconstruction unit 310, a filter unit 312, and a DPB 314. Any or all of the CPB memory 320, the entropy decoding unit 302, the prediction processing unit 304, the inverse quantization unit 306, the inverse transform processing unit 308, the reconstruction unit 310, the filter unit 312, and the DPB 314 , may be implemented in one or more processors or in processing circuitry. For example, a unit of video decoder 300 may be implemented as one or more circuits or logic elements, as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Moreover, video decoder 300 may include additional or alternative processors or processing circuits to perform these and other functions.

[0201] Prediction processing unit 304 includes a motion compensation unit 316 and an intra prediction unit 318. Prediction processing unit 304 may include additional units for performing predictions according to other prediction modes. By way of example, prediction processing unit 304 may include a palette unit, an intra block copy unit (which may form part of motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, video decoder 300 may include more, fewer, or different functional components.

[0202] CPB memory 320 may store video data, such as an encoded video bitstream, to be decoded by components of video decoder 300. Video data stored in CPB memory 320 may be obtained from computer-readable medium 110 (FIG. 1), for example. CPB memory 320 may include a CPB that stores encoded video data (eg, syntax elements) from an encoded video bitstream. CPB memory 320 may also store video data other than syntactic elements of coded pictures, such as temporary data representing output from various units of video decoder 300. DPB 314 generally stores decoded pictures that video decoder 300 may output and/or use as reference video data when decoding subsequent data or pictures of the encoded video bitstream. CPB memory 320 and DPB 314 may be formed by any of a variety of memory devices, such as DRAM, including SDRAM, MRAM, RRAM, or other types of memory devices. CPB memory 320 and DPB 314 may be provided by the same memory device or separate memory devices. In various examples, CPB memory 320 may be on-chip with other components of video decoder 300 or off-chip with respect to those components.

[0203] Additionally or alternatively, in some examples, video decoder 300 may retrieve coded video data from memory 120 (FIG. 1). That is, memory 120 may store data as described above in conjunction with CPB memory 320. Similarly, memory 120 may store instructions to be executed by video decoder 300 when some or all of the functionality of video decoder 300 is implemented in software to be executed by processing circuitry of video decoder 300. .

[0204] The various units shown in FIG. 16 are shown to assist in understanding the operations performed by video decoder 300. A unit may be implemented as a fixed function circuit, a programmable circuit, or a combination thereof. Similar to FIG. 15, fixed function circuits refer to circuits that provide a specific function and are preset with respect to the operations that can be performed. Programmable circuit refers to a circuit that can be programmed to perform a variety of tasks and to provide flexible functionality in the operations that may be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. Although fixed function circuits may execute software instructions (eg, to receive parameters or output parameters), the types of operations that fixed function circuits perform generally remain unchanged. In some examples, one or more of the units can be discrete circuit blocks (fixed function or programmable), and in some examples, one or more of the units can be integrated circuits. obtain.

[0205] Video decoder 300 may include a programmable core formed from an ALU, an EFU, digital circuits, analog circuits, and/or programmable circuits. In examples where the operations of video decoder 300 are implemented by software executing on programmable circuitry, on-chip or off-chip memory stores software instructions (e.g., object code) that video decoder 300 receives and executes. obtain.

[0206] Entropy decoding unit 302 may receive encoded video data from the CPB and entropy decode the video data to reproduce syntax elements. The prediction processing unit 304, the inverse quantization unit 306, the inverse transform processing unit 308, the reconstruction unit 310, and the filter unit 312 generate decoded video data based on syntax elements extracted from the bitstream. obtain.

[0207] Generally, video decoder 300 reconstructs pictures block by block. Video decoder 300 may perform a reconstruction operation on each block individually (where the block currently being reconstructed, i.e., the block currently being decoded, may be referred to as the "current block"). ).

[0208] Entropy decoding unit 302 includes syntax elements that define quantized transform coefficients of a quantized transform coefficient block, as well as quantization parameters (QP) and/or transform mode indication(s). It is possible to entropy decode the transformation information such as . Dequantization unit 306 determines the degree of quantization and, similarly, the QP associated with the quantized transform coefficient block to determine the degree of dequantization that dequantization unit 306 should apply. Can be used. Dequantization unit 306 may perform, for example, a bitwise left shift operation to dequantize the quantized transform coefficients. Inverse quantization unit 306 may thereby form a transform coefficient block that includes transform coefficients.

[0209] After inverse quantization unit 306 forms the transform coefficient block, inverse transform processing unit 308 applies one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. can be applied. For example, inverse transform processing unit 308 may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotation transform, an inverse transform, or another inverse transform to the block of transform coefficients.

[0210] Further, prediction processing unit 304 generates a prediction block according to the prediction information syntax element entropy decoded by entropy decoding unit 302. For example, motion compensation unit 316 may generate a predictive block if the prediction information syntax element indicates that the current block is inter-predicted. In this case, the prediction information syntax element may indicate the reference picture in DPB 314 from which the reference block is to be retrieved, as well as a motion vector that identifies the location of the reference block in the reference picture relative to the location of the current block in the current picture. Motion compensation unit 316 may generally perform the inter-prediction process in a manner substantially similar to that described with respect to motion compensation unit 224 (FIG. 15).

[0211] As another example, if the prediction information syntax element indicates that the current block is intra-predicted, intra prediction unit 318 generates the prediction block according to the intra prediction mode indicated by the prediction information syntax element. obtain. Again, intra prediction unit 318 may generally perform the intra prediction process in a manner substantially similar to that described with respect to intra prediction unit 226 (FIG. 15). Intra prediction unit 318 may retrieve data for adjacent samples for the current block from DPB 314 .

[0212] Reconstruction unit 310 may reconstruct the current block using the predictive block and the residual block. For example, reconstruction unit 310 may add samples of the residual block to corresponding samples of the predictive block to reconstruct the current block.

[0213] Filter unit 312 may perform one or more filter operations on the reconstructed block. For example, filter unit 312 may perform a deblocking operation to reduce blockiness artifacts along the edges of the reconstructed block. The operation of filter unit 312 is not necessarily implemented in all examples.

[0214] Video decoder 300 may store the reconstructed blocks in DPB 314. For example, in instances where the operations of filter unit 312 are not performed, reconstruction unit 310 may store reconstructed blocks in DPB 314. In examples where the operations of filter unit 312 are implemented, filter unit 312 may store filtered reconstructed blocks in DPB 314 . As explained above, DPB 314 may provide reference information to prediction processing unit 304, such as samples of the current picture for intra prediction and previously decoded pictures for subsequent motion compensation. Additionally, video decoder 300 may output decoded pictures (eg, decoded video) from DPB 314 for subsequent presentation on a display device, such as display device 118 of FIG. 1.

[0215] FIG. 17 shows an example implementation of filter unit 312 in FIG. 16. Filter unit 216 in FIG. 15 may be implemented in the same manner. Filter units 216, 312 may perform the techniques of this disclosure, possibly in conjunction with other components of video encoder 200 or video decoder 300. In FIG. 16, filter unit 312 includes a deblock filter 342, an SAO filter 344, and an ALF unit 346. SAO filter 344 may be configured to determine offset values for the samples of the block, for example, in the manner described in this disclosure. ALF unit 346 may similarly filter blocks of video data in the manner described in this disclosure. For example, ALF unit 346 applies a first stage ALF to the reconstructed samples of the reconstruction block to determine a first sample modification value, and to determine a second sample modification value. applying a second stage ALF to the reconstructed sample and determining filtered reconstructed samples based on the reconstructed sample, the first sample modification value, and the second sample modification value; can be configured.

[0216] Filter unit 312 may include fewer filters and/or may include additional filters. Additionally, the particular filters shown in FIG. 17 may be implemented in a different order. Other loop filters (either within or after the coding loop) may be used to smooth pixel transitions or otherwise improve video quality. The filtered reconstructed video blocks output by filter unit 312 may be stored within DPB 314, which stores reference pictures used for subsequent motion compensation. DPB 314 may be part of, or separate from, additional memory that stores the decoded video for later presentation on a display device, such as display device 118 of FIG. 1.

[0217] FIG. 18 is a flowchart illustrating an example process for encoding a current block in accordance with the techniques of this disclosure. The current block may comprise the current CU. Although described with respect to video encoder 200 (FIGS. 1 and 14), it should be understood that other devices may be configured to implement a process similar to that of FIG. 18.

[0218] In this example, video encoder 200 first predicts the current block (650). For example, video encoder 200 may form a predictive block for a current block. Video encoder 200 may then calculate a residual block for the current block (652). To compute the residual block, video encoder 200 may compute the difference between the original unencoded block and the predictive block for the current block. Video encoder 200 may then transform the residual block and quantize the transform coefficients of the residual block (654). Video encoder 200 may then scan the quantized transform coefficients of the residual block (656). During or following scanning, video encoder 200 may entropy encode the transform coefficients (658). For example, video encoder 200 may encode transform coefficients using CAVLC or CABAC. Video encoder 200 may then output entropy encoded data for the block (660).

[0219] FIG. 19 is a flowchart illustrating an example process for decoding a current block of video data in accordance with the techniques of this disclosure. The current block may comprise the current CU. Although described with respect to video decoder 300 (FIGS. 1 and 15), it should be understood that other devices may be configured to perform a process similar to that of FIG. 19.

[0220] Video decoder 300 may receive entropy encoded data for the current block, such as entropy encoded prediction information and entropy encoded data for transform coefficients of a residual block corresponding to the current block (670). Video decoder 300 may entropy decode the entropy encoded data to determine prediction information for the current block and to recover transform coefficients of the residual block (672). Video decoder 300 may predict the current block using, for example, an intra-prediction or inter-prediction mode indicated by the prediction information for the current block to calculate a prediction block for the current block (674). . Video decoder 300 may then backscan the reconstructed transform coefficients to create a block of quantized transform coefficients (676). Video decoder 300 may then inverse quantize the transform coefficients and apply an inverse transform to the transform coefficients to generate a residual block (678). Video decoder 300 may finally decode the current block by combining the predictive block and the residual block (680).

[0221] FIG. 20 is a flowchart illustrating an example process for decoding a current block of video data in accordance with the techniques of this disclosure. Although described with respect to video decoder 300 (FIGS. 1 and 15), it should be understood that other devices may be configured to perform a process similar to that of FIG. 20. For example, the video decoding loop of video encoder 200, including filter unit 216, may perform the technique of FIG. 20.

[0222] Video decoder 300 applies a first stage ALF to the reconstructed samples of the reconstruction block, where applying the first stage ALF comprises (710). In this context, the reconstructed samples can be either reconstructed samples before deblock filtering or reconstructed samples after deblock filtering. To apply the first stage ALF, video decoder 300 determines a first class index for the reconstructed samples (712) and, based on the first class index, selects a filter from a first set of filters. A filter is selected (714) and a filter from the first set of filters is applied (716) to the reconstructed sample to determine a first intermediate sample value. As explained above, video decoder 300 may select a first set of filters from multiple sets of fixed filters based on a quantization parameter for the reconstruction block. The first set of filters may be, for example, fixed filters. That is, the first set of filters may be a set of filters stored within the video codec rather than signaled with the video data. The first set of filters may include, for example, at least one 9x9 diamond-shaped filter, or any size or shape filter described above. As shown in FIGS. 9 and 11, a video decoder may determine multiple intermediate sample values.

[0223] To determine the first class index, video decoder 300 determines an activity value for the reconstructed sample, determines a direction for the reconstructed sample, and based on the activity value and the direction: A classifier may be determined. As explained above, the number of directions can be M _D,i = (S+1) x (S+2), which means with S equal to 6, there are 56 directions. .

[0224] Video decoder 300 applies a second stage ALF to the reconstructed samples (720). To apply the second stage ALF, video decoder 300 determines a second class index for the reconstructed samples (722) and, based on the second class index, filters from the second set of filters. selecting (724) a second filter and applying (726) the second filter to the reconstructed sample to determine a first sample modification value; Determine the sample change value of 2. The second set of filters may, for example, be signaled filters, meaning that the filters in the second set are signaled, at least in part, within the video data.

[0225] Video decoder 300 determines filtered reconstructed samples based on the reconstructed samples, the first sample modification value, and the second sample modification value (730). To determine filtered reconstructed samples based on the reconstructed samples, the first sample modification value, and the second sample modification value, video decoder 300 may, for example, assign the first sample modification value to the reconstructed sample. and the second sample modification value may be added. Equation (28) above represents one example of how video decoder 300 may determine a value for the filtered reconstructed samples.

[0226] To determine a second sample modification value based on the first intermediate sample value, video decoder 300 clips the second sample modification value to determine a clipped sample modification value. The first sample modification value and the clipped sample modification value may then be added to the reconstructed sample. Video decoder 300 may receive clipped sample modifications within video data. To determine a second sample modification value based on the first intermediate sample value, video decoder 300 operates by determining a difference between the reconstructed sample and the first intermediate sample value. The second sample modification value may also be determined based on the first intermediate sample value. Equations (28) and (30) above represent examples of how video decoder 300 may determine the second sample modification value.

[0227] Video decoder 300 may output decoded pictures of video data that include filtered reconstructed samples. In some coding scenarios, video decoder 300 may perform one or more additional filtering operations on the filtered reconstructed samples before output. Video decoder 300 may be configured to operate by storing pictures in a decoded picture buffer for use in decoding subsequent pictures, by storing decoded pictures on a storage medium for later display, or by storing decoded pictures in a storage medium for later display, or in real-time or The decoded pictures may be output by outputting the decoded pictures to a display device for near real-time display. In an example where the technique of FIG. 20 is performed by a video encoder, the video encoder may decode the picture by storing the picture in a decoded picture buffer, for example, for use in encoding subsequent pictures. Pictures can be output.

[0228] The numbered sections below indicate one or more aspects of the devices and techniques described in this disclosure.

[0229] Clause 1A. A method of decoding video data, the method comprising: determining a reconstructed block of video data; and applying a filter to a sample of the reconstructed block of video data to generate a filtered reconstructed block. Prepare for things to do.

[0230] Clause 2A. Applying the filter to the sample includes determining an activity value for the sample and/or a direction for the sample, and selecting a filter from the set of filters based on the determined activity value and/or direction. The method described in Clause 1A, comprising:

[0231] Clause 3A. The method according to clause 2A, wherein the set of filters comprises a fixed set of filters.

[0232] Clause 4A. 2A. The method of clause 2A, wherein the set of filters comprises a signaled set of filters.

[0233] Clause 5A. Applying the filter to the sample includes applying a first filter to the sample to determine an intermediate filtered sample and applying a second filter to the intermediate filtered sample to determine a sampled final filter. The method according to any of clauses 1A to 4A, comprising applying a filter of.

[0234] Clause 6A. A method according to any of clauses 1A to 5A, wherein the method of decoding is performed as part of a video coding process.

[0235] Clause 7A. A device for coding video data, the device comprising one or more means for performing the method according to any of clauses 1A to 6A.

[0236] Clause 8A. 7A. The device according to clause 7A, wherein the one or more means comprises one or more processors implemented in circuitry.

[0237] Clause 9A. The device according to any of clauses 7A and 8A, further comprising a memory for storing video data.

[0238] Clause 10A. 9A. The device according to any of clauses 7A-9A, further comprising a display configured to display decoded video data.

[0239] Clause 11A. A device according to any of clauses 7A to 10A, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0240] Clause 12A. The device according to any of clauses 7A-11A, wherein the device comprises a video decoder.

[0241] Clause 13A. The device according to any of clauses 7A-12A, wherein the device comprises a video encoder.

[0242] Clause 14A. A computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to perform the method according to any of clauses 1A-6A.

[0243] Clause 1B. A method of decoding video data, the method comprising: decoding at least some of the pictures of the video data; and generating filtered samples for at least some of the pictures. applying one or more adaptive loop filters (ALF) to at least a portion of the pictures; and determining whether to perform a second stage of the ALF on at least a portion of the pictures. and when the decision is not to perform the second stage of ALF, avoid decoding the filter coefficients for one or more ALFs for the second stage of ALF, and for at least a portion of the pictures. The method comprises avoiding performing a second stage of ALF and using the filtered samples to generate final samples for at least a portion of the picture.

[0244] Clause 2B. Using the filtered samples to generate the final samples

and calculating the final sample according to, where:

represents one of the final samples at position (x, y), w _i comprises the weight value for the i-th ALF of the ALF or ALFs, and R'(x, y , i) comprises one of the filtered samples generated by the i-th ALF at position (x,y).

[0245] Clause 3B. The method of clause 2B, further comprising determining the weight values w _i from fixed configuration data.

[0246] Clause 4B. 2B. The method of clause 2B, further comprising determining a weight value w _i from the decoded filter usage information.

[0247] Clause 5B. The method of clause 2B, further comprising decoding the weight values w _i .

[0248] Clause 6B. Decoding the weight values w _i may be a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a subpicture header, a coding tree unit (CTU) header, a coding tree block (CTB) header, or a subblock. 5B. The method according to clause 5B, comprising decoding the weight values w _i from one or more of the headers.

[0249] Clause 7B. The method according to any of clauses 2B-6B, further comprising decoding data for syntax elements representing combinations of weight values w _i used for one or more of the final samples.

[0250] Clause 8B. 7B. The method of any of clauses 1B-7B, wherein determining whether to perform the second stage of ALF comprises decoding data representative of whether to perform the second stage of ALF.

[0251] Clause 9B. Decoding data representing whether to perform the second stage of ALF includes sequence parameter set (SPS), picture parameter set (PPS), slice header, sub-picture header, coding tree unit (CTU) header, coding 8B. The method of clause 8B, comprising decoding data of one or more of a tree block (CTB) header or a sub-block header.

[0252] Clause 10B. decoding data representing whether to perform the second stage of ALF comprises decoding a value for the syntax element, the value indicating whether the second stage of ALF is to be bypassed; The method according to any of clauses 8B and 9B, wherein the method is:

[0253] Clause 11B. Clause 1B~, wherein determining whether to perform the second stage of ALF comprises adaptively deriving whether to perform the second stage of ALF according to previously decoded coding information. 7B.

[0254] Clause 12B. Previously decoded coding information may include the quantization parameter (QP) for a block in at least a portion of a picture, the size of the block, or whether the block is predicted using intra prediction or inter prediction. The method of clause 11B, comprising one or more of:

[0255] Clause 13B. The method according to any of clauses 1B-12B, wherein using the filtered samples to generate the final samples comprises scaling the filtered samples to generate the final samples. .

[0256] Clause 14B. The method according to any of clauses 1B-13B, further comprising encoding at least a portion of the picture before decoding at least a portion of the picture.

[0257] Clause 15B. A device for decoding video data, the device comprising one or more means for performing the method according to any of clauses 1B to 14B.

[0258] Clause 16B. 15B. The device according to clause 15B, wherein the one or more means comprises one or more processors implemented in circuitry.

[0259] Clause 17B. 15. The device of any of clauses 15B and 16B, further comprising a display configured to display decoded video data.

[0260] Clause 18B. The device according to any of clauses 15B-17B, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0261] Clause 19B. The device according to clauses 15B-18B, further comprising a memory configured to store video data.

[0262] Clause 20B. A computer-readable storage medium having instructions stored thereon that, when executed, cause a processor of a device for decoding video data to perform the method according to any of clauses 1B to 14B.

[0263] Clause 21B. A device for decoding video data, the device comprising means for decoding at least a portion of a picture of the video data and an ALF for generating filtered samples for at least a portion of the picture. means for applying one or more adaptive loop filters (ALF) to at least a portion of the picture during the first stage; and performing a second stage of the ALF on at least a portion of the picture. means for determining whether and to avoid decoding filter coefficients for one or more ALFs for the second stage of the ALF when the decision is not to perform the second stage of the ALF; means for avoiding performing the second stage of ALF on at least a portion of the picture when the decision is not to perform the second stage of ALF; and at least a portion of the picture. and means for using the filtered samples to generate final samples for.

[0264] Clause 1C. A method of decoding video data, the method comprising: applying a first stage adaptive loop filter (ALF) to reconstructed samples of a reconstruction block; determining a first class index for the reconstructed sample; selecting a filter from a first set of filters based on the first class index; and determining a first intermediate sample value. applying a filter from the first set of filters to the reconstructed sample to determine, applying a second stage ALF to the reconstructed sample; Applying the stage ALF includes determining a second class index for the reconstructed sample and selecting a second filter from the second set of filters based on the second class index. , applying a second filter to the reconstructed sample to determine a first sample modification value, and determining a second sample modification value based on the first intermediate sample value. and determining filtered reconstructed samples based on the reconstructed samples, the first sample modification values, and the second sample modification values.

[0265] Clause 2C. The method of clause 1C, wherein the first set of filters comprises fixed filters.

[0266] Clause 3C. A method according to any of clauses 1C-2C, wherein the second set of filters comprises signaled filters determined based on syntax signaled in the video data.

[0267] Clause 4C. Determining a first class index includes determining an activity value for the reconstructed sample, determining a direction for the reconstructed sample, and determining a first class index based on the activity value and the direction. and determining the index.

[0268] Clause 5C. 4C. The method of clause 4C, wherein determining the direction comprises assigning one of 56 values to the direction.

[0269] Clause 6C. The method of any of clauses 1C-5C, wherein the first set of filters comprises 9x9 diamond shaped filters.

[0270] Clause 7C. determining a filtered reconstruction sample based on the reconstructed sample, the first sample modification value, and the second sample modification value; 6C.

[0271] Clause 8C. Determining a second sample modification value based on the first intermediate sample value includes: clipping the second sample modification value to determine a clipped sample modification value; , the first sample modification value and the clipped sample modification value.

[0272] Clause 9C. 8C. The method of clause 8C, further comprising receiving a clipped sample modification value within the video data.

[0273] Clause 10C. Any of clauses 1C-9C, wherein determining the second sample modification value based on the first intermediate sample value comprises determining a difference between the reconstructed sample and the first intermediate sample value. Method described in Crab.

[0274] Clause 11C. 10C. The method of any of clauses 1C-10C, further comprising selecting the first set of filters from the plurality of fixed sets of filters based on a quantization parameter for the reconstruction block.

[0275] Clause 12C. Applying the first stage ALF includes determining a third class index for the reconstructed samples and determining a third class index from the second set of filters based on the third class index. selecting a third filter from the second set of filters and applying a third filter from the second set of filters to the reconstructed sample to determine a second intermediate sample value; and applying the second stage ALF to the reconstructed sample determines a second sample modification value based on the first intermediate sample value and the second intermediate sample value. The method according to any of clauses 1C to 11C, further comprising determining.

[0276] Clause 13C. 12C. The method of any of clauses 1C-12C, further comprising adding a predicted sample value to the residual sample value to determine the reconstructed sample.

[0277] Clause 14C. 13. The method of any of clause 13C, further comprising applying a deblocking filter to a sum of predicted and residual sample values to determine the reconstructed samples.

[0278] Clause 15C. 14. The method according to any of clauses 1C to 14C, further comprising outputting a decoded picture of video data, the decoded picture comprising filtered reconstructed samples.

[0279] Clause 16C. A method according to any of clauses 1C to 14C, wherein the method of decoding is performed as part of a video encoding process.

[0280] Clause 17C. A device for decoding video data, the device comprising: a memory configured to store video data; and a circuit implemented with a first stage adaptive loop filter on the reconstructed samples of the reconstruction block. (ALF), wherein to apply the first stage ALF, the one or more processors determine a first class index for the reconstructed sample; selecting a filter from the first set of filters based on a class index of the filter and applying the filter from the first set of filters to the reconstructed sample to determine a first intermediate sample value; and applying a second stage ALF to the reconstructed sample, the one or more processors further configured to: apply a second stage ALF to the reconstructed sample; determining a second class index for the reconstructed sample; selecting a second filter from the second set of filters based on the second class index; and determining a first sample modification value. and further configured to apply a second filter to the reconstructed sample and determine a second sample change value based on the first intermediate sample value to perform the reconstruction. one or more processors configured to: determine a filtered reconstructed sample based on the sample, the first sample modification value, and the second sample modification value; .

[0281] Clause 18C. 17C. The apparatus according to clause 17C, wherein the first set of filters comprises fixed filters.

[0282] Clause 19C. 18C. The device according to any of clauses 17C-18C, wherein the second set of filters comprises signaled filters determined based on syntax signaled in the video data.

[0283] Clause 20C. To determine the first class index, the one or more processors determine an activity value for the reconstructed sample, a direction for the reconstructed sample, an activity value, and a direction. 19C. The device according to any of clauses 17C-19C, further configured to: determine a first class index based on.

[0284] Clause 21C. 20C. The device of clause 20C, wherein the one or more processors are further configured to assign one of 56 values to the direction to determine the direction.

[0285] Clause 22C. Apparatus according to any of clauses 17C-21C, wherein the first set of filters comprises 9x9 diamond-shaped filters.

[0286] Clause 23C. The one or more processors apply a first filter to the reconstructed sample to determine a filtered reconstructed sample based on the reconstructed sample, the first sample modification value, and the second sample modification value. 22. The device according to any of clauses 17C-22C, further configured to add one sample change value and a second sample change value.

[0287] Clause 24C. one or more processors to determine a second sample modification value based on the first intermediate sample value; clipping the second sample modification value to determine a clipped sample modification value; 23. The device of any of clauses 17C-23C, further configured to: and add the first sample modification value and the clipped sample modification value to the reconstructed sample.

[0288] Clause 25C. 24C. The device of clause 24C, wherein the one or more processors are further configured to receive clipped sample modifications within the video data.

[0289] Clause 26C. one or more processors for determining a difference between the reconstructed sample and the first intermediate sample value to determine a second sample modification value based on the first intermediate sample value; A device according to any of clauses 17C to 25C, further comprising:

[0290] Clause 27C. of clauses 17-26C, wherein the one or more processors are further configured to select the first set of filters from the plurality of fixed filter sets based on a quantization parameter for the reconstruction block. Any device listed.

[0291] Clause 28C. To apply the first stage ALF, one or more processors determine a third class index for the reconstructed samples and a second class index of the filter based on the third class index. selecting a third filter from the set; and applying a third filter from the second set of filters to the reconstructed sample to determine a second intermediate sample value. the one or more processors are further configured to apply a second stage ALF to the reconstructed samples; The device according to any of clauses 17C-27C, further configured to determine a sample change value of .

[0292] Clause 29C. 28. The device of any of clauses 17C-28C, wherein the one or more processors are further configured to add predicted sample values to residual sample values to determine reconstructed samples.

[0293] Clause 30C. 28C. The device of clause 28C, wherein the one or more processors are further configured to apply a deblocking filter to the sum of the predicted sample value and the residual sample value to determine the reconstructed sample.

[0294] Clause 31C. The device according to any of clauses 17C to 30C, wherein the one or more processors are further configured to output decoded pictures of video data, the decoded pictures comprising filtered reconstructed samples. .

[0295] Clause 32C. A device according to any of clauses 17C to 31C, wherein the device comprises a video encoding device.

[0296] Clause 33C. The device according to any of clauses 17C to 32C, wherein the one or more processors are further configured to output decoded pictures of video data, the decoded pictures comprising filtered reconstructed samples. .

[0297] Clause 34C. 33. The device of clause 33C, wherein the device comprises a wireless communication device, further comprising a receiver configured to receive encoded video data.

[0298] Clause 35C. The device according to clause 34C, wherein the wireless communication device comprises a telephone handset, wherein the receiver is configured to demodulate the signal comprising encoded video data in accordance with a wireless communication standard. .

[0299] Clause 36C. A device according to any of clauses 17C to 35C, further comprising a display configured to display decoded video data.

[0300] Clause 37C. A device according to any of clauses 17C to 36C, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0301] Clause 38C. A device according to any of clauses 17C to 37C, wherein the device comprises a video encoding device.

[0302] Clause 39C. when executed by the one or more processors, applying a first stage adaptive loop filter (ALF) to the reconstructed samples of the reconstruction block; To apply the first stage ALF, one or more processors determine a first class index for the reconstructed samples and, based on the first class index, a first class index of the filter. further configured to select a filter from the set and apply the filter from the first set of filters to the reconstructed sample to determine a first intermediate sample value; applying a second stage ALF to the sample; and wherein the one or more processors determine a second class index for the reconstructed sample to apply the second stage ALF. selecting a second filter from a second set of filters based on a second class index; and applying a second filter to the reconstructed sample to determine a first sample modification value. further configured to apply a filter and determine a second sample modification value based on the first intermediate sample value; and determining a filtered reconstructed sample based on a sample modification value of 2.

[0303] Clause 1D. A method of decoding video data, the method comprising: applying a first stage adaptive loop filter (ALF) to reconstructed samples of a reconstruction block; determining a first class index for the reconstructed sample; selecting a filter from a first set of filters based on the first class index; and determining a first intermediate sample value. applying a filter from the first set of filters to the reconstructed sample to determine, applying a second stage ALF to the reconstructed sample; Applying the stage ALF includes determining a second class index for the reconstructed sample and selecting a second filter from the second set of filters based on the second class index. , applying a second filter to the reconstructed sample to determine a first sample modification value, and determining a second sample modification value based on the first intermediate sample value. and determining filtered reconstructed samples based on the reconstructed samples, the first sample modification values, and the second sample modification values.

[0304] Clause 2D. The method of clause ID, wherein the first set of filters comprises fixed filters.

[0305] Clause 3D. The method of clause ID, wherein the second set of filters comprises signaled filters determined based on syntax signaled within the video data.

[0306] Clause 4D. Determining a first class index includes determining an activity value for the reconstructed sample, determining a direction for the reconstructed sample, and determining a first class index based on the activity value and the direction. and determining an index.

[0307] Clause 5D. 4D. The method of clause 4D, wherein determining the direction comprises assigning one of 56 values to the direction.

[0308] Clause 6D. The method of clause ID, wherein the first set of filters includes 9x9 diamond shaped filters.

[0309] Clause 7D. determining a filtered reconstruction sample based on the reconstructed sample, the first sample modification value, and the second sample modification value; 2. The method according to clause 1D, comprising summing the sample change values of two samples.

[0310] Clause 8D. Determining a second sample modification value based on the first intermediate sample value includes: clipping the second sample modification value to determine a clipped sample modification value; , the first sample modification value and the clipped sample modification value.

[0311] Clause 9D. 8D. The method of clause 8D, further comprising receiving clipped sample modification values within the video data.

[0312] Clause 10D. The method of clause 1D, wherein determining the second sample modification value based on the first intermediate sample value comprises determining a difference between the reconstructed sample and the first intermediate sample value. .

[0313] Clause 11D. The method of clause ID, further comprising selecting the first set of filters from the plurality of fixed sets of filters based on a quantization parameter for the reconstruction block.

[0314] Clause 12D. applying the first stage ALF includes determining a third class index for the reconstructed sample; and selecting a second filter from the second set of filters based on the third class index. and applying a second filter from the second set of filters to the reconstructed sample to determine a second intermediate sample value; The method of clause ID, wherein applying the stage ALF further comprises determining a second sample change value based on the first intermediate sample value and the second intermediate sample value.

[0315] Clause 13D. The method of clause ID further comprising adding predicted sample values to residual sample values to determine reconstructed samples.

[0316] Clause 14D. 13. The method of clause 13D, further comprising applying a deblocking filter to the sum of the predicted sample values and the residual sample values to determine the reconstructed samples.

[0317] Clause 15D. The method of clause ID, further comprising outputting a decoded picture of video data, wherein the decoded picture comprises filtered reconstructed samples.

[0318] Clause 16D. The method according to clause ID, wherein the method of decoding is performed as part of a video encoding process.

[0319] Clause 17D. A device for decoding video data, the device comprising: a memory configured to store video data; and a circuit implemented with a first stage adaptive loop filter on the reconstructed samples of the reconstruction block. (ALF), wherein to apply the first stage ALF, the one or more processors determine a first class index for the reconstructed sample; selecting a filter from the first set of filters based on a class index of the filter and applying the filter from the first set of filters to the reconstructed sample to determine a first intermediate sample value; and applying a second stage ALF to the reconstructed sample, the one or more processors further configured to: apply a second stage ALF to the reconstructed sample; determining a second class index for the reconstructed sample; selecting a second filter from the second set of filters based on the second class index; and determining a first sample modification value. and further configured to apply a second filter to the reconstructed sample and determine a second sample change value based on the first intermediate sample value to perform the reconstruction. one or more processors configured to: determine a filtered reconstructed sample based on the sample, the first sample modification value, and the second sample modification value; .

[0320] Clause 18D. 17D. The apparatus according to clause 17D, wherein the first set of filters comprises fixed filters.

[0321] Clause 19D. 17D. The device of clause 17D, wherein the second set of filters comprises signaled filters determined based on syntax signaled within the video data.

[0322] Clause 20D. To determine the first class index, the one or more processors determine an activity value for the reconstructed sample, a direction for the reconstructed sample, an activity value, and a direction. The device of clause 17D, further configured to: determine the first class index based on the first class index.

[0323] Clause 21D. 20D. The device of clause 20D, wherein the one or more processors are further configured to assign one of 56 values to the direction to determine the direction.

[0324] Clause 22D. The device of clause 17D, wherein the first set of filters includes 9x9 diamond shaped filters.

[0325] Clause 23D. The one or more processors apply a first filter to the reconstructed sample to determine a filtered reconstructed sample based on the reconstructed sample, the first sample modification value, and the second sample modification value. 17D. The device of clause 17D, further configured to add one sample modification value and a second sample modification value.

[0326] Clause 24D. one or more processors to determine a second sample modification value based on the first intermediate sample value; clipping the second sample modification value to determine a clipped sample modification value; and adding the first sample modification value and the clipped sample modification value to the reconstructed sample.

[0327] Clause 25D. 24D. The device of clause 24D, wherein the one or more processors are further configured to receive clipped sample modifications within the video data.

[0328] Clause 26D. one or more processors for determining a difference between the reconstructed sample and the first intermediate sample value to determine a second sample modification value based on the first intermediate sample value; 17D, further comprising:

[0329] Clause 27D. 17D, wherein the one or more processors are further configured to select the first set of filters from the plurality of fixed filter sets based on a quantization parameter for the reconstruction block. device.

[0330] Clause 28D. To apply the first stage ALF, one or more processors determine a third class index for the reconstructed samples and a second class index of the filter based on the third class index. selecting a second filter from the set; and applying a second filter from the second set of filters to the reconstructed sample to determine a second intermediate sample value. the one or more processors are further configured to apply a second stage ALF to the reconstructed samples; The device of clause 17D, further configured to determine a sample change value of .

[0331] Clause 29D. 17D. The device of clause 17D, wherein the one or more processors are further configured to add predicted sample values to residual sample values to determine reconstructed samples.

[0332] Clause 30D. 28D. The device of clause 28D, wherein the one or more processors are further configured to apply a deblocking filter to the sum of the predicted sample value and the residual sample value to determine the reconstructed sample.

[0333] Clause 31D. 17D. The device of clause 17D, wherein the one or more processors are further configured to output decoded pictures of video data, where the decoded pictures comprise filtered reconstructed samples.

[0334] Clause 32D. The device according to clause 17D, wherein the device comprises a video encoding device.

[0335] Clause 33D. 17D. The device of clause 17D, wherein the one or more processors are further configured to output decoded pictures of video data, where the decoded pictures comprise filtered reconstructed samples.

[0336] Clause 34D. 33. The device of clause 33D, wherein the device comprises a wireless communication device further comprising a receiver configured to receive encoded video data.

[0337] Clause 35D. 34. The device of clause 34D, wherein the wireless communication device comprises a telephone handset, wherein the receiver is configured to demodulate the signal comprising encoded video data in accordance with a wireless communication standard.

[0338] Clause 36D. 17D. The device of clause 17D, further comprising a display configured to display decoded video data.

[0339] Clause 37D. The device according to clause 17D, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0340] Clause 38D. The device according to clause 17D, wherein the device comprises a video encoding device.

[0341] Clause 39D. when executed by the one or more processors, applying a first stage adaptive loop filter (ALF) to the reconstructed samples of the reconstruction block; To apply the first stage ALF, one or more processors determine a first class index for the reconstructed samples and, based on the first class index, a first class index of the filter. further configured to select a filter from the set and apply the filter from the first set of filters to the reconstructed sample to determine a first intermediate sample value; applying a second stage ALF to the sample; and wherein the one or more processors determine a second class index for the reconstructed sample to apply the second stage ALF. selecting a second filter from a second set of filters based on a second class index; and applying a second filter to the reconstructed sample to determine a first sample modification value. further configured to apply a filter and determine a second sample modification value based on the first intermediate sample value; and determining a filtered reconstructed sample based on a sample modification value of 2.

[0342] Depending on the example above, some acts or events of any of the techniques described herein may be performed in different sequences, added, merged, or removed entirely (e.g., all It is recognized that the acts or events described may not be necessary for the practice of the present techniques). Moreover, in some examples, actions or events may be performed concurrently rather than sequentially, eg, through multi-threaded processing, interrupt processing, or multiple processors.

[0343] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. . Computer-readable media refers to a computer-readable storage medium that corresponds to a tangible medium, such as a data storage medium or a communication protocol, including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. may include a medium. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media that is non-transitory, or (2) a communication medium such as a signal or carrier wave. The data storage medium can be any data storage medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. It can be any available media. A computer program product may include a computer readable medium.

[0344] By way of example and not limitation, such computer readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, etc. , or may comprise any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, instructions may be transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave. where coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other transitory media, but instead refer to non-transitory tangible storage media. Disk and disc as used herein refer to compact disc (CD), laserdisc (disc), optical disc (disc), digital versatile disc (disc) ( (DVD), floppy disk (disk), and Blu-ray disk (disc), where a disk (disk) typically reproduces data magnetically, and a disk (disc) (disc) Regenerate optically using a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0345] The instructions may be executed by one or more processors, such as one or more DSPs, general purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Accordingly, the terms "processor" and "processing circuitry" as used herein may refer to any of the structures described above, or any other structure suitable for implementing the techniques described herein. . Additionally, in some aspects, the functionality described herein may be provided in dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a composite codec. . Additionally, the techniques may be fully implemented in one or more circuits or logic elements.

[0346] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs) or sets of ICs (eg, chipsets). Although various components, modules, or units have been described in this disclosure to highlight functional aspects of devices configured to implement the disclosed techniques, those components, modules, or Units do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined or interacted with in a codec hardware unit, including one or more processors as described above, along with suitable software and/or firmware. It may be provided by a collection of operable hardware units.

[0347] Various examples have been described. These and other examples are within the scope of the following claims.

Claims (38)

A method of decoding video data, the method comprising:
applying a first stage adaptive loop filter (ALF) to the reconstructed samples of the reconstructed block;
determining a first class index for the reconstructed sample;
selecting a filter from a first set of filters based on the first class index;
applying the filter from the first set of filters to the reconstructed sample to determine a first intermediate sample value;
applying a second stage ALF to the reconstructed sample;
determining a second class index for the reconstructed sample;
selecting a second filter from a second set of filters based on the second class index;
applying the second filter to the reconstructed sample to determine a first sample modification value;
determining a second sample change value based on the first intermediate sample value;
determining filtered reconstructed samples based on the reconstructed samples, the first sample modification values, and the second sample modification values. 2. The method of claim 1, wherein the first set of filters comprises fixed filters. 2. The method of claim 1, wherein the second set of filters comprises signaled filters determined based on syntax signaled within the video data. Determining the first class index comprises:
determining an activity value for the reconstructed sample;
determining an orientation for the reconstructed sample;
2. The method of claim 1, comprising: determining the first class index based on the activity value and the direction. 5. The method of claim 4, wherein determining the direction comprises assigning one of 56 values to the direction. 2. The method of claim 1, wherein the first set of filters includes 9x9 diamond shaped filters. determining the filtered reconstructed samples based on the reconstructed samples, the first sample modification values, and the second sample modification values; 2. The method of claim 1, comprising adding a sample modification value and the second sample modification value. determining the second sample modification value based on the first intermediate sample value, clipping the second sample modification value to determine a clipped sample modification value; 2. The method of claim 1, comprising adding the first sample modification value and the clipped sample modification value to a reconstructed sample. 9. The method of claim 8, further comprising: receiving the clipped sample modification value within the video data. 12. Determining the second sample modification value based on the first intermediate sample value comprises determining a difference between the reconstructed sample and the first intermediate sample value. The method described in 1. 2. The method of claim 1, further comprising: selecting the first set of filters from a plurality of fixed sets of filters based on a quantization parameter for the reconstruction block. Applying the first stage ALF includes determining a third class index for the reconstructed sample and extracting a third class index from a second set of filters based on the third class index. further comprising selecting a filter and applying the third filter from the second set of filters to the reconstructed sample to determine a second intermediate sample value;
Applying the second stage ALF to the reconstructed samples determines the second sample modification value based on the first intermediate sample value and the second intermediate sample value. further prepare for
The method according to claim 1. The method of claim 1, further comprising: adding predicted sample values to residual sample values to determine the reconstructed samples. 14. The method of claim 13, further comprising: applying a deblocking filter to the sum of the predicted sample values and the residual sample values to determine the reconstructed samples. outputting a decoded picture of the video data, further comprising: the decoded picture comprising the filtered reconstructed samples;
The method according to claim 1. The method of claim 1, wherein the method of decoding is performed as part of a video encoding process. A device for decoding video data, the device comprising:
a memory configured to store video data;
carried out in the circuit,
applying a first stage adaptive loop filter (ALF) to the reconstructed samples of the reconstruction block;
determining a first class index for the reconstructed sample;
selecting a filter from a first set of filters based on the first class index;
applying the filter from the first set of filters to the reconstructed sample to determine a first intermediate sample value; and
applying a second stage ALF to the reconstructed samples;
determining a second class index for the reconstructed sample;
selecting a second filter from a second set of filters based on the second class index;
applying the second filter to the reconstructed sample to determine a first sample modification value;
further configured to: determine a second sample modification value based on the first intermediate sample value;
one or more configured to: determine filtered reconstructed samples based on the reconstructed samples, the first sample modification values, and the second sample modification values; A device comprising: a processor; 18. The device of claim 17, wherein the first set of filters comprises fixed filters. 18. The device of claim 17, wherein the second set of filters comprises signaled filters determined based on syntax signaled within the video data. In order to determine the first class index, the one or more processors:
determining an activity value for the reconstructed sample;
determining an orientation for the reconstructed sample;
18. The device of claim 17, further configured to: determine the first class index based on the activity value and the direction. 21. The device of claim 20, wherein to determine the direction, the one or more processors are further configured to assign one of 56 values to the direction. 18. The device of claim 17, wherein the first set of filters includes 9x9 diamond shaped filters. the one or more processors to determine the filtered reconstructed samples based on the reconstructed samples, the first sample modification values, and the second sample modification values; 18. The device of claim 17, further configured to add the first sample modification value and the second sample modification value to reconstructed samples. the one or more processors to determine a clipped sample modification value based on the first intermediate sample value; 18. The method of claim 17, further configured to clip a modified value and add the first sample modified value and the clipped sample modified value to the reconstructed sample. device. the one or more processors,
25. The device of claim 24, further configured to receive the clipped sample modification value within the video data. to determine the second sample modification value based on the first intermediate sample value, the one or more processors determine the difference between the reconstructed sample and the first intermediate sample value; 18. The device of claim 17, further configured to determine. the one or more processors,
18. The device of claim 17, further configured to: select the first set of filters from a plurality of fixed sets of filters based on a quantization parameter for the reconstruction block. In order to apply the first stage ALF, the one or more processors may determine a third class index for the reconstructed samples and apply a filter based on the third class index. selecting a third filter from the second set of filters and applying the third filter from the second set of filters to the reconstructed samples to determine a second intermediate sample value; further configured to do and to do;
In order to apply the second stage ALF to the reconstructed samples, the one or more processors may perform the process based on the first intermediate sample values and the second intermediate sample values. further configured to determine a second sample change value;
18. A device according to claim 17. the one or more processors,
18. The device of claim 17, further configured to add predicted sample values to residual sample values to determine the reconstructed samples. the one or more processors,
30. The device of claim 29, further configured to apply a deblocking filter to the sum of the predicted sample value and the residual sample value to determine the reconstructed sample. the one or more processors,
18. The device of claim 17, further configured to output a decoded picture of the video data, wherein the decoded picture comprises the filtered reconstructed samples. 18. The device of claim 17, wherein the device comprises a video encoding device. 33. The device of claim 32, wherein the device comprises a wireless communication device further comprising a receiver configured to receive encoded video data. 34. The device of claim 33, wherein the wireless communication device comprises a telephone handset, wherein the receiver is configured to demodulate the signal comprising the encoded video data in accordance with a wireless communication standard. 18. The device of claim 17, further comprising a display configured to display decoded video data. 18. The device of claim 17, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box. 18. The device of claim 17, wherein the device comprises a video encoding device. when executed by one or more processors, the one or more processors:
applying a first stage adaptive loop filter (ALF) to the reconstructed samples of the reconstruction block; ,
determining a first class index for the reconstructed sample;
selecting a filter from a first set of filters based on the first class index;
applying the filter from the first set of filters to the reconstructed sample to determine a first intermediate sample value; and
applying a second stage ALF to the reconstructed samples;
determining a second class index for the reconstructed sample;
selecting a second filter from a second set of filters based on the second class index;
applying the second filter to the reconstructed sample to determine a first sample modification value;
further configured to: determine a second sample modification value based on the first intermediate sample value;
a computer-readable storage medium storing instructions for: determining a filtered reconstructed sample based on the reconstructed sample, the first sample modification value, and the second sample modification value; .

Images