RU2789522C2 - Methods for intermediate prediction based on pattern, based on reduction in encoding and decoding delay - Google Patents

Abstract

FIELD: video encoding/decoding.

SUBSTANCE: invention relates to the field of a video signal encoding and decoding. Methods for video encoding and decoding for reduction in delay in intermediate encoding based on a pattern are proposed. Methods include the current image and at least one reference image. For at least the current block in the current image, a corresponding predicted value is generated (for example, using a prediction with motion compensation) for each discrete count in an area of the pattern, adjacent to the current block. When predicted values are generated for each discrete count in the area of the pattern, a process is activated to determine a parameter of an intermediate prediction based on the pattern by means of predicted values in the area of the pattern and values of discrete counts in the reference image. This process can be activated without waiting for restored values of discrete counts in the area of the pattern. The intermediate prediction based on the pattern of the current block is then performed using the determined parameter of the intermediate prediction based on the pattern.

EFFECT: increase in the efficiency of encoding.

38 cl, 18 dwg, 1 tbl

Description

BACKGROUND OF THE INVENTION

[0001] Video coding systems are widely used to compress digital video signals to reduce the need to store and/or the transmission bandwidth of such signals. Among various types of video coding systems such as block-based, burst-based, and object-based systems, hybrid block-based video coding systems are now the most widely used and installed. Examples of block-based video coding systems include international video coding standards such as MPEG1/2/4 Part 2; H.264/MPEG-4 Part 10 AVC; VC-1; and the latest video coding standard called High Efficiency Video Coding (HEVC), which was developed by JCT-VC (Joint Collaborative Video Coding Team) ITU-T/SG16/Q.6/VCEG and ISO/IEC/MPEG.

[0002] The first version of the HEVC standard was approved in January 2013 and offers approximately 50% bit rate savings at equivalent perceived quality compared to the previous generation of the H.264/MPEG AVC video coding standard. While the HEVC standard provides significant coding improvements over past standards, there is evidence that higher coding efficiency can be achieved with additional coding tools over HEVC. Based on this, both VCEG and MPEG started research work for new coding technologies for future video coding standardization. In October 2015, the ITU-T VCEG and ISO/IEC MPEG formed the Joint Video Exploration (JVET) team to begin a substantial study of improved technologies that could enable significant improvements in coding efficiency over HEVC. In the same month, a software codebase called the Joint Exploration Model (JEM) was established for future video coding research work. The JEM Reference Software was based on the HEVC Test Model (HM) which was developed by JCT-VC for HEVC. Additional proposed coding tools can be integrated into the JEM software and tested using the JVET Common Test Conditions (CTC).

SUMMARY OF THE INVENTION

[0003] Exemplary embodiments include techniques that are used in encoding and decoding (collectively "coding") video. In some embodiments, a method is provided for encoding or decoding a video that includes a current picture and at least a first reference picture. For at least the current block in the current picture, a predicted value is generated (eg, using motion compensated prediction) for each discrete sample in the pattern region adjacent to the current block. At least one inter template prediction parameter is determined by comparing predicted values of at least a subset of samples in the template region with corresponding sample values (which may be reconstructed sample values) in at least one region of the reference template in at least the first reference image. Intermediate prediction based on the template of the current block is performed using a specific intermediate prediction parameter based on the template.

[0004] In some embodiments, a process for determining a template-based intermediate prediction parameter is activated in response to generating predicted values for each sample in the template region. This allows the prediction of sample values in the current block to continue in parallel with the recovery of sample values in the template region.

[0005] In some embodiments, at least one of the discrete samples in the pattern region has an encoded non-zero remainder. In some embodiments, after a template-based intermediate prediction parameter is determined, an encoded non-zero residual value is added to at least one of the predicted values in the template region to generate a corresponding recovered sample value.

[0006] In some embodiments, a subset of samples in the template region is selected to include only samples that have zero residuals. Coded block flag values for respective blocks containing discrete samples may be used to identify those discrete samples that have zero residuals. In some embodiments, a template region has at least two sub-regions (e.g., a first sub-sample sub-region above the current block and a second sub-sample sub-region to the left of the current block), and samples in the sub-region are selected only if all the samples in that sub-region are zero. residual values.

[0007] In some embodiments, overlay block motion compensation (OBMC) is performed on at least some of the discrete samples in the template region after the template-based intermediate prediction parameter is determined.

[0008] In some embodiments, the samples used to determine the template-based intermediate prediction parameter are weighted based on the estimated (or actual) absolute values of the residuals of the corresponding samples in the template region, where larger weights are used for lower estimated (or actual) absolute values of residuals. Estimated (or actual) absolute residual values may be based on information obtained without performing dequantization or inverse transformation. In some embodiments, the absolute value of the remainder of a discrete sample is determined based on the value of the encoded block flag of the block containing the corresponding discrete sample. In some embodiments, the absolute value of the remainder of the discrete sample is estimated based on the total energy of the transform coefficients for the block containing the corresponding discrete sample. In some embodiments, a first predefined lower weight is used for discrete samples with non-zero estimated absolute residual values, and a second predefined higher weight is used for discrete samples with absolute residual values that are estimated (or known) to be zero.

[0009] In some embodiments, instead of using only the predicted values in the template region to determine the intermediate prediction parameter based on the template, at least some of the discrete sample values in the template region are partially recovered for use in determining the parameter. For example, the values of the discrete samples can be partially restored by adding the residual component of the DC prediction to the predicted values. The DC component can be obtained without performing an inverse transformation. In some embodiments, this partial reconstruction is performed only for the values of discrete samples in blocks encoded using either DCT-II or DCT-V. In some embodiments, partial recovery of sample values is performed only for those sample values in blocks that are not encoded using NSST.

[0010] In some embodiments, the implementation of the intermediate prediction based on the template of the current block is performed using local illumination compensation (LIC). In such embodiments, the template-based intermediate prediction parameters include at least one scaling factor b and at least one offset c.

[0011] In some embodiments, the implementation of the intermediate prediction based on the template of the current block is performed using frame rate upconverting (FRUC) based on the template. In such embodiments, the template-based intermediate prediction parameters include a motion vector for predicting the current block. In some embodiments, both LIC and FRUC are used to predict the current block.

[0012] Additional methods described here use adaptive template size. One such method is provided for encoding or decoding a video that includes a current picture and at least a first reference picture. For at least the current block in the current picture, a template size is selected based on the current block size, and a prediction for the current block is generated using the template-based intermediate prediction. For example, samples in a template region that is adjacent to the current block and that has a selected template size can be compared with corresponding sample values in at least a first reference template to determine at least one intermediate prediction parameter based on the template, and a determined the parameter(s) may be used in performing intermediate prediction based on the template of the current block.

[0013] Additionally, the methods described herein use pattern segments when the encoding or decoding of the current block in one pattern segment is constrained so as not to use discrete samples in another template segment for intermediate coding based on the current block's template. Prediction modes besides template-based intermediate coding may still use discrete samples (or other coding information such as motion vectors) in another segment of the template. In one such method, a plurality of pattern segments is defined in the current image, where each pattern segment includes a plurality of blocks. The prediction mode is determined to encode each block in the current pattern segment. The prediction mode is selected from at least one intermediate prediction mode based on a template and at least one prediction mode not based on a template. A prediction is generated for each block in the current pattern segment, where the prediction of any block in the current pattern segment using the pattern-based prediction mode is constrained so that it does not use for prediction any discrete samples that are in the current picture but outside the current segment. template. Pattern segment boundary information may be signaled in the bitstream. For example, information about the number of coding tree elements (CTUs) in each template segment or the number of CTU rows in each template segment may be signaled in the bitstream. The use of template segments as described herein allows different template segments to be encoded or decoded in parallel.

[0014] Additional embodiments include encoder and decoder (collectively "codec") systems configured to perform the methods described herein. Such systems may include a processor and a durable computer storage medium storing instructions that are operable when executed on the processor to carry out the methods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG. 1A is a system diagram illustrating an exemplary communications system in which one or more of the disclosed embodiments may be implemented.

[0016] FIG. 1B is a system diagram illustrating an example of a wireless transmit/receive unit (WTRU) that may be used within the communication system of FIG. 1A, according to one embodiment.

[0017] FIG. 2 depicts an exemplary block-based video encoder.

[0018] FIG. 3 depicts an exemplary block-based video decoder.

[0019] Figure 4 depicts local illumination compensation.

[0020] FIG. 5 shows an example of pattern matching based on frame rate upconversion (FRUC).

[0021] FIG. 6 depicts discrete samples used to compute boundary discontinuities for use in predicting the sign of the transform coefficients.

[0022] FIG. 7 depicts potentially suitable motion vector difference options used in one example of finding the sign of the motion vector difference.

[0023] FIG. 8 depicts an overlay block motion compensation (OBMC) process for sub-unit modes, where OBMC is applied to all sub-CUs (e.g., sub-CU A) using MVs from all four adjacent blocks (e.g., shaded sub-CUs a, b, c, d).

[0024] FIG. 9 shows an example of parallel decoding by a HEVC decoder.

[0025] FIG. 10 shows an example of a decoding delay in JEM.

[0026] FIG. 11 depicts reduced decoding delay with MCP discrete samples (with OBMC) as a template for template-based intermediate prediction techniques.

[0027] FIG. 12 depicts reduced decoding delay with MCP discrete samples (without OBMC) as a template for template-based intermediate prediction techniques.

[0028] FIG. 13 depicts examples of discrete template samples used for template-based intermediate prediction.

[0029] Fig.14 depicts the lowest frequency response of the primary transformations that are used in JEM.

[0030] FIG. 15 shows an example in which the current image is divided into three pattern segments.

[0031] FIG. 16 depicts an exemplary process for encoding a block using template-based intermediate prediction, according to one embodiment.

[0032] FIG. 17 depicts an exemplary process for decoding a block using template-based intermediate prediction, according to one embodiment.

EXAMPLE NETWORKS FOR THE IMPLEMENTATION OF IMPLEMENTATION OPTIONS

[0033] FIG. 1A is a diagram illustrating an exemplary communications system 100 in which one or more of the disclosed embodiments may be implemented. Communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. Communication system 100 may allow multiple wireless users to access such content by sharing system resources including wireless bandwidth. For example, communication systems 100 may employ one or more channel access techniques such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), Single carrier FDMA (SC-FDMA), DFT-distributed Special Word Zero Tail OFDM (ZT UW DTS-s OFDM), Special Word OFDM (UW-OFDM), Resource Block Filtering OFDM, Multiple Carrier with Filter Bank (FBMC) etc.

[0034] As depicted in FIG. 1A, communication system 100 may include wireless transmitting/receiving units (WTRUs) 102a, 102b, 102c, 102d, RAN 104, CN 106, public switched telephone network (PSTN) 108, Internet 110 and other networks 112, although it should be understood that the disclosed embodiments involve any number of WTRUs, base stations, networks, and/or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device capable of operating and/or communicating in a wireless environment. As an example, WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a "station" and/or "STA", may be configured to transmit and/or receive wireless signals and may include user equipment (UE) , mobile station, fixed or mobile subscriber unit, subscription-based unit, pager, cell phone, digital assistant (PDA), smart phone, laptop, netbook, personal computer, wireless sensor, hotspot or Mi-Fi device, device internet of things (IoT), wristwatch or other wearable device, helmet-mounted display (HMD), vehicle, drone, medical device and applications (e.g. remote surgery tool), industrial device and applications (e.g. robot and/or other wireless devices operating in industrial and/or automated processing contexts), consumer electronic device, device operating in commercial and/or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be referred to interchangeably as a UE.

[0035] Communication systems 100 may also include base station 114a and/or base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to provide access to one or more communication networks, such as CN 106, Internet 110 and/or other networks 112. As an example, base stations 114a, 114b may be a base transceiver station (BTS), node-B, e-node-B, source node-B, source e-node-B, gNB , NR Node-B, entity manager, access point (AP), wireless router, and the like. the number of interconnected base stations and/or network elements.

[0036] Base station 114a may be part of RAN 104, which may also include other base stations and/or network elements (not shown), such as base station control (BSC), radio network control (RNC), relay nodes, etc. Base station 114a and/or base station 114b may be configured to transmit and/or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a wireless service to a specific geographic area, which may be relatively fixed or may change over time. The cell may be further divided into cell sectors. For example, a cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, ie, one for each cell sector. In one embodiment, the base station 114a may employ multiple input multiple output (MIMO) technology and may enable multiple transceivers for each sector of a cell. For example, beamforming can be used to transmit and/or receive signals in desired spatial directions.

[0037] Base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d via air interface 116, which may be any suitable wireless link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Air interface 116 may be established using any suitable radio access technology (RAT).

[0038] More specifically, as noted above, communication system 100 may be a multiple access system and may employ one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, base station 114a in RAN 104 and WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA) that may establish air interface 116 using Wideband CDMA (WCDMA). WCDMA may include communication protocols such as High Speed Packet Access (HSPA) and/or Enhanced HSPA (HSPA+). HSPA may include High Speed Downlink (DL) Packet Access (HSDPA) and/or High Speed UL Packet Access (HSUPA).

[0039] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement a radio technology, such as UMTS Enhanced Terrestrial Radio Access (E-UTRA), which may establish an over-the-air interface 116 using Long Term Evolution (LTE), and/ or LTE Enhanced (LTE-A), and/or LTE Enhanced Pro (LTE-A Pro).

[0040] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement a radio technology, such as NR radio access, that may establish air interface 116 using new radio (NR).

[0041] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, 102c can perform LTE radio access and NR radio access together, eg, using dual connectivity (DC) principles. Thus, the air interface employed by WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and/or transmissions sent to/from multiple types of base stations (eg, eNBs and gNBs).

[0042] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., wireless fidelity (WiFi), IEEE 802.16 (i.e., microwave broadband access technology ( WiMAX)), CDMA2000, CDMA2000 IX, CDMA2000 EV-DO, Intermediate Standard 2000 (IS-2000), Intermediate Standard 95 (IS-95), Intermediate Standard 856 (IS-856), Global System for Mobile Communications (GSM), enhanced data rates for the development of GSM (EDGE), GSM EDGE (GERAN), etc.

[0043] Base station 114b in FIG. 1A may be a wireless router, source Node-B, source e-Node-B, or access point, for example, and may employ any suitable RAT to enable wireless connectivity in a localized area, such as a location. businesses, home, vehicle, campus, industrial facility, air corridor (eg, for use by drones), roadway, etc. In one embodiment, base station 114b and WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to set up a wireless local area network (WLAN). In one embodiment, base station 114b and WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base station 114b and WTRUs 102c, 102d may use a cellular RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a pico cell or femto cell . As depicted in FIG. 1A, base station 114b may have a direct connection to Internet 110. Thus, base station 114b may not be required to access Internet 110 via CN 106.

[0044] RAN 104 may be in communication with CN 106, which may be any type of network configured to provide voice, data, applications, and/or Voice over Internet Protocol (VoIP) services to one or more WTRUs 102a, 102b, 102c , 102d. The data may have varying Quality of Service (QoS) requirements such as different bandwidth requirements, delay requirements, error tolerance requirements, reliability requirements, data bandwidth requirements, mobility requirements, etc. The CN 106 may provide call control, formation services accounts, location-based mobile services, prepaid calls, Internet connectivity, video distribution, etc. and/or perform high-level security features such as user authentication. Although not depicted in FIG. 1A, it should be understood that RAN 104 and/or CN 106 may be in direct or indirect communication with other RANs that use the same RAT as RAN 104 or a different RAT. For example, in addition to being connected to a RAN 104 that may employ NR radio technology, CN 106 may also be in communication with another RAN (not shown) employing GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technologies.

[0045] CN 106 may also act as a gateway for WTRUs 102a, 102b, 102c, 102d to access PSTN 108, Internet 110, and/or other networks 112. PSTN 108 may include circuit switched telephone networks that provide traditional telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and/or Internet Protocol (IP) in the TCP/IP Internet protocol suite. . Networks 112 may include wired and/or wireless networks owned and/or operated by other service providers. For example, networks 112 may include another CN connected to one or more RANs that may use the same RAT as RAN 104 or a different RAT.

[0046] Some or all of the WTRUs 102a, 102b, 102c, 102d in communication system 100 may include multiple mode capabilities (e.g., WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks via various wireless links). For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a that may employ cellular radio technology and with a base station 114b that may employ IEEE 802 radio technology.

[0047] FIG. 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1B, WTRU 102 may include a processor 118, a transceiver 120, a transmitter/receiver element 122, a speaker/microphone 124, a keyboard 126, a display/ touchpad 128, non-removable memory 130, removable memory 132, power supply 134, global positioning system (GPS) chipset 136, and/or other peripheral devices 138, among other things. It should be understood that WTRU 102 may include any subcombination of the above elements while remaining in accordance with the embodiment.

[0048] The processor 118 may be a general purpose processor, a special purpose processor, a standard processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a control medium, a microcontroller, application specific integrated circuits (ASICs), programmable gate array user (FPGA), any other type of integrated circuit (IC), state machine, and the like. Processor 118 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that provides the ability of the WTRU 102 to operate in a wireless environment. Processor 118 may be combined with transceiver 120, which may be combined with transmitter/receiver element 122. Although FIG. 1B depicts processor 118 and transceiver 120 as separate components, it should be understood that processor 118 and transceiver 120 may be integrated together in an electronic assembly. or crystal.

[0049] Transmitting/receiving element 122 may be configured to transmit signals to or receive signals from a base station (eg, base station 114a) via air interface 116. For example, in one embodiment, transmitting/receiving element 122 may be an antenna configured capable of transmitting and/or receiving RF signals. In one embodiment, the transmitter/receiver element 122 may be an emitter/detector capable of transmitting and/or receiving IR signals, UV signals, or visible light signals, for example. In yet another embodiment, transmitter/receiver element 122 may be configured to transmit and/or receive both RF signals and light signals. It should be understood that the transmitting/receiving element 122 may be configured to transmit and/or receive any combination of wireless signals.

[0050] Although the transmit/receive element 122 is shown in FIG. 1B as a single element, WTRU 102 may include any number of transmit/receive elements 122. More specifically, WTRU 102 may employ MIMO technology. Thus, in one embodiment, WTRU 102 may include two or more transmit/receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over air interface 116.

[0051] Transceiver 120 may be configured to modulate signals to be transmitted by transmitting/receiving element 122 and demodulating signals to be received by transmitting/receiving element 122. As noted above, WTRU 102 may have multiple mode capabilities. Thus, transceiver 120 may include a plurality of transceivers to enable WTRU 102 to communicate over multiple RATs, such as NR and IEEE 802.11, for example.

[0052] Processor 118 of WTRU 102 may be associated with and may receive user input from speaker/microphone 124, keyboard 126, and/or display/touch panel 128 (e.g., a liquid crystal display (LCD) based display unit or organic light emitting diodes (OLED)). Processor 118 may also output user data to speaker/microphone 124, keyboard 126, and/or display/touchpad 128. Additionally, processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and/or or removable memory 132. Non-removable memory 130 may include random access memory (RAM), read only memory (ROM), a hard drive, or any other type of storage device. Removable memory 132 may include a Subscriber Identity Module (SIM) card, a flash card, a secure digital (SD) memory card, and the like. In other embodiments, processor 118 may access information from and store data in a memory that is not physically located in the WTRU 102, such as on a server or home computer (not shown).

[0053] Processor 118 may receive power from power source 134 and may be configured to allocate and/or manage power to other components in WTRU 102. Power source 134 may be any suitable device for powering WTRU 102. For example, power source 134 may include one or more dry cells (e.g. nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0054] The processor 118 may also be combined with the GPS chipset 136, which may be configured to provide location information (eg, longitude and latitude) regarding the current location of the WTRU 102. In addition to or instead of the information from the GPS chipset 136 , WTRU 102 may receive location information via air interface 116 from a base station (eg, base stations 114a, 114b) and/or determine its location based on the timing of signals received from two or more nearby base stations. It should be understood that the WTRU 102 may acquire location information by any suitable location method while remaining in accordance with the embodiment.

[0055] The processor 118 may further be combined with other peripheral devices 138, which may include one or more software and/or hardware modules that provide additional features, functionality, and/or wired or wireless connectivity. For example, peripheral devices 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and/or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a headset, a Bluetooth® module, a radio with frequency modulation (FM), digital music player, media player, video game player, web browser, virtual reality and/or augmented reality (VR/AR) device, activity tracker, and the like. Peripheral devices 138 may include one or multiple sensors, the sensors may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, a direction sensor, a proximity sensor, a temperature sensor, a time sensor; geolocation sensor; altimeter, ambient light sensor, touch sensor, magnetometer, barometer, gesture sensor, biometric sensor and/or humidity sensor.

[0056] The WTRU 102 may include a full duplex radio for which the transmission and reception of some or all of the signals (eg, associated with specific subframes for both of the UL (eg, for transmission) and downlink (eg, for reception)) may be combined and/or simultaneous. A full duplex radio may include an interference handling unit to reduce and/or substantially eliminate internal interference through either hardware (e.g., a limiter) or signal processing by a processor (e.g., a separate processor (not shown) or by processor 118) . In one embodiment, the WRTU 102 may include a half duplex radio for which the transmission and reception of some or all of the signals (e.g., associated with specific subframes for either UL (e.g., for transmission) or downlink (e.g., for reception) ).

[0057] While the WTRU is described in FIGS. 1A-1B as a wireless terminal, it is contemplated that in specific representative embodiments that such a terminal may use (eg, temporarily or permanently), the wired communication interfaces with the communications network.

[0058] In view of FIGS. 1A-1B and the corresponding description of FIGS. 1A-1B, one, or more, or all of the functions described herein in relation to one or more of: WTRU 102a-d, base station 114a-b, e- node-B 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF 182a-b, UPF 184a-b, SMF 183a-b, DN 185a-b and/or any other device(s), described here may be performed by one or more emulation devices (not shown). Emulation devices may be one or more devices configured to emulate one or more or all of the functions described herein. For example, emulation devices may be used to test other devices and/or simulate network and/or WTRU functions.

[0059] Device emulation may be configured to perform one or more tests on other devices in a lab environment and/or in an operator network environment. For example, one or more emulation devices may perform one or more or all of the functions while being fully or partially implemented and/or used as part of a wired and/or wireless communication network in order to test other devices within the communication network. One or more emulation devices can perform one, or several, or all functions, while being temporarily implemented/used as part of a wired and/or wireless communication network. The emulation device may be directly coupled with another device for testing purposes and/or may perform the test using wireless over the air.

[0060] One or more emulation devices can perform one or more, including all, functions, while not being implemented / installed as part of a wired and / or wireless communication network. For example, emulation devices may be used in a test scenario in a test lab and/or an uninstalled (eg, test) wired and/or wireless network in order to test one or more components. One or more emulation devices can be test equipment. Direct RF bonding and/or wireless communication via RF circuits (eg, which may include one or more antennas) may be used by emulation devices to transmit and/or receive data.

DETAILED DESCRIPTION

Block-based hybrid video coding.

[0061] Like the HEVC Test Model (HM), the Joint Exploration Model (JEM) software is also based on a block-based hybrid video coding framework (100). 2 is a block diagram of a block-based hybrid video coding system. It should be noted that in this application, the terms "restored" and "decoded" can be used interchangeably. Usually, but not necessarily, the term "recovered" is used on the encoder side, while "decoded" is used on the decoder side.

[0062] Before encoding, the video sequence may undergo pre-processing, such as applying a color transform to the input color image (for example, converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a re-rendering of the components of the input image, in order to obtain a signal distribution that is more resistant to compression (for example, using histogram equalization of one of the color components). Metadata can be associated with preprocessing and attached to the bitstream.

[0063] The input video signal 102 is processed block by block. The HEVC datasheet distinguishes between "blocks" and "elements", where "block" refers to a specific area in an array of discrete samples (e.g., brightness, Y), and "element" includes associated blocks of all encoded color components (e.g., Y, Cb, Cr, or monochrome), syntax elements, and prediction data that are associated with blocks (eg, motion vectors). In this application, the term "block" can be used to refer to a data array of various sizes, and it can be used to refer to a macroblock and a sector, as defined in H.264/AVC, any of a code tree element (CTU), a coding element (CU), prediction unit (PU), transform unit (TU), coding unit (CB), prediction unit (PB), and transform unit (TB) as in HEVC, superblock or division in AV1, CTU, CU, TU, CB and TB, as in VVC (flexible video coding) or other video coding standards.

[0064] In HEVC, extended block sizes are used to efficiently compress high definition video signals (1080p and above). In HEVC, a CU can be up to 64×64 pixels in size. The CU may be further divided into prediction units for which separate prediction methods are applied. For each input video block (MB or CU), spatial prediction (160) and/or temporal prediction (162) may be performed.

[0065] Spatial prediction (or "intra prediction") uses pixels from discrete samples of already encoded neighboring blocks (which are referred to as reference discrete samples) in the same picture/video segment to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal.

[0066] Temporal prediction (also called "inter-prediction" or "motion-compensated prediction") uses recovered pixels from already encoded video images to predict the current block of video. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for some given video block is typically signaled by one or more motion vectors that indicate the amount and direction of motion between the current block and its reference block. Also, if multiple reference pictures are supported (as is the case for recent video coding standards such as H.264/AVC or HEVC), then for each video block, its reference picture index is additionally sent; and the reference index is used to identify from which reference picture in the reference picture store (164) the temporal prediction signal originates.

[0067] After spatial and/or temporal prediction, the mode decision block (180) in the encoder selects the best prediction mode, for example, based on a rate-to-distortion optimization method. The prediction block is then subtracted from the current video block (116); and the prediction residual is decorrelated using transform (104) and quantized (106).

[0068] The encoder decodes the encoded block to provide a link for additional prediction. The quantized residual coefficients are inversely quantized (110) and inversely transformed (112) to form a reconstructed residual, which is then added back to a prediction block (126) to form a reconstructed video block.

[0069] The encoder may also skip the transform and apply quantization directly to the untransformed residual signal. The encoder can also bypass both transformation and quantization, i.e. the remainder is encoded directly without applying a transformation or quantization process. In direct Pulse Code Modulation (PCM) coding, no prediction is applied and the bin samples are directly encoded into the bitstream.

[0070] Additional in-loop filtering such as an anti-block filter, SAO (discrete sample adaptive offset), and adaptive loop filters may be applied (166) to the reconstructed video block before it is placed in the reference picture store (164) and used to encode future video blocks. In order to generate output video bitstream 120, all of the coding mode (intermediate or intra), prediction mode information, motion information, and quantized residual coefficients are sent to the entropy coding block (108) to be further compressed and packed to form a bitstream .

[0071] FIG. 3 is a general block diagram of a block-based video decoder (200). A video decoder generally performs a decoding pass that is the reverse of a corresponding encoding pass that performs video decoding as part of encoding video data. The video bitstream 202 is first decompressed and entropy decoded in entropy decoding block 208 . The coding mode and prediction information are sent to either the spatial prediction block 260 (in the case of intra coding) or the temporal prediction block 262 (in the case of intermediate coding) to generate a prediction block. The residual transform coefficients are sent to inverse quantizer 210 and inverse transform 212 to reconstruct the residual block. The prediction block and the residual block are then summed at 226. The reconstructed block may further undergo in-loop filtering (266) before it is stored in the reference image store 264. The reconstructed video (220) in the reference picture store may then be stored, transmitted, or used to drive the display device, as well as used to predict future blocks of video.

[0072] The decoded image may further undergo post-processing, such as reverse color conversion (for example, conversion from YCbCr 4:2:0 to RGB 4:4:4) or reverse remapping performing a reverse process to the reverse mapping process performed in preprocessing. coding. Post-processing may use the metadata found in the precoding processing and signaled in the bitstream.

[0073] Both of HEVC and JEM adhere to block-based hybrid motion-compensated video coding/decoding workflows as shown in FIG. 2 and FIG. 3, and are based on the same functional units such as spatial prediction (i.e., i.e. intra prediction), temporal prediction (i.e. intermediate prediction), transformation, quantization, entropy coding and loop filters. However, several intermediate coding modules, especially those related to motion-compensated prediction, have been further extended and improved.

Template-based intermediate prediction techniques.

[0074] In HEVC, the coding parameters used for intermediate prediction (eg, motion vector (MV), reference index, weighted prediction parameters) are determined in the encoder by optimizing the rate-to-distortion ratio (R-D) and signaled to the decoder. Thus, the overhead signaling used to encode these intermediate encoding parameters can account for a significant portion of the output bitstream. In order to avoid signaling these parameters, two template-based intermediate prediction techniques are applied in JEM by finding these intermediate coding parameters in the decoder based on the sample samples of the template, for example, reconstructed neighboring samples of the current block that are decoded earlier. The first method is called Local Illumination Compensation (LIC). The LIC compensates for motion-compensated prediction based on scaling and offset, which are found based on discrete pattern samples, to solve the problem of local illumination variation between different images. The second method is called frame rate upconverting (FRUC) pattern mode in which motion information (MV and reference indexes) is found in the decoder based on pattern matching.

[0075] In addition to LIC and FRUC, other template-based methods have also been proposed for application to both intermediate prediction and residual coefficient signaling. In these methods, the recovered adjacent samples are used to decode the current block. In the following, aspects of these template-based encoding methods are also briefly described.

Local illumination compensation.

[0076] LIC is an encoding tool that is used in JEM to solve the problem of local illumination changes that exist in adjacent images in time, as described in J. Chen, E. Alshina, G. J. Sullivan, J. R. Ohm, J. Boyce, "Algorithm description of Joint Exploration Test Model 7 (JEM7)", JVET-G1001, July 2017, Toronto, Italy. LIC is based on a linear model where a scaling factor and an offset are applied to reference samples to obtain prediction samples of the current block. Concretely, LIC can be mathematically modeled by the following equation:

P ( x, y )= b ∙ P _r ( x + v _x , y + v _y )+ c (1)

where P ( x, y ) is the prediction signal of the current block in coordinates ( x, y ); P _r { x + v _x , y + v _y ) is the reference block pointed to by the motion vector ( v _x , v _y ); b and c are the respective scaling factor and offset that are applied to the reference block. 4 depicts the LIC process. In FIG. 4, when LIC is applied to a block of video, a linear minimum mean square error (LLMSE) method is employed to find the values of the LIC parameters ( b and c ) by minimizing the difference between adjacent discrete samples of the current block (pattern T in FIG. 4) and their respective sample references in temporal reference pictures (i.e. either T0 or T1 in FIG. 4), for example:

(2)

(3)

- соответствующий опорный дискретный отсчет дискретного отсчета шаблона на основе вектора движения (либо L0, либо L1) текущего блока. Дополнительно, чтобы уменьшить вычислительную сложность, дискретные отсчеты шаблона и опорные дискретные отсчеты шаблона могут быть субдискретизированы (субдискретизация 2:1), чтобы найти параметры LIC, например, только затененные дискретные отсчеты на фиг.4 могут быть использованы, чтобы найти б и в.where N represents the number of discrete samples of the pattern that are used to find the LIC parameters; T ( x _i , y _i ) - discrete sample of the current block template in coordinates ( x _i , y _i ); And

- the corresponding template sample sample reference based on the motion vector (either L0 or L1) of the current block. Additionally, to reduce computational complexity, the template samples and reference template samples can be downsampled (2:1 downsampling) to find the LIC parameters, for example, only the shaded samples in FIG. 4 can be used to find b and c .

Template-based upscaling.

[0077] In order to reduce traffic information overhead, FRUC is supported for intermediate blocks in JEM. When FRUC is enabled, both MV and reference picture indexes of the current block are not signaled; instead, they are generated on the decoder side. Specifically, in order to find the FRUC motion, a set of preliminary MV potential candidates generated from the current block's spatial and temporal neighbors is tested and the candidate candidate that results in the minimum sum absolute difference (SAD) is selected as the original MV. Then a local search around the original MV is performed and the MV with the minimum SAD is used as the MV for the entire block. In the current FRUC, two search algorithms are supported, including pattern matching and two-way matching. Compared to the template, the top and/or left decoded adjacent samples (without applying in-loop filters) of the current block are used to find block motion information by searching for the MV that provides the best match between the template and its corresponding block (e.g., a corresponding block having the same the same size as the template) in the reference image. 5 depicts a template-based FRUC.

Prediction of the sign of the transformation coefficients.

[0078] In some pattern-based coding methods, a sign prediction method is applied to reduce signaling overhead for signaling signs for transform coefficients. Examples of such methods are described in the works of Y.-W. Chen, et al., "Description of SDR, HDR and 360° video coding technology proposal by Qualcomm and Technicolor - low and high complexity versions", JVET-J0021, April 2018, San Diego, USA; and A. Alshin et al., "Description of SDR, HDR and 360° video coding technology proposal by Samsung, Huawei, GoPro, and HiSilicon - mobile application scenario", JVET-J0024, April 2018, San Diego, USA .

[0079] The sign prediction methods operate to perform a plurality of inverse transforms on the transform coefficients of a coding unit. For each inverse transform, the sign of the non-zero transform coefficient is set to either negative or positive. The sign combination that minimizes the overhead function is chosen as a sign predictor to predict the signs of the current block's transform coefficients. As one example to illustrate this idea, assuming the current block contains two non-zero coefficients, there are four possible combinations of signs, namely (+, +), (+, -), (-, +) and (-,-) . For all four combinations, the cost function is calculated and the minimum cost combination is chosen as the sign predictor. The cost function in this example is calculated as a measure of the discontinuity of discrete samples located on the border between the current block and its causal neighbors. As shown in Fig.6, the cost function is calculated as the sum of the absolute values of the second derivatives of the boundary discrete samples of the current block and the restored neighboring discrete samples above and to the left of the current block as follows:

(4)

(4)

where R _{x , y} is the reconstructed adjacent discrete sample, P _{x , y} is the prediction of the current block, and r _{x , y} is the residual hypothesis in coordinates ( x, y ).

Finding the difference sign of the motion vector.

[0080] Template matching techniques have also been proposed to reduce motion vector difference (MVD) sign signaling. In one such technique, based on the absolute values of the received MVD, a list of potentially suitable MVDs is generated by various combinations of sign values for each of the horizontal and vertical MVDs. The cost of each potentially eligible MVD is then computed using the template samples (restored neighboring samples) of the current block. Potentially eligible MVDs are sorted based on computed cost values. The final MVD is selected by sending an index into a sorted list of potential candidates from encoder to decoder. 7 depicts one example to illustrate the idea of deriving the motion vector difference sign, where A, B, C, and D are four possible MVD candidates that are generated by assigning different sign values to the received absolute MVD values. The index is signaled to identify one of the four potential candidates, and the identified potential candidate is used to reconstruct the final MVs of the current block.

Motion vector prediction based on template comparison.

[0081] In some implementations, for each conventional intermediate-coded block in HM and JEM, two potentially good motion vector predictors (MVPs) are generated, and the best prediction quality candidate is selected by encoder-to-decoder signaling the MVP index. In the work of Y.-W. Chen et al. (above) pattern-based MVP finding method is used to avoid MVP signaling. Specifically, pattern matching is used to find the MVP on the decoder side. First, two potentially eligible default MVPs are tested, and the candidate that results in a lower SAD between the template and its reference is selected as the starting point. Then, a local pattern matching search around the origin is performed, and the MV that results in the lowest matching overhead is selected as the MVP for the current block.

Reordering potentially suitable movement options.

[0082] In C.-W. Hsu et al., "Description of SDR video coding technology proposal by MediaTek", JVET-J0018, April 2018, San Diego, USA, a method of reordering potentially suitable motion options is used to improve the efficiency of the combining mode. Specifically, after the list of original candidate merges is generated, the original candidate merges in the list are reordered based on the pattern matching overhead that is computed between the template samples of the current CU and the corresponding reference template samples using the movement of the candidate merge . After reordering, potentially suitable joins with lower overhead can be placed before potentially suitable joins with higher costs. Thus, the efficiency of signaling indexes of potentially suitable joins can be improved by spending fewer bits on potentially suitable joins, resulting in better prediction quality.

Reordering the conversion syntax.

[0083] When Enhanced Multiple Transform (EMT) and/or Non-Separable Secondary Transform (NSST) are used, a reordering of the transform syntax has been proposed by C.-W. Hsu et al. (above) for use in reordering the indexes of a set of transforms that can be chosen, for example, based on the same cost function as used to predict the sign of the transform coefficients (as described above in the "Sign Prediction of the Transform Coefficients" section), using reconstructed neighboring discrete samples of the block. Potentially eligible EMT and/or NSST mapping options are reordered on a cost basis, and those with lower overhead will be assigned short codewords.

Block motion compensation with overlay.

[0084] Overlapping block motion compensation (OBMC) is used in the JEM reference software to remove block artifacts in the motion compensation step. In JEM, OBMC is performed for all intermediate block boundaries except for the right and bottom block boundaries. Additionally, when a block is divided into a plurality of subblocks and each subblock is associated with its own MV (eg, FRUC blocks), OBMC is also performed for each subblock boundary. Fig.8 depicts the concept of OBMC. Specifically, when OBMC is applied to a subblock (eg, subblock A in FIG. 8), in addition to the MV of the current subblock, the MVs of the four neighboring subblocks are also used to find prediction signals of the current subblock. Then, a plurality of prediction signals using the MVs of neighboring subblocks are averaged to generate a final prediction of the current subblock.

Adaptive multiple kernel transform.

[0085] In addition to the DCT-II and DST-VII kernel transforms that are used in HEVC, the Adaptive Multiple Transform (AMT) tool is used to encode the residuals of both internal and intermediate blocks in JEM. Specifically, AMT represents four additional core transforms from the set of DCT/DST transforms, including DCT-VIII, DCT-V, DST-VII, and DST-I. AMT is applied to all coding blocks whose width and height are not greater than 64, and a flag is signaled to indicate whether AMT is enabled or not. When the flag is 0, this indicates that DCT-II is being used as the transform for the block; otherwise (ie, the flag is 1), three subsets of transforms (each containing two different AMT core transforms defined in Table 1) are defined. When AMT is applied to an intra block, a subset of transforms is first selected based on the block's intra prediction direction. Then, two additional flags are signaled to indicate which transform (out of the two kernel transforms in the selected subset of transforms) is used as the horizontal transform and vertical transform, respectively. For intermediate blocks, only the transform subset #0, which consists of DCT-VIII and DST-VII, is used.

Table 1. Three subsets of transformations defined in the JEM.

transform subset no. Nuclear transformations 0 DST-VII, DCT-VIII 1 DST-VII, DST-I 2 DST-VII, DCT-V

Mode-dependent non-separable secondary transformation.

[0086] Since the correlation between spatial adjacent discrete samples is generally less than in the time domain, there are often strong correlations within residual discrete samples generated by intra prediction. To further improve the efficiency of intra coding, a tool called mode-dependent non-separable secondary transform (NSST) is applied in JEM by applying a non-separable transform to transform coefficients of intra blocks. Specifically, if both the width and the height of the block are not less than 8, an 8×8 non-separable transform is applied to the upper left region of the 2D transform coefficient array of the 8×8 block size; otherwise (i.e. either width or height is 4, which is the minimum size of an encoding element in JEM), a 4×4 non-separable transform is applied to the upper left region (in size min(8,W)×min(8,H)) of the transform coefficients block. To illustrate NSST, it assumes that the input X is a 4×4 block, defined as:

To apply NSST, the input block is translated into a vector as follows:

(5)

, где

указывает вектор коэффициентов преобразования, и T - матрица преобразования NSST 16Ч16. Вектор коэффициентов

16Ч1 преобразуется как блок 4Ч4 с использованием порядка сканирования для этого блока (горизонтального, вертикального или диагонального). В JEM, вместо использования матричных умножений, преобразование гиперкуба Гивенса (HyGT) на основе осуществления бабочки используется, чтобы уменьшить вычислительную сложность неразделяемого преобразования.The NSST is then applied as

, Where

indicates a vector of transform coefficients, and T is a NSST 16×16 transform matrix. Coefficient vector

16×1 is converted as a 4×4 block using the scan order for that block (horizontal, vertical, or diagonal). In JEM, instead of using matrix multiplications, the Givens hypercube transform (HyGT) based on butterfly implementation is used to reduce the computational complexity of the non-separable transform.

Problems Resolved in Exemplary Embodiments.

[0087] Like its predecessors, the HEVC standard employs motion-compensated prediction (MCP) to effectively reduce temporal redundancy between pictures, thereby achieving high intermediate coding efficiency. Because the MCP only uses samples from already decoded pictures to predict the samples in the current picture, there is no relationship between the MCPs of the spatial neighboring blocks. This means that the MCPs of intermediate blocks in the same picture/segment are independent of each other. Thus, the decoding processes of a plurality of intermediate blocks in the current picture can be performed in parallel, for example, they can be assigned to different streams to take advantage of the parallelism.

[0088] As described above, some template-based intermediate prediction methods (eg, FRUC and LIC based on template matching) are applied in JEM. To avoid signaling coding parameters, template-based intermediate prediction methods find these parameters in both the encoder and decoder using the reconstructed samples of the current block's spatial neighbors. Thus, when a block is encoded by one of these template-based intermediate prediction techniques, its decoding process waits until its neighboring block samples (template samples of the current block) have been completely recovered. This can complicate channel design, especially on the decoder side, thus leading to a significant increase in complexity for hardware implementation.

[0089] In order to understand the parallel processing problem caused by the intermediate prediction pattern based methods, FIGS. 9 and 10 show examples of comparing HEVC and JEM decoding processes. To help explain, four consecutive coding blocks of equal block size in a picture are used as examples, where each coding block is decoded by a separate decoding stream, and the decoding complexity of each separate decoding module (e.g., MCP, OBMC, dequantization, and inverse transform) is assumed to be one and the same for these four coding blocks. It is further assumed that all coding blocks in the JEM example are encoded based on one of the template-based intermediate prediction techniques. In Fig.9 and Fig.10 blocks represent the process _of decoding MCP, OBMC, dequantization and inverse transformation, and the variables T _MCP , T _OBMC , T _de-quant and T _{inv -tra ns} - decoding time of these four modules. As shown in FIG. 9, since four coding blocks can be decoded in parallel, the total decoding time of HEVC is equal to the decoding time _of one coding block, i.e., T _MCP + T _de-quant + T _{inv -tra ns} .

[0090] In view of the dependency represented by template-based prediction techniques, for the JEM decoding process (as shown in FIG. 10), the decoding of each individual coding block cannot be activated until its spatial neighbor blocks are completely recovered. Thus, the total decoding time of the JEM is equal to the sum of the decoding times of the four blocks, i.e., T _total =4*( T _MCP + T _OBMC + T _{de - quant} + T _{inv - trans} ). It should be noted that although the example in FIG. 10 assumes that OBMC is used, the same delay problem exists even when OBMC is not used. In general, intermediate encoded blocks in HEVC can be recovered independently of each other because the MCP process in HEVC only requires those discrete samples from reference pictures that have already been completely recovered. This means that it is easy to parallelize the restoration of intermediate blocks in HEVC. In contrast, the use of template-based methods such as FRUC and LIC in JEM represents a dependency between adjacent intermediate encoded blocks. If an intermediate block is encoded using one of these modes, that block's MCP cannot start until its neighboring blocks are completely recovered, thus greatly increasing the delay.

Overview of Embodiments.

[0091] In order to solve the delay problems described above, methods are described herein to reduce the encoding/decoding delay of template-based intermediate prediction methods while maintaining major coding efficiency. In the proposed methods, some of the functions of the template-based intermediate prediction methods remain the same as in existing designs. For example, for LIC, the parameter finding and discrete sample linear adjustment processes remain the same; and for FRUC, the motion search process based on pattern matching remains the same. However, the generation of pattern samples used by the intermediate pattern-based prediction is modified to reduce the dependence between adjacent blocks so that the overall encoding/decoding delay due to the intermediate pattern-based prediction is reduced. Specifically, compared to existing template-based methods in JEM, the changes proposed in this disclosure include those described below.

[0092] Compared to existing template-based intermediate prediction methods in JEM, where reconstructed adjacent samples are used to find coding parameters, it is proposed to use a prediction signal (i.e., prediction samples generated from MCP and, if applicable, OBMC ) spatial neighbors as discrete samples of the template for the current block. Thus, encoding/decoding of a pattern-based coding block can be activated as soon as a prediction signal of its adjacent blocks becomes available. Using only the prediction signal as a template may be less accurate than a completely reconstructed signal because the reconstructed residual has not yet been added. This may result in some deterioration in encoding performance. In order to reduce the performance penalty, additional methods are proposed to further improve the performance of template-based intermediate prediction coding when a prediction signal is used as a source of template samples.

[0093] In order to reduce the template-based intermediate prediction delay, it is further proposed to partition the image/segment into a plurality of "pattern segments" so that decoding of intermediate blocks within the template segment can be performed independently of other template segments. In order to achieve such parallelism, it is proposed to prohibit the coding block from using samples that are not in the same segment of the block template as the template samples. Additionally, to reduce potential coding loss, template segments break only the dependency of template samples, but still allow other encoding processes (e.g., in-loop filters, intra prediction, advanced motion vector prediction (AMVP), and so on) to operate by crossing segment boundaries. template.

[0094] Without loss of generality, in the following discussion, FRUC and LIC based on pattern matching are used as examples to explain the proposed methods. However, the schemes to be described are also applicable to other pattern-based coding schemes in which reconstructed adjacent block samples are used during the encoding and/or decoding process.

Template-based intermediate prediction based on the prediction signal.

[0095] As stated above, compared to HEVC, using reconstructed adjacent samples as a template for template-based intermediate prediction methods is disadvantageous for parallel coding/decoding for codec implementation, since template-based coding/decoding of a coding block must wait, until all of its causal adjacent discrete samples are completely reconstructed by MCP, OBMC (if applicable), dequantization, and inverse transformation.

[0096] In order to reduce encoding/decoding delay, in one embodiment of the disclosure, it is proposed to perform template-based intermediate prediction based on a prediction signal, instead of a completely reconstructed signal, from neighboring blocks. In one embodiment, instead of using the reconstructed neighbor samples, the spatial neighbor block prediction samples are used as a template for the current block in FRUC motion search, LIC parameter discovery, and/or other template-based parameter discovery. Based on such a design, encoding/decoding of a pattern-based coding block can start immediately after a prediction signal of its neighboring blocks is generated, without waiting for the reconstruction of the residuals of neighboring blocks. This can result in significant delay reduction for intermediate prediction pattern based methods since the encoding/decoding delay caused by dequantization and demapping of neighboring blocks is now eliminated.

[0097] Based on the assumptions in FIG. 9 and FIG. 10, FIG. 11 depicts a parallel decoding process when the method proposed here is applied to template-based intermediate prediction methods. As shown in FIG. 11, since the decoding process of each pattern-based coding block can be immediately activated after intermediate prediction of its neighboring block, the decoding delay between two successive blocks is equal to the decoding time for generating the prediction signal of the first block, i.e., T _latency = T _MCP + T _OBMC . Thus, the total decoding time of the four coding blocks is now reduced to T _total =4*( T _MCP + T _OBMC )+ T _{de - quant} + T _{in v - trans} .

[0098] In FIG. 11, OBMC is applied to generate intermediate block prediction samples, whose samples are then used as template samples to find the corresponding coding parameters (eg, FRUC MVs and LIC parameters) for its neighboring blocks. While OBMC can improve prediction quality by removing block artifacts due to the many motion compensations that are performed, it also introduces an increase in encoding/decoding complexity, thus leading to significant encoding/decoding delay when combined with template-based intermediate prediction methods. On the other hand, since OBMC only modifies edge bin samples, its impact on the accuracy of calculating coding parameters is relatively low considering its increase in complexity for generating a prediction signal.

[0099] Thus, in order to achieve a better performance/latency trade-off, it is proposed in some embodiments to use the prediction signal generated before OBMC as discrete template samples for template-based intermediate prediction methods. 12 shows a parallel decoding process when the above method is applied to the intermediate prediction based on the template. As shown in FIG. 12, since decoding of each pattern-based block can immediately start after its neighbor's MCP has ended, the decoding delay between two neighboring blocks is further reduced to T _latency = T _MCP . Accordingly, the total decoding time of the four coding blocks now becomes T _total =4* T _MCP + T _OBMC + T _{de - quant} + T _{inv - trans} .

[0100] It should be noted that in FIGS. 11 and 12, it is assumed that OBMC is enabled for the current block and/or its neighboring blocks. This is just an example. The proposed pattern matching techniques based on the prediction signal are equally applicable when OMBC is not enabled.

[0101] In some embodiments, intra-block prediction samples may also be used as template samples for decoding template-coded blocks. However, due to the fact that intra-coded blocks also use the reconstructed discrete samples of their respective spatial neighbors to generate discrete prediction samples, such use of discrete prediction samples may not be optimal for hardware design due to propagated encoding/decoding delay from adjacent intra-coded discrete samples. to the current template block. In some embodiments, for example, when using prediction samples for the pattern-based methods disclosed herein, neighboring block prediction samples that are intra-coded are prohibited from being used as template samples of the current block. Specifically, in such embodiments, if a discrete pattern sample originates from an intra-coded neighboring block, it is considered to be inaccessible to the current block.

[0102] In FIGS. 11 and 12, prediction samples (generated either after or before OBMC is applied) are used to replace the recovered samples as template samples for template-based intermediate prediction methods. Although the methods can significantly reduce encoding/decoding delay, differences between the prediction discrete samples and the reconstructed residual samples (recovered residual signal) can result in coding parameters that are found for template-based intermediate prediction (e.g., motion vectors for FRUC or parameters weight/offset for LIC) become less accurate, thus leading to a negative impact on the overall encoding performance. Additional methods are further provided herein to compensate for a possible degradation in coding performance when a prediction signal is involved in template-based intermediate prediction.

Template-based intermediate prediction by adaptively enabling discrete prediction samples based on the existence of non-zero residuals.

[0103] Embodiments in which the prediction signal is used in template-based prediction can retain much of the intermediate template-based prediction coding efficiency, especially in cases where the recovery signal and the prediction signal are very similar. However, the use of a prediction signal may be less satisfactory when there are large prediction errors (ie, non-zero residuals with large absolute values) for adjacent template samples. In another situation, if adjacent template samples are encoded with zero residuals, it is still safe to use them as template samples for the current block, provided they are the same as the recovered samples.

[0104] In some embodiments, it is proposed to use only those discrete prediction samples associated with zero residuals as template discrete samples when template-based intermediate prediction (eg, FRUC and/or LIC) is applied. In some embodiments, the decision whether a prediction sample has a non-zero residual is made based on a coded block flag (CBF) signaled in the bitstream that indicates whether the block contains non-zero transform coefficients. When the flag is equal to one, it means that there are non-zero remainders in the current block; thus, its prediction samples may be less reliable for use in template-based intermediate prediction (in which case, the use of such prediction samples as template samples may be prohibited). Otherwise (i.e. if CBF is 0) it signals that all remainders associated with the current block are zero; accordingly, it is permissible to use prediction samples for intermediate prediction based on the template (and in this case, the use of such prediction samples as template samples may be allowed).

[0105] Fig. 13 depicts one example to illustrate such a method. 13, adjacent sample positions are labeled A ₀ and A ₁ , and neighboring sample positions labeled 10-L ₃ belong to the top neighbor block Blk0 and left neighbor block Blk2, respectively, both of which have a CBF of 0. Since there is only a zero remainder for these two adjacent blocks, the prediction samples at these positions, labeled A ₀ and A ₁ and L ₀ -L ₃ , can be used as template samples in template-based methods (e.g., FRUC and/or LIC). However, since the adjacent sample positions labeled A ₂ and A ₃ belong to the top neighboring block Blk1 whose CBF is 1, the corresponding prediction samples cannot be used as template samples in some embodiments; this is so because they are different from the reconstructed values of the discrete samples, and thus can be considered less accurate.

[0106] In some such methods, the decision on whether all residuals are equal to zero is made for each adjacent discrete sample, and decisions about discrete samples in the same template may differ from each other. Such sample-based discrimination may be unfavorable for practical hardware implementations, for example, special care may be required for the SAD FRUC calculation and LIC parameter finding process, where sample samples in a template are treated differently based on whether they have residuals.

[0107] In order to avoid such complexity, it is proposed to completely enable/disable discrete samples of the subregion of the template, which is either above or to the left of the current block. Specifically, by such a method, the samples in the template sub-region (either the left template sub-region or the upper template sub-region) of a block are allowed to be used as template samples only if all residuals associated with all the samples in that template sub-region are zero. Based on the same example in FIG. 13, since all discrete samples in the left pattern sub-region (i.e., L ₀ -L ₃ ) belong to the same neighboring block whose CBF is 0, the left pattern sub-region is resolved for an intermediate template based predictions. On the other hand, since the adjacent block, which contains samples A ₂ and A ₃ , has residuals, the entire top sub-region of the template (ie, A ₀ -A ₃ ) is not used as sample samples of the template.

Template-based intermediate prediction based on weighted discrete prediction samples.

[0108] As discussed above, whether there are residuals or not is a factor in determining how similar the prediction samples and recovered samples are. In some embodiments, discrete prediction samples that have small residuals play a more important role than discrete prediction samples that have large residuals in order to achieve an accurate estimate of coding parameters for template-based intermediate prediction. In some such embodiments, it is proposed to apply unequal weights to different discrete prediction samples based on their residual values.

[0109] Again, using the same example in FIG. 13, since the discrete prediction samples A ₀ and A ₁ and L ₀ -L ₃ have no residuals, they are considered more reliable. When using these discrete samples to estimate the template parameters, a larger weighting factor can be assigned to these discrete samples. Conversely, since prediction samples A ₂ and A ₃ have non-zero residuals, these samples will be given less weight when used as template samples. The weighted discrete prediction samples will be used as input for estimating the template parameters. Specifically, with weighted template samples in comparison with the FRUC template, the weighted SAD between the block template samples and its matched samples in the reference picture is calculated as:

(6)

where T ( x _i , y _i ) is a discrete pattern sample in coordinates ( x _i , y _i ); T _r ( x _i +v _x , y _i + v _y ) is the corresponding reference sample of the sample sample based on the motion vector ( v _x , v _y ) in the reference image; and w _i is the weight that is assigned to T ( x _i , y _i ). Additionally, with the integration of weighting factors into finding the scaling factor and offset for the LIC, equations (2) and (3) become:

(7)

(8)

[0110] Various methods can be used to calculate the value of w _i . In one embodiment, it is proposed to compute a weight value for each discrete prediction sample according to the absolute value of its remainder. Concretely, in this method, the weight coefficient assigned to the discrete sample of the template is calculated as:

(9)

where W ( x _i , y _i ) is the unnormalized pattern sample sample weight T ( x _i , y _i ), which can be found based on the absolute value of the sample sample residual Res ( x _i , y _i ) as:

(10)

[0111] Since all primary transforms and secondary transforms that are applied in JEM are orthogonal transforms, the total energy of the coding element residual signal is preserved before and after the transform/inverse transform. Thus, with the exception of a slight influence of rounding errors caused by the conversion of floating point operations to fixed point operations in the conversion/inverse conversion, the total energy of the residual discrete samples of the block is essentially the same as that of the block transformation coefficients. Based on such a characteristic, in order to reduce the encoding/decoding delay, the absolute value ratio that is used in equation (10) can be calculated in the transform domain without applying dequantization and inverse transform.

[0112] In another embodiment, two fixed weights w ₀ and w ₁ are used (where w ₀ > w ₁ and w ₀ + w ₁ =1); w ₀ is used for all discrete prediction samples that have zero residuals, and w ₁ is used for all discrete prediction samples that have non-zero residuals.

Template-based intermediate prediction based on adding a DC offset to discrete prediction samples.

[0113] As depicted in FIG. 2 and FIG. 3, as in HEVC and its predecessor standards, 2D transform and 2D inverse transform are also used in JEM to achieve good energy compression on the prediction residual. For most cases, after horizontal and/or vertical conversion, most of the signal energy in the coding block is concentrated in a few low frequency coefficients. In particular, for particular coding blocks that are covered in areas that do not have strong directional edges, textures, and/or fast moving objects, the error between the original signal and the block prediction signal (i.e., block residuals) can be relatively constant or slowly varying over different positions of discrete samples within the block. In such a case, the 2D transform can uniformly compress the energy of the residuals into a single factor (eg, the DC factor for the standard DCT-II transform that is used in HEVC and previous standards) in the transform domain.

[0114] As discussed above, the accuracy of the found template parameters depends on the similarity between the prediction signal and the reconstructed template sample signal. A closer look at how the FRUC motion search is performed and how the LIC parameter finding is performed, it can be seen that the values of the discrete samples of the pattern are often estimated in a cumulative manner (either in the SAD form or in the following equations (2) and (3)). This means that the high frequency components in the template signal can be much less important than the low frequency components of the template. In some embodiments, it is proposed to correct the prediction signal by adding only the DC component to the discrete prediction samples before they are used as template samples for intermediate prediction pattern based methods. This will correct the most important sample rate of the pattern (i.e. the DC component) without causing any delay, since recovery of the DC component is relatively easy and does not require inverse conversion.

[0115] In block-based video coding, when lossy coding is applied, the transform coefficients (including the DC coefficient) must be quantized before being sent from the encoder to the decoder. Thus, in some embodiments, in order to obtain the correct value of DC residuals ( DC _rec ), the DC coefficient ( DC _coef ) that is parsed from the bitstream will be dequantized before being added to the prediction signal, for example, as follows:

(eleven)

where round (∙) is a function that rounds the incoming float value to the nearest integer; Q _step - quantization step size; and QP is the quantization parameter. Additionally, due to the increase in dynamic range caused by 2D orthogonal transforms (eg, kernel transforms in the DCT/DST set), another factor is introduced into Equation (11) to ensure that the calculated DC value is in the correct dynamic range. Specifically, in the proposed method, the following DC offset is added to the discrete prediction samples:

(12)

where nT _h and nT _v are the size of the horizontal and vertical transformations that are applied to the block. Equation (12) is a floating point operation. In practice, the divisions and multiplications in equation (12) can be approximated by multiplying the scaling factors followed by a right shift of the appropriate bits. Based on this consideration, in one embodiment, equation (12) becomes:

(13)

), чтобы аппроксимировать значение

, которые вычисляются как:where M _{B size} and L _{B size} are the scaling factor and the number of right shift bits that are used (together with right shift

) to approximate the value

, which are calculated as:

(14)

[0116] As shown in equation (14), M _BSize and L _BSize are introduced to compensate for the 0.5-bit right offset when the increase in dynamic range due to the size of the 2D transform cannot be done cleanly by right offsets. Based on the existing quantization design in HEVC and JEM, the quantization step size increases by about 2 ^1/6 times with every QP increment and increases by about 2 ^0.5 times for every 3 QP increments. Thus, in another embodiment, instead of the scaling-based method (as in Equation (14)) it is proposed to decrease the QP value by 3 to approximate a 0.5-bit right shift when the 2D transform dynamic range increase is not exactly a power of 2. Specifically, in this method, the DC offset is calculated as:

(15)

where QP _offset is:

(16)

[0117] In addition to the standard DCT-II kernel transformation, other transformation bases, such as DCT-VIII, DCT-V, DST-VII, DST-I, and KLT-based transformation bases, are used in both the primary and secondary transformation steps. conversions in the current JEM to further improve the conversion coding efficiency. Due to the different characteristics of these conversion basis functions, the conversion gain at the lowest frequency may not correspond to the true DC value.

[0118] Fig. 14 depicts the lowest frequency response of the primary transform functions that are used in the current JEM, where each drawing element corresponds to a pair of horizontal transform and vertical transform. As shown in FIG. 14, apart from the DCT-II+DCT-II and DCT-V+DCT-V transforms, the lowest frequency responses of the other nuclear transform basis functions are far from accurate for approximating the actual DC value. Based on such an observation, in one embodiment, it is proposed to add a DC offset (eg, the offset calculated in equation (12)) to the prediction signal only when the block is encoded by either DCT-II or DCT-V primary transform; for blocks that use other primary transform functions, the prediction signal will be directly used as template samples without adding an offset.

[0119] In another embodiment, it is proposed to add the DC offset to the prediction signal only when the block is encoded without NSST; for blocks that are encoded with NSST, the prediction signal will be directly used as sample samples of the template without adding an offset. In another embodiment, it is proposed to add a DC offset to the prediction signal of all blocks, regardless of which primary transform function is used and whether NSST is applied.

[0120] In another embodiment, when a block is encoded with AMT or NSST capability, it is proposed to introduce a different DC scaling factor to compensate for the difference between the lowest frequency transform factor and the actual DC value. By introducing such a scaling factor, Equation (12) additionally becomes:

(17)

where Scale _DC is the DC scaling factor, which can be calculated as the ratio of the signal energy of the lowest frequency response of the applied transform functions (for example, those shown in Fig. 14) to the signal energy of the DCT-II transform, for which the lowest frequency response is true DC. As one special case, when DCT-II is used as the primary transform, the value of Scale _DC is 1, i.e. no scaling is applied.

Template-based intermediate prediction based on adaptive template size.

[0121] In the current JEM intermediate prediction pattern based methods, a fixed pattern size (eg, four rows/columns of adjacent samples for FRUC pattern matching mode and one row/column of adjacent samples for LIC) is used. However, such a design may not be optimal due to the fact that the correlation between coding element samples and its template samples is dependent on the selected template size. For example, choosing a large template size has the disadvantage that template samples tend to be far from the target block. Thus, the correlation between the current block and its template may not be sufficient. On the other hand, a large template size may also provide advantages in that it may provide a more reliable estimate in the presence of noise (eg, video capture noise and noise caused by the encoding process). Some embodiments operate to select an appropriate template size for various block sizes, which must be large enough against noise while not exceeding a size limit to provide sufficient correlation between template and target block.

[0122] In one embodiment, template size 2 is used for blocks for which either the width or height is less than or equal to 8. Otherwise (both width and height are greater than 8), the larger template size 4 (i.e., L = 4) used. In another embodiment, the pattern size may be adaptively selected at the encoder and signaled to the decoder. To do this, additional syntax elements can be introduced, for example, at the sequence, image and/or segment level, to define the appropriate template sizes that are used for different block sizes. Various methods may be applied in the encoder to determine the optimal template size (eg, for some given block size). For example, analysis may be performed at the encoder to determine the noise that is contained in the original video as well as the encoding noise that the encoding process may introduce (eg, based on a QP value). If the noise level is high, then a large pattern size may be chosen to cancel out the noise; otherwise (i.e., the noise level is low), a small template size can be chosen to provide sufficient correlation between the samples of the current block and the template.

Template-based intermediate prediction based on the segment of the template.

[0123] As noted above, one reason for encoding/decoding delay for template-based intermediate prediction methods is the dependency between the MCP of a template-based coding block and the reconstruction of its adjacent discrete samples. Further, as shown in FIG. 10, the worst-case encoding/decoding delay introduced by pattern-based prediction directly relates to the maximum number of consecutive blocks that are encoded by pattern-based intermediate prediction methods.

[0124] In some embodiments, "template segments" are used to limit the maximum encoding/decoding delay of template-based intermediate prediction techniques. Specifically, using template segments, an image/segment is divided into a plurality of individual slices (or template segments) and the decoded samples of each block in the slice are only allowed to be used as template samples for template-based encoding of other blocks in the same slice. , but it is not possible to use as discrete samples of the pattern for encoding based on the pattern of blocks in any of the other slices. Thus, the encoding/decoding of blocks in each pattern segment can be completely independent of the other pattern segments.

[0125] Additionally, in embodiments, unlike the standard segments, the template segments break only the dependency of the intermediate prediction based on the template; they do not affect other decoding processes. For example, for in-loop filters, intra-prediction, and AMVP, an operation may still be performed by crossing the pattern segment boundaries. In this way, potential coding loss due to removal of the discrete pattern sample dependency between pattern segments can be minimized. 15 provides one example to illustrate the concept of the proposed template segment. As shown in FIG. 15, the decoding processes of multiple pattern segments can now be performed in parallel by assigning them to different streams.

[0126] Similar to other parallelism tools in HEVC and JEM (e.g., segments, cells, and parallel wavefront processing (WPP)), for template segments, the number of template segments and the position of each template segment within the image/segment can be freely chosen by the encoder and signaled to the decoder. Different choices (eg, number of pattern segments, as well as their sizes and positions) can lead to different trade-offs between encoding performance and encoding/decoding parallelism. In general, having more pattern segments improves concurrency, but also reduces coding efficiency.

[0127] In terms of template segment signaling, syntax elements can be added at the sequence and/or image level. For example, the number of CTUs in each segment of the template may be signaled in a sequence parameter set (SPS) and/or picture parameter set (PPS), or may be signaled in a segment header. Other variations of the syntax elements may be used, for example the number of CTU lines may be used or the number of template segments in each picture may be used, etc. A flag may be signaled to indicate at the sequence or picture level whether template segments are used.

[0128] In another embodiment, the numbers of CTUs or CTU rows of each template segment may be different from each other. For example, a syntax element may be signaled first in the SPS, PPS, and/or segment to indicate the number of template segments in the current sequence, picture, or segment, followed by a set of other syntax elements that specify the corresponding number of CTUs or CTU lines in each template segment. respectively. Additionally, in the above consideration, the boundary of each template segment is always aligned with the CTU boundaries (so that the template segment size is a multiple of the CTU size). In another embodiment, it is proposed to signal in the bitstream the size of the base element (which must be a multiple of the minimum coding element size as defined in the SPS) that is used to make up the template segment.

Applying the proposed template-based methods to other template-based encoding methods.

[0129] In the above discussion, the pattern comparison of FRUC and LIC is used as examples to illustrate proposed template-based methods, such as the methods described in "Template-Based Intermediate Prediction Based on Prediction Signal" and "Template-Based Intermediate Prediction based on template segment" above. However, in some embodiments, the proposed methods apply to other template-based coding schemes. The proposed methods as discussed above can be adapted to other pattern-based coding methods where reconstructed neighboring samples are used to encode/decode the current block, including pattern-based coding methods such as transform coefficient sign prediction, difference sign derivation motion vectors, motion vector prediction based on a template, reordering of potentially suitable join options, and reordering of the transformation syntax.

[0130] For example, when using the proposed template sample sampling for these template-based encoding methods based on prediction samples instead of reconstructed neighboring samples, prediction samples (with or without the addition of a DC component) are used to calculate the cost functions that are used, to select sign values (eg, predicting the sign of the transform coefficient and deriving the sign of the difference of the motion vectors) and motion vector (eg, pattern-based motion prediction and reordering of potentially suitable joins). As one example, when pattern finding based on prediction is applied to predicting the sign of the transform coefficients, the cost function described in equation (4) becomes:

where Pred is the neighboring sample prediction signal, P is the prediction of the current block, and r is the residual hypothesis.

[0131] More generally, when a plurality of prediction techniques (e.g., AMVP or combining mode like HEVC, FRUC, LIC, template-based MVD sign prediction, template-based MV prediction, template-based sign prediction, and OBMC) are available, a version of the prediction signal may be generated for an adjacent block using one or more prediction techniques. This original version may be further adjusted based on one or more other prediction techniques to refine a prediction signal that may be added to the prediction residual for the adjacent block to reconstruct the adjacent block. To encode or decode the current block, a neighboring block prediction signal (original version or adjusted version) may be used to generate a pattern when applying intermediate prediction based on the pattern to the current block.

[0132] In the examples as shown in FIGS. 9-12, prediction processes (MCP, OBMC) are assumed to be performed prior to prediction residual generation processes (dequantization and inverse transformation). However, the prediction processes may be performed independently of the prediction residual generation processes, and the prediction block and the residual block may be obtained in orders other than those described previously. It should be noted that the present techniques for using adjacent prediction samples for intermediate prediction based on the current block pattern will still be applicable when the residual block is generated in a different order.

[0133] FIG. 16 depicts an exemplary method 1600 for encoding a block using template-based intermediate prediction, according to one embodiment. Method 1600 begins at initialization block 1605. During the initialization step, the encoder may access discrete neighbor block prediction samples, which can be used to generate the pattern and CBFs of the neighboring blocks. At 1610, a template for the current block is generated based on discrete prediction samples of adjacent blocks. As previously described, the pattern may be formed by adjacent samples from the top and left adjacent blocks, or the pattern may be formed by additionally using information indicating whether adjacent samples correspond to zero or non-zero residuals. The pattern size can also be adapted to the block size or noise level. The prediction samples used for the template may be obtained after performing some or all of the intermediate prediction processes. The corresponding pattern in the reference image is then generated using the reconstructed samples.

[0134] Using templates, intermediate prediction based on the template is then performed. At block 1620, FRUC is performed to obtain a motion vector. At 1630, a weight and bias are obtained for the LIC. The parameters obtained from blocks 1620 and 1630 may be used to generate a prediction for the current block at block 1640. Other prediction techniques, such as OBMC, may also be used in generating the prediction. At 1650, prediction residuals for the current block may be generated by subtracting the prediction block from the original block. The prediction residuals can then be quantized, transformed and entropy encoded.

[0135] FIG. 17 depicts an exemplary method 1700 for decoding a block using template-based intermediate prediction, according to one embodiment. Method 1700 begins at initialization block 1705. During the initialization step, the decoder may access discrete neighboring block prediction samples, which may be used to generate the template, neighboring block CBFs, and quantized transform coefficients for the current block. At 1710, a template for the current block is generated based on discrete prediction samples of adjacent blocks. In order for the bitstream to be correctly decoded, the template on the decoder side must be formed in the same way as on the encoder side. The corresponding pattern in the reference image is then generated using the reconstructed samples.

[0136] Using templates, intermediate prediction based on the template is then performed. At 1720, FRUC is executed to obtain a motion vector. At 1730, a weight and bias are obtained for the LIC. The parameters obtained from steps 1720 and 1730 may be used to generate a prediction for the current block at 1740. Other intermediate prediction techniques, such as OBMC, may also be used in generating the prediction. At 1750, the current block may be decoded by adding the prediction block to the prediction residuals of the current block.

[0137] In the examples of FIG. 16 and FIG. 17, FRUC and LIC are used to generate a prediction block. Other pattern-based coding techniques may also be performed to help predict (and/or reconstruct) the current block. It should be noted that the encoding or decoding method can continue with FRUC only (no LIC), LIC only (no FRUC), both of FRUC and LIC, or more generally with any possible combination of template-based encoding methods, such as described earlier.

[0138] Various methods and other aspects described in this application can be used to modify modules, such as intermediate prediction, and / or modules (162, 104, 112, 262, 212) of the video encoder 100 and decoder 200, as shown in figure 2 and figure 3. In addition, these aspects are not limited to VVC or HEVC and may apply, for example, to other standards and recommendations and extensions of any such standards and recommendations.

[0139] Various numerical values are used in this application, such as the block size used to determine the size of the template. Specific values are provided for exemplary purposes, and the described aspects are not limited to these specific values.

[0140] Various implementations include decoding. "Decoding" used in this application may cover all or part of the processes performed, for example, in the received encoded sequence in order to produce the final output signal suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. Whether the term "decoding" is intended to refer specifically to a subset of operations, or more generally to a broader decoding process, will be clear based on the context of the specific descriptions and is assumed to be easily understood by those skilled in the art.

[0141] Various implementations use coding. Similar to the discussion of "decoding" above, "encoding" used in this application may cover all or part of the processes performed on, for example, an input video sequence in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as splitting, differential coding, transformation, quantization, and entropy coding.

[0142]

[0143] While the features and elements are described above in specific combinations, one of ordinary skill in the art will appreciate that each feature or element may be used alone or in any combination with other features and elements. Additionally, the methods described herein may be carried out in a computer program, software, or firmware residing on a computer-readable medium for execution by a computer or processor. Examples of non-volatile computer-readable media include, but are not limited to, Read Only Memory (ROM), Random Access Memory (RAM), registry, cache memory, semiconductor memory devices, magnetic media such as internal hard drives and removable drives, magneto-optical media, and optical media such as CD-ROMs and digital versatile discs (DVDs). A processor in association with software may be used to implement a video encoder, a video decoder, or both, an RF transceiver for use in a UE, WTRU, terminal, base station, RNC, or any host computer.

[0144] In addition, in the embodiments described above, processing platforms, computing systems, controls, and other devices containing processors are specified. These devices may include at least one central processing unit ("CPU") and memory. In accordance with the practices of those skilled in the art of computer programming, reference to actions and symbolic representations of operations or instructions may be performed by various CPU and memory media. Such actions and operations or instructions may be referred to as "executable", "computer-executable", or "CPU-executable".

[0145] One of ordinary skill in the art will appreciate that actions and symbolically represented operations or instructions involve the manipulation of electrical signals by the CPU. The electrical system represents data bits that can result in electrical signal conversion or reduction and storage of the data bits in memory locations within the memory system to thereby reconfigure or otherwise alter CPU operation as well as other signal processing. Memory locations where data bits are stored are physical locations that have particular electrical, magnetic, or optical properties corresponding to or representing the data bits. It should be understood that the exemplary embodiments are not limited to the aforementioned platforms or CPUs, and that other platforms and CPUs may support the methods provided.

[0146] The data bits may also be stored on a computer-readable medium, including magnetic disks, optical disks, and any other volatile (eg, random access memory ("RAM")) or non-volatile (eg, read only memory ("ROM")) system. mass memory readable by the CPU. The computer-readable medium may include co-operating or interconnected computer-readable media that exists exclusively within the processing system or distributed across a plurality of interconnected processing systems, which may be local or remote to the processing system. It should be understood that representative embodiments are not limited to the aforementioned memory media and that other platforms and memory media may support the methods described.

[0147] In an exemplary embodiment, any of the operations, processes, etc. described herein may be performed as computer-readable instructions stored on a computer-readable medium. Machine readable instructions may be executed by a processor of a mobile unit, a network element, and/or any other computing device.

[0148] The use of hardware or software is generally (but not always, as the choice between hardware and software can become significant in particular contexts) a design decision representing a trade-off between cost and efficiency. There may be various means by which the processes and/or systems and/or other technologies described herein may be carried out (for example, hardware, software and/or firmware), and the preferred means may vary depending on the context, in which processes and/or systems and/or other technologies are applied. For example, if the developer determines that speed and accuracy are of paramount importance, the developer may opt for primarily hardware and/or firmware. If flexibility is paramount, the developer may opt for a primarily software implementation. Alternatively, the developer may choose to use some combination of hardware, software, and/or firmware.

[0149] The above detailed description has set forth various embodiments of devices and/or processes through the use of block diagrams, block diagrams, and/or examples. Since such block diagrams, block diagrams and/or examples contain one or more functions and/or operations, it will be clear to experts in the field of technology that each function and/or operation in such block diagrams, block diagrams or examples can be performed, according to individually and/or in the aggregate, a wide range of hardware, software, firmware, or virtually any combination thereof. Suitable processors include, by way of example, a general purpose processor, an application specific processor, a standard processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, application specific integrated circuits (ASICs) , custom ASSPs, Field Programmable Gate Arrays (FPGAs), any other type of integrated circuit (IC), and/or state machine.

[0150] Although features and elements are provided above in specific combinations, one of ordinary skill in the art will appreciate that each feature or element may be used alone or in any combination with other features and elements. The present disclosure should not be limited in the sense of the specific embodiments described in this application, which are intended to be illustrative of various aspects. Many modifications and variations may be made without departing from its spirit and scope, which will be apparent to those skilled in the art. No element, act or instruction used in the description of this application should be construed as being critical or necessary to the invention unless they are expressly provided as such. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those listed here, will be apparent to those skilled in the art from the above descriptions. Such modifications and variations are intended to be within the scope of the appended claims. The present disclosure is to be limited only by the terms of the appended claims together with the full scope of the equivalents for which such claims are the basis. It should be understood that this disclosure is not limited to specific methods or systems.

[0151] It should also be understood that the terminology used here is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0152] In specific representative embodiments, several parts of the invention described herein may be implemented by application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), and/or other integrated formats. However, those skilled in the art will appreciate that certain aspects of the embodiments disclosed herein may, in whole or in part, equivalently be implemented in integrated circuits as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (for example, as one or more programs running on one or more microprocessors), as firmware, or in virtually any combination of the foregoing, and that circuit design and/or coding for the software and/or firmware would be within the skill of one of ordinary skill in the art in light of this disclosure. Additionally, those skilled in the art will appreciate that the mechanisms of the invention described herein may be distributed as a software product in a variety of different forms and that an illustrative embodiment of the invention described herein is applicable regardless of the particular type of signal carrier used to actually effect the distribution. Examples of signal media include, but are not limited to, the following: a recordable type media such as floppy disk, hard disk drive, CD, DVD, magnetic tape, computer memory, etc., and transmission type media such as digital and/or analog communication media (eg, fiber optic cable, waveguide, wired link, wireless link, etc.).

[0153] The invention described herein sometimes illustrates various components contained in or combined with various other components. It should be understood that such depicted architectures are merely examples and that many other architectures may actually be implemented that achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is essentially "linked" so that the desired functionality can be achieved. Thus, any two components here that are combined to achieve specific functionality can be considered to be "linked" to each other so that the desired functionality is achieved, regardless of architectures or intermediate components. Likewise, any two components so associated may also be considered to be "operably connected" or "operably connected" to each other to achieve the desired functionality, and any two components capable of being so associated may also be considered to be "operably connected" to each other. another to achieve the desired functionality. Specific examples of operably interoperable components include, but are not limited to, physically compatible components and/or physically interoperable components and/or wirelessly interoperable components and/or wirelessly interoperable components and/or logically interoperable components, and/or components with the possibility of logical interaction.

[0154] With respect to the use of essentially any plural and/or singular terms herein, those skilled in the art may translate them from the plural to the singular and/or from the singular to the plural according to context and/or application. Various singular/plural conversions may be explicitly suggested here for the sake of clarity.

[0155] It will be understood by those skilled in the art that, in general, terms used herein and especially in the appended claims (e.g., the body text of the appended claims) are generally meant to be "open" terms (e.g., the term " including" shall be interpreted as "including but not limited to the included", the term "having" shall be interpreted as "having at least", the term "includes" shall be interpreted as "includes but is not limited to the included " etc.). Those of skill in the art will further appreciate that if a particular number of the stated statement of a claim is intended, that assumption will be explicitly listed in the claim, and in the absence of such a listing, there is no such assumption. For example, where only one element is intended, the term "single" or similar may be used. As an aid to understanding, the following appended claims and/or descriptions herein may contain the use of the provisional phrases "at least one" and "one or more" to introduce a statement of claims. However, the use of such phrases should not be construed as implying that a singular opening clause restricts any particular clause containing such an initial clause to embodiments containing only one such statement, even when the same clause includes the provisional phrases "one or more" or "at least one" and a singular formulation (eg, a singular formulation should be interpreted to mean "at least one" or "one or more"). Additionally, even if a specific amount of provisional clause statement is explicitly stated, those skilled in the art will appreciate that such statement is to be interpreted as meaning at least the number stated (e.g., a simple statement of "two statements" without other modifiers means at least two statements or two or more statements). Also, where a symbol similar to "at least one of A, B, and C, etc." is used, in general, such a construction is meant in the sense that one skilled in the art will understand the convention (e.g., "a system having at least one of A, B, and C" would include, but not be limited to, systems, that have only A, only B, only C, A and B together, A and C together, B and C together and/or A, B and C together, etc.). Where a symbol similar to "at least one of A, B, or C, etc." is used, in general, such a construction is meant in the sense in which a person skilled in the art will understand the convention (for example, "a system having at least one of A, B, or C" would include, but not be limited to, systems, that have only A, only B, only C, A and B together, A and C together, B and C together and/or A, B and C together, etc.). It will be further understood by those skilled in the art that virtually any separating word and/or phrase representing two or more alternative terms, either in the description, or in the claims, or in the drawings, should be understood to imply the possibility of including one of the terms, any of terms or both terms. For example, the phrase "A or B" would be understood to include the possibilities "A" or "B" or "A and B". Additionally, the term "any of" followed by a listing of a plurality of elements and/or a plurality of element categories used herein is intended to include "any of", "any combination", "any plurality" and/or "any combination of plurality of "elements and/or categories of elements alone or in combination with other elements and/or other categories of elements. In addition, as used herein, the term "set" or "group" is intended to include any number of elements, including zero. Additionally, as used herein, the term "amount" is intended to include any number including zero.

[0156] Additionally, when features or aspects of a disclosure are described in terms of Markush groups, those skilled in the art will appreciate that the disclosure is also thus described in terms of any individual member or subgroup of members of a Markush group.

[0157] As will be understood by one of skill in the art, for any and all purposes, such as in terms of providing a written description, all ranges disclosed herein also cover any and all of their possible subranges and combinations of subranges. Any range mentioned can be readily understood to be sufficient to describe and enable the same range to be partitioned into at least equal halves, thirds, quarters, fifths, tenths, etc. As a non-limiting example, each range discussed herein can be easily broken down into lower third, middle third, and upper third, etc. As will also be appreciated by one skilled in the art, all terms such as "up to", "at least", "greater than", " less than", etc., include the named number and refer to ranges that may subsequently be sub-ranged as discussed above. Finally, as one of skill in the art will appreciate, the range includes every single element. Thus, for example, a group having 1-3 cells refers to groups having 1, 2, or 3 cells. Similarly, a group having 1-5 cells refers to groups having 1, 2, 3, 4, or 5 cells, and so on.

[0158] In addition, the claims should not be read as being limited to the order or elements provided, unless otherwise indicated. Additionally, use of the term "agent for" in any claim is intended to invoke Specification 35 U.S.C. §112, ¶ 6 or the remedy-plus-function formula format, and any claim without the term "a remedy for" is not meant to be so.

[0159] It is contemplated that the systems may be implemented in software on microprocessors/mainframe computers (not shown). In particular embodiments, one or more of the functions of the various components may be implemented in software that controls the general purpose computer.

[0160] Additionally, although the invention is illustrated and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Indeed, various modifications may be made in detail within the scope and range of equivalents of the claims and without departing from the scope of the invention.

Claims (62)

1. A method for decoding a video containing a current image and at least a first reference image, the method comprising the steps of, for at least the current block in the current image: generate a corresponding predicted value for each discrete sample in the area of the template adjacent to the current block; perform intermediate (inter) prediction based on the template for the current block using (1) predicted values of at least a subset of samples in the template region and (2) corresponding sample values in at least one region of the reference pattern in at least the first reference image, wherein the intermediate prediction based on the template includes any of the steps in which: determining at least one template-based intermediate prediction parameter, wherein the template-based intermediate prediction is performed using the determined template-based intermediate prediction parameter, sorting the plurality of available template-based intermediate prediction parameters, and performing the template-based intermediate prediction for the current block using the selected one of the available template-based intermediate prediction parameters, wherein an index indicating the selected one of the available template-based intermediate prediction parameters is signaled in bit stream, and performing a pattern-based selection of at least two potential candidate motion vector predictors and performing a local pattern-based search on the selected potential candidate motion vector predictors to select a motion vector predictor; And adding a corresponding non-zero residual value to at least one of the predicted values in the template region to generate a corresponding reconstructed sample value. 2. The method of claim 1, wherein the corresponding predicted value for each sample in the pattern region is directly from the motion compensated prediction. 3. The method of claim 1, wherein a respective predicted value for each sample in the pattern region is generated using motion compensated prediction and overlay block motion compensation (OBMC). 4. The method of claim 1, further comprising selecting a subset of samples in the pattern region such that it includes only samples having zero residuals. 5. The method of claim 4, wherein said selection of a subset of discrete samples selects only the discrete samples in a respective block having a flag indicating that no remainder is encoded for that respective block. 6. The method of claim 4, wherein the template region contains at least two sub-regions, wherein the samples in the sub-region are selected only if all of the samples in that sub-region have zero residuals. 7. The method of claim 1, wherein restoring sample values in the template region and performing intermediate prediction based on the template for the current block are performed in parallel. 8. The method of claim 1, wherein at least one template-based intermediate prediction parameter for the current block is determined by applying weights associated with predicted values of at least a subset of discrete samples in the template region, wherein the weights are based on absolute residuals. values of the corresponding discrete samples in the template area, with larger weights for lower residual absolute values. 9. The method of claim 8, wherein the first weighting factor is used for discrete samples with non-zero absolute residual values, and the second weighting factor is used for discrete samples with absolute residual values equal to zero, and the second weighting factor is higher than the first weighting factor. 10. The method of claim 1, further comprising adjusting the predicted values of at least a subset of the samples in the template region, the adjusted predicted values being used in performing intermediate prediction based on the template for the current block. 11. The method of claim 10, wherein the adjustment comprises adding a residual DC prediction component to the predicted values. 12. The method of claim 10, wherein the predicted value adjustment is performed only for the values of discrete samples in blocks encoded using either DCT-II or DCT-V. 13. A method for encoding a video containing a current picture and at least a first reference picture, the method comprising, for at least the current block in the current picture: generate a corresponding predicted value for each discrete sample in the area of the template adjacent to the current block; perform intermediate (inter) prediction based on the template for the current block using (1) predicted values of at least a subset of samples in the template region and (2) corresponding sample values in at least one region of the reference pattern in at least the first reference image, wherein the intermediate prediction based on the template includes any of the steps in which: determining at least one template-based intermediate prediction parameter, wherein the template-based intermediate prediction is performed using the determined template-based intermediate prediction parameter, sorting the plurality of available template-based intermediate prediction parameters, and performing the template-based intermediate prediction for the current block using the selected one of the available template-based intermediate prediction parameters, wherein an index indicating the selected one of the available template-based intermediate prediction parameters is signaled in bit stream, and performing a pattern-based selection of at least two potential candidate motion vector predictors and performing a local pattern-based search on the selected potential candidate motion vector predictors to select a motion vector predictor; And adding a corresponding non-zero residual value to at least one of the predicted values in the template region to generate a corresponding reconstructed sample value. 14. The method of claim 13, wherein the respective predicted value for each discrete sample in the pattern region is directly from the motion compensated prediction. 15. The method of claim 13, wherein a respective predicted value for each sample in the pattern region is generated using motion compensated prediction and overlay block motion compensation (OBMC). 16. The method of claim 13, further comprising selecting a subset of samples in the pattern region such that it includes only samples having zero residuals. 17. The method of claim 16, wherein said selection of a subset of discrete samples selects only the discrete samples in the respective block having a flag indicating that no remainder is encoded for that respective block. 18. The method of claim 16, wherein the template region contains at least two sub-regions, wherein the samples in the sub-region are selected only if all of the samples in that sub-region have zero residuals. 19. The method of claim 13, wherein restoring sample values in the pattern region and performing intermediate prediction based on the pattern for the current block are performed in parallel. 20. The method of claim 13, wherein at least one template-based intermediate prediction parameter for the current block is determined by applying weights associated with predicted values of at least a subset of samples in the template region, wherein the weights are based on absolute residuals. values of the corresponding discrete samples in the template area, with larger weights for lower residual absolute values. 21. The method of claim 20, wherein the first weighting factor is used for discrete samples with non-zero absolute residual values, and the second weighting factor is used for discrete samples with absolute residual values equal to zero, and the second weighting factor is higher than the first weighting factor. 22. The method of claim 13, further comprising adjusting the predicted values of at least a subset of samples in the template region, the adjusted predicted values being used in performing intermediate prediction based on the template for the current block. 23. The method of claim 22, wherein the adjustment comprises adding a residual DC prediction component to the predicted values. 24. The method of claim 22, wherein the predicted value adjustment is performed only for the values of discrete samples in blocks encoded using either DCT-II or DCT-V. 25. Device for video decoding, containing a processor and memory, configured to: generate a corresponding predicted value for each discrete sample in the template area adjacent to the current block in the current image; perform intermediate (inter) prediction based on the template for the current block using (1) predicted values of at least a subset of samples in the template region and (2) corresponding sample values in at least one region of the reference pattern in at least the first reference image, wherein the intermediate prediction based on the template includes any of: determining at least one template-based intermediate prediction parameter, wherein the template-based intermediate prediction is performed using the determined template-based intermediate prediction parameter, sorting a plurality of available pattern-based intermediate prediction parameters; and performing pattern-based intermediate prediction for the current block using the selected one of the available pattern-based intermediate prediction parameters, wherein an index indicating the selected one of the available pattern-based intermediate prediction parameters is signaled in a bitwise stream, and performing a pattern-based selection of at least two potential candidate motion vector predictors and performing a local pattern-based search on the selected potential candidate motion vector predictors to select a motion vector predictor; And add a non-zero residual value to at least one of the predicted values in the template region to restore the corresponding value of the recovered sample. 26. The apparatus of claim 25, wherein the apparatus is further configured to select a subset of the samples in the pattern region to include only the samples having zero residuals. 27. The device according to claim 25, wherein restoring the values of discrete samples in the template area and performing intermediate prediction based on the template for the current block are performed in parallel. 28. The apparatus of claim 25, wherein at least one template-based intermediate prediction parameter for the current block is determined by applying weights associated with predicted values of at least a subset of discrete samples in the template region, the weights being based on absolute values. residuals of the corresponding discrete samples in the template area, with larger weighting factors for lower absolute values of the residuals. 29. The device according to claim 28, wherein the first weighting factor is used for discrete samples with non-zero absolute residual values, and the second weighting factor is used for discrete samples with absolute residual values equal to zero, and the second weighting factor is higher than the first weighting factor. 30. The apparatus of claim 25, wherein the apparatus is further configured to adjust predicted values of at least a subset of discrete samples in the template region, the adjusted predicted values being used in performing intermediate prediction based on the template for the current block. 31. The apparatus of claim 30, wherein the adjustment comprises adding a DC prediction residual component to the predicted values. 32. Device for video encoding, containing a processor and memory, configured to: generate a corresponding predicted value for each discrete sample in the template area adjacent to the current block in the current image; perform intermediate (inter) prediction based on the template for the current block using (1) predicted values of at least a subset of samples in the template region and (2) corresponding sample values in at least one region of the reference pattern in at least the first reference image, wherein the intermediate prediction based on the template includes any of: determining at least one template-based intermediate prediction parameter, wherein the template-based intermediate prediction is performed using the determined template-based intermediate prediction parameter, sorting a plurality of available pattern-based intermediate prediction parameters; and performing pattern-based intermediate prediction for the current block using the selected one of the available pattern-based intermediate prediction parameters, wherein an index indicating the selected one of the available pattern-based intermediate prediction parameters is signaled in a bitwise stream, and performing a pattern-based selection of at least two potential candidate motion vector predictors and performing a local pattern-based search on the selected potential candidate motion vector predictors to select a motion vector predictor; And add a non-zero residual value to at least one of the predicted values in the template region to restore the corresponding value of the recovered sample. 33. The apparatus of claim 32, wherein the apparatus is further configured to select a subset of the samples in the pattern region to include only the samples having zero residuals. 34. The device according to claim 32, wherein restoring the values of discrete samples in the template area and performing intermediate prediction based on the template for the current block are performed in parallel. 35. The apparatus of claim 32, wherein at least one template-based intermediate prediction parameter for the current block is determined by applying weights associated with predicted values of at least a subset of discrete samples in the template region, the weights being based on absolute values. residuals of the corresponding discrete samples in the template area, with larger weighting factors for lower absolute values of the residuals. 36. The apparatus of claim 35, wherein the first weighting factor is used for discrete samples with non-zero absolute residual values, and the second weighting factor is used for discrete samples with absolute residual values equal to zero, and the second weighting factor is higher than the first weighting factor. 37. The apparatus of claim 32, wherein the apparatus is further configured to adjust predicted values of at least a subset of discrete samples in the template region, the adjusted predicted values being used in performing intermediate prediction based on the template for the current block. 38. The apparatus of claim 37, wherein the adjustment comprises adding a residual DC prediction component to the predicted values.

Images