TW202416714A - Intra prediction mode improvements based on available reference samples - Google Patents

Abstract

In one implementation, the wide-angle process can be modified to disallow Intra Prediction Modes (IPMs) that would make use of padded reference samples, especially when other IPMs would make use of available reference samples. More generally, depending on whether the reference samples are available or not, certain intra prediction modes can be removed or added, or the MPMs can be reordered. The signaling process can then be modified to handle additional modes or removed modes, especially if this can lead to different numbers of IPMs. The storage and propagation of intra modes can also be modified to handle the change made to IPMs.

Description

Improvements to the intra-frame prediction model based on available reference samples

Embodiments of the present invention generally relate to a method and apparatus for intra-frame prediction in video encoding and decoding.

To achieve high compression efficiency, image and video coding schemes usually use prediction and transformation to exploit spatial and temporal redundancy in video content. In general, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame image correlation, and then the difference between the original block and the predicted block (usually expressed as prediction error or prediction residue) is transformed, quantized, and entropy encoded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transformation, and prediction.

According to one embodiment, a video decoding method is proposed, which includes: identifying the availability of one or more reference samples for a block to be decoded in an image; obtaining an intra-frame prediction mode for the block in response to the availability of the one or more reference samples; obtaining an intra-frame prediction mode from the intra-frame prediction mode; and performing intra-frame prediction of the block to be decoded based on the intra-frame prediction mode of the block to form a prediction block for the block.

According to another embodiment, a video coding method is proposed, which includes: identifying the availability of one or more reference samples for a block to be encoded in an image; obtaining a set of intra-frame prediction modes for the block in response to the availability of the one or more reference samples; selecting an intra-frame prediction mode from the set of intra-frame prediction modes; and performing intra-frame prediction of the block to be encoded based on the intra-frame prediction mode of the block to form a prediction block for the block.

According to another embodiment, a device for video decoding is provided, which includes one or more processors, wherein the one or more processors are configured to: identify the availability of one or more reference samples for a block to be decoded in an image; obtain an intra-frame prediction mode for the block in response to the availability of the one or more reference samples; obtain an intra-frame prediction mode from the intra-frame prediction mode; and perform intra-frame prediction of the block to be decoded based on the intra-frame prediction mode of the block to form a prediction block for the block.

According to another embodiment, a device for video encoding is proposed, which includes one or more processors, wherein the one or more processors are configured to: identify the availability of one or more reference samples for a block to be encoded in an image; obtain a group of intra-frame prediction modes for the block in response to the availability of the one or more reference samples; select an intra-frame prediction mode from the group of intra-frame prediction modes; and perform intra-frame prediction of the block to be encoded based on the intra-frame prediction mode of the block to form a prediction block for the block.

One or more embodiments also provide a computer program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform an encoding method or a decoding method according to any of the embodiments described herein. One or more of the embodiments also provide a computer-readable storage medium having stored thereon instructions for video encoding or decoding according to the methods described herein.

One or more embodiments also provide a computer-readable storage medium having video data generated according to the above method stored thereon. One or more embodiments also provide a method and apparatus for transmitting or receiving video data generated according to the method described herein.

FIG. 1 is a block diagram showing an example of a system in which various aspects and embodiments may be implemented. System 100 may be embodied as a device including various components described below and configured to execute one or more of the aspects described in the present application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptops, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The components of system 100 may be embodied singly or in combination in a single integrated circuit, multiple ICs, and/or discrete components. For example, in at least one embodiment, the processing and encoder/decoder components of system 100 are distributed across multiple ICs and/or discrete components. In various embodiments, the system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input ports and/or output ports. In various embodiments, the system 100 is configured to implement one or more of the aspects described in the present application.

System 100 includes at least one processor 110 configured to execute instructions loaded therein for implementing various aspects, such as those described in the present application. Processor 110 may include embedded memory, input/output interfaces, and various other circuit systems known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and/or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and/or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drive, and/or optical disk drive. As non-limiting examples, storage device 140 may include internal storage devices, attached storage devices, and/or network accessible storage devices.

The system 100 includes a codec module 130 that is configured to, for example, process data to provide encoded video or decoded video, and the codec module 130 may include its own processor and memory. The codec module 130 represents a module(s) that may be included in a device to perform encoding and/or decoding functions. As is known, a device may include one or both encoding and decoding modules. Additionally, as known to those of ordinary skill in the art, the codec module 130 may be implemented as a separate component of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software.

Program code to be loaded onto the processor 110 or the encoder/decoder 130 to execute various aspects described in the present application may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder/decoder module 130 may store one or more of various items during execution of the procedures described in the present application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bit streams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

In some embodiments, memory inside the processor 110 and/or the encoder/decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be either the processor 110 or the encoder/decoder module 130) is used for one or more of these functions. The external memory can be the memory 120 and/or the storage device 140, such as dynamic volatile memory and/or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, fast external dynamic volatile memory (such as RAM) is used as working memory for video encoding and decoding operations, such as for MPEG-2, HEVC, or VVC.

Input to the components of system 100 may be provided by various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives RF signals transmitted through the air, such as by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and/or (iv) an HDMI input terminal.

In various embodiments, the input device of block 105 has associated respective input processing elements as are known in the art. For example, the RF portion may be associated with elements adapted to (i) select a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-convert the selected signal, (iii) again band-limit to a narrower frequency band to select a signal band which may be referred to as a channel in some embodiments, (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF portion of various embodiments includes one or more components to perform such functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, down-converters, demodulators, error correctors, and demultiplexers. The RF portion may include a tuner that performs various such functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to a baseband. In one set-top box embodiment, the RF portion and its associated input processing elements receive RF signals transmitted through a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments reconfigure the order of the above (and other) components, remove some of these components, and/or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

Additionally, the USB and/or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices across the USB and/or HDMI connections. It will be appreciated that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, in a separate input processing IC or in the processor 110 as desired. Similarly, aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 110 as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, the processor 110 and the encoder/decoder 130, which operate in combination with memory and storage elements to process the data stream as desired for presentation on an output device.

The various components of system 100 may be provided within an integrated housing, within which the various components may interconnect and transmit data therebetween using a suitable connection arrangement 115 (e.g., an internal bus as known in the art, including an I2C bus, wiring, and a printed circuit board).

The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented in, for example, wired and/or wireless media.

In various embodiments, data is streamed to the system 100 using a Wi-Fi network (such as IEEE 802.11). The Wi-Fi signals of these embodiments are received via a communication channel 190 and a communication interface 150 adapted for Wi-Fi communication. The communication channel 190 of these embodiments is generally connected to an access point or router that provides access to an external network (including the Internet) for allowing streaming applications and other over-the-top communications. Other embodiments use a set-top box that delivers data via an HDMI connection of the input block 105 to provide the streamed data to the system 100. Still other embodiments use an RF connection of the input block 105 to provide the streamed data to the system 100.

The system 100 can provide output signals to various output devices, including a display 165, speakers 175, and other peripheral devices 185. In various examples of embodiments, the other peripheral devices 185 include one or more of a stand-alone DVR, an optical disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 100. In various embodiments, control signals are communicated between the system 100 and the display 165, speakers 175, or other peripheral devices 185 using communications such as AV.Link, CEC, or other communications protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output device may be connected to the system 100 via the communication interface 150 using the communication channel 190. The display 165 and the speaker 175 may be integrated into a single unit with other components of the system 100 in an electronic device (e.g., a television). In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.

For example, if the RF portion of input 105 is part of a separate set-top box, the display 165 and speakers 175 may alternatively be separate from one or more of the other components. In various embodiments where the display 165 and speakers 175 are external components, the output signals may be provided via dedicated output connections (including, for example, an HDMI port, a USB port, or a COMP output).

FIG2 illustrates an exemplary video encoder 200, such as a VVC (Versatile Video Coding) encoder. FIG2 may also illustrate an encoder that improves the VVC standard or an encoder that uses techniques similar to VVC.

In this application, the terms "reconstructed" and "decoded" are used interchangeably, the terms "encoded" and "coded" are used interchangeably, and the terms "image", "picture", and "frame" are used interchangeably. Usually, but not necessarily, the term "reconstructed" is used at the encoder side, and "encoded" is used at the decoder side.

Prior to encoding, the video sequence may undergo pre-encoding processing (201) such as applying a color conversion to the input color image (e.g., from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input image components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

In a coder 200, an image is encoded by coder elements as described below. The image to be encoded is divided (202) and processed in units such as CUs (coding units). Each unit is encoded using, for example, either intra or inter mode. When a unit is encoded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and compensation (270) are performed. The coder decides (205) which of the intra or inter modes is to be used for encoding the unit and indicates the intra/inter decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (210) the predicted block from the original image block.

The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with motion vectors and other syntax elements (such as image segmentation information), are entropy encoded (245) to output a bitstream. As a non-limiting example, context-based adaptive binary arithmetic coding (CABAC) can be used to encode the syntax elements into the bitstream.

The encoder can skip the conversion and apply quantization directly to the unconverted residual signal. The encoder can skip both the conversion and quantization, that is, the residue is directly encoded without applying a conversion or quantization process.

The encoder decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residue. The decoded prediction residue and the prediction block are combined (255) to reconstruct the image block. An in-loop filter 265 is applied to the reconstructed image to perform, for example, deblocking/SAO (Sample Adaptive Offset)/ALF (Adaptive Loop Filter) filtering to reduce coding artifacts. The filtered image is stored in a reference image buffer (280).

FIG3 shows a block diagram of an example video decoder 300. In decoder 300, a bit stream is decoded by decoder elements as described below. Video decoder 300 generally performs a decoding phase that is the inverse of the encoding phase described in FIG2. Encoder 200 also typically performs video decoding as part of encoding video data.

Specifically, the input to the decoder includes a video bitstream that may be generated by the video encoder 200. The bitstream is first entropy decoded 330 to obtain transform coefficients, prediction modes, motion vectors, and other encoded information. The image segmentation information indicates how the image is segmented. The decoder can therefore divide (335) the image based on the decoded image segmentation information. The transform coefficients are dequantized (340) and inversely transformed (350) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (355) to reconstruct the image block. The predicted block can be obtained (370) from intra-frame prediction (360) or motion compensated prediction (i.e., inter-frame prediction) (375). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored at a reference image buffer (380). It should be noted that for a given image, the contents of the reference image buffer 380 on the decoder 300 side are identical to the contents of the reference image buffer 280 on the encoder 200 side for the same image.

The decoded image may be further subjected to post-decoding processing (385), such as color inversion (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping of the remapping process performed in the pre-coding process (201). The post-decoding process may use metadata derived in the pre-coding process and signaled in the bitstream. Intra-frame prediction and reference sample substitution

The intra-frame prediction process in HEVC and VVC consists of three steps: •     reference sample generation; •     intra-frame sample prediction; and •     post-processing of the predicted samples.

The reference sample generation process is illustrated in FIG4 . The pixel value at common coordinates (x, y) is denoted by P(x, y). The reference sample ref[] is also known to be L-shaped. For a prediction unit (PU) of size NxN, the top row of (2N + 2*refIdx) decoded samples is formed by previously reconstructed top and top-right pixels. Similarly, the left row of (2N + 2*refIdx) samples is formed by reconstructed left and bottom-left pixels. In VVC, reference lines and sample rows can be at a distance of more than one sample (d = refIdx) from the current block, as depicted in FIG4 . The signaling index “mrlIdx” is used to indicate what value of “d” should be used.

Corner pixels at the upper left position are also used to fill in the gaps between the upper row and left row references. In FIGS. 5A and 5B , the dashed areas correspond to unavailable areas of the image (e.g., beyond the border or not yet reconstructed), and missing reference samples are represented by dotted lines. As shown in FIG. 6 , if some samples at the top or left are unavailable ( 610 ) because the corresponding CU is not in the same slice, or the current CU is at the frame boundary (e.g., as shown in FIG. 5A ), or the current CU is at the bottom right after a quadtree split (as shown in FIG. 5B ), a method called reference sample replacement is performed, in which the missing samples are copied from available samples in a clockwise and counterclockwise direction ( 630 ). Such replicated samples are also referred to as “padded reference samples”, and when the reconstructed samples are used as reference samples, they are also referred to as “non-padded reference samples”.

If reconstructed top/left reference samples are available, the reconstructed reference samples are copied (620) to the reference sample buffer. After the reference sample replacement process, intra-frame sample prediction is performed (640). Then, depending on the current CU size and prediction mode, the reference samples are filtered using a specified filter.

Intra-frame sample prediction involves predicting the pixels of the target CU based on reference samples. There are different prediction modes. Planar and DC prediction modes are used to predict smooth and gradually changing areas, where angular (defined as angles from 45 degrees to −135 degrees in the clockwise direction) prediction modes are used to capture different directional structures. For square blocks, HEVC supports 33 directional prediction modes, which are indexed from 2 to 34. These prediction modes correspond to different prediction directions, as shown in Figure 7A. The numbers in the figure represent the prediction mode index associated with the corresponding direction. Modes 2 to 17 indicate horizontal prediction (H-26 to H+32), while modes 18 to 34 indicate vertical prediction (V-32 to V+32).

In VVC, there are 65 angular prediction modes, which correspond to the 33 angular directions defined in HEVC, and the other 32 directions each correspond to the direction in the middle of the adjacent pair, as shown in Figure 7B. For a square block, a mode less than 34 indicates horizontal prediction, and a mode greater than 34 indicates vertical prediction.

As mentioned above, the angular direction can be distinguished as vertical or horizontal. As shown in Figure 7C, the prediction mode in the horizontal direction uses only the left reference samples, or some left and some top reference samples. Similarly, the prediction mode in the vertical direction uses only the top reference samples, or some top and some left reference samples. The positive horizontal direction uses only the left reference samples for prediction. Similarly, the positive vertical direction uses only the top reference samples for prediction. Negative horizontal and vertical directions use reference samples from both the left and top for prediction.

In VVC, for non-square blocks, the non-allowed conventional directional intra prediction mode is replaced with a wide angle intra prediction mode, as shown in Figures 8A, 8B, and 8C. Table 1 lists the permuted intra modes and added wide angle modes for different aspect ratios. It should be noted that a block ratio of 32 is included in Table 1, but it cannot be used in practice because the segmentation does not allow W/H=32 or H/W=32. In Figure 9, the dotted line indicates the wide angle intra prediction mode (Wide Angle Intra Prediction Mode, WAIP). Note that in ECM, the indexes -1 to -14 presented in Figure 9 are remapped to become from 1 to -12 so that the angle mode index is continuous. Mode -15 (remapped to -13) and mode 81 are also not present in Figure 9 because there are no block sizes that can use them, but these modes are handled by the reference software. Table 1 Aspect ratio W/H Permuted In-Frame Prediction Mode Index Angle mode added 2 2 to 7 67 to 72 4 2 to 11 67 to 76 8 2 to 13 67 to 78 16 2 to 15 67 to 80 32 2 to 16 67 to 81 1 without without 1/2 61 to 66 -4 to 1 1/4 57 to 66 -8 to 1 1/8 55 to 66 -10 to 1 1/16 53 to 66 -12 to 1 1/32 52 to 66 -13 to 1

For a given angular prediction mode, the predictor samples on the reference array are copied along the corresponding direction within the target PU. Some predictor samples may have integer positions, in which case they match the corresponding reference samples; other predictor positions will have fractional parts indicating that their positions will fall between two reference samples. In the latter case, the closest reference samples are used to interpolate the predictor samples (post-processing of the prediction samples). In HEVC, linear interpolation of the two closest reference samples is performed to calculate the predictor sample values. In VVC, to interpolate the predictor samples, a 4-tap filter fT[] is used, which is selected depending on the intra-frame mode direction.

In addition to the directional mode, the DC mode fills the prediction with the average of L-shaped samples (except for the rectangular CU which uses the average of the longer side reference samples), and the planar mode spatially interpolates the reference samples as shown in Figure 10. Intra-frame prediction mode coding

Since there are multiple available intra prediction modes, the decoder needs mode information to form a prediction for an intra-coded CU. The encoder encodes the mode information using one or more Most Probable Mode (MPM) groups. For example, ECM-5.0 (Enhanced Compression Model 5.0) uses a first MPM list (with 6 MPMs) and a secondary MPM list (with 16 MPMs). The first MPM list is constructed by adding candidate intra prediction mode indices in sequence based on the intra prediction mode index used after the current brightness coding block, where the first MPM index is reserved for planar mode. The neighboring indices added are left neighbor, above, bottom left, top right, and top left.

The secondary MPM list is constructed by first adding the indices of the first and second DIMD (Decoder-side Intra Mode Derivation) modes of the current luma coding block; then adding the increasing and decreasing indices of the first angle MPM (mpm[1]+1, mpm[1]-1, mpm[1]+2, mpm[1]-2, mpm[1]+3, mpm[1]-3, mpm[1]+4, mpm[1]-4, mpm[2]+1, mpm[2]-1, mpm[2]+2, mpm[2]-2, mpm[2]+3, mpm[2]-3, mpm[2]+4, mpm[2]-4, etc.) in such a way that no redundant mode indices exist in the MPM list (neither in the primary MPM list nor in the secondary list). If the selected intra prediction mode does not belong to the first and secondary MPM lists, the remaining intra prediction modes (IPM) are encoded using truncated binary coding for 45 symbols.

In one embodiment, it is proposed to modify the intra-frame prediction process so that the wide-angle intra-frame prediction mode is selected not only based on the block aspect ratio, but also based on the availability of reference samples. Therefore, IPMs that would normally use mostly padded reference samples will not be allowed, while IPMs that would normally not be allowed but would actually use non-padded reference samples will be allowed. The signaling is then modified to use these changes. This can be done by adding a context that uses this change to the CABAC encoded bits (bins), or by changing the code to handle added modes or removed modes when the total number of modes is different from the original 67 indices.

The propagation of intra-frame modes is also modified to propagate more accurate mode values and to handle the case where neither the neighboring intra-frame modes nor their 180° counterparts are available for the current block. Available intra-frame modes are selected based on the availability of neighboring reference samples .

FIG. 11 illustrates which samples are tested for availability to know which IPMs use fill reference samples according to one embodiment. Solid lines (1110, 1115) indicate the angle IPMs (modes 12 to 76) used by ECM-6.0 for a PU with a W/H ratio of 4. Dashed lines (1120, 1125) indicate the modes (modes 8 to 72) used by ECM-6.0 for a PU with a W/H ratio of 2. Dotted lines (1130, 1135) indicate the modes (modes 2 to 66) used by ECM-6.0 for a PU with a W/H ratio of 1.

As shown in FIG11 , if a set of modes wishes to use only available reference samples for intra-frame prediction, it may be necessary to determine which samples are required to be available for this set of modes for a particular block width/height ratio. For example, as shown in FIG11 , if reference sample A is not available (then the samples to the left of A may also not be available), at least one mode among modes 73 to 76 (i.e., the mode for W/H = 4 but not for W/H < 4) will use at least one padding sample. If reference sample B is not available (then all samples between A and B are not available), all modes 73 to 76 will use at least one padding reference sample.

When the reference samples used for an intra-frame prediction mode are determined to be unavailable, we can decide to remove this intra-frame prediction mode because the prediction will have poor quality. In addition, if the reference samples used for these intra-frame prediction modes are all available, we can check whether we can add back some other intra-frame prediction modes. It should be noted that some of the intra-frame prediction modes we propose to add back are modes removed by WAIP. But because these intra-frame prediction modes use reconstructed reference samples (rather than filled reference samples), the prediction quality can still be better. For example, as shown in Figure 11, if X is available (then all reference samples above X are available), then modes 8 to 11 (i.e., modes used for W/H = 2 but not for W/H > 2) will use non-filled reference samples and can be added. If Y is available (then all reference samples above Y are available), then modes 2 to 7 (i.e., modes for W/H = 1 but not for W/H > 1) will use non-filled reference samples and can be added. Note that we can perform intra-frame mode removal without adding back other intra-frame prediction modes, or add some intra-frame prediction modes without removing intra-frame modes.

We can also use the intra-frame prediction modes listed in Table 1 for removing/adding intra-frame prediction modes. In one embodiment, the allowed modes are based on a set of modes designed for another block size. For example, depending on the available adjacent reference samples, a square block may use a mode designed for W/H = 2, as shown in Table 1. It should be noted here that some of the intra-frame modes in the resulting intra-frame patterns may still use padding reference samples, but generally use fewer padding samples than the original intra-frame prediction mode. In the following, several examples are described in detail.

Let ref[0; 0] be the position of the top left sample of the PU to be encoded/decoded, let D be the distance of the reference line given by mrlIdx, and let W and H be the width and height of the current PU. -      If W ＞ H, ○    If the upper samples from ref[-1-D; -1-D] up to ref[2*W+W*D/H-1; -1-D] are available (i.e., non-filling samples), the set of patterns defined for aspect ratio W/H can be used, for example, pattern [E001] used in ECM-6.0 as described in Table 1; ○    Otherwise (if at least one of these samples is not available), if all left samples from ref[-1-D; -1-D] up to ref[-1-D; H+W/2 +H*D/(W/2)- 1] are available, the set of patterns defined for aspect ratio (W/2)/H can be used. Otherwise, (if at least one of these left samples is also not available), the set of patterns defined for aspect ratio W/H can be used. [E002] -      Otherwise, if W < H, ○     If the left samples from ref[-1-D; -1-D] up to ref[-1-D; 2*H+H*D/W-1] are available (i.e., non-filling samples), the set of patterns defined for aspect ratio W/H can be used, for example, the pattern used in ECM-6.0. [E003] ○     Otherwise (if at least one of these samples is not available), if all upper samples from ref[-1-D; -1-D] up to ref[W+H/2+W*D/(H/2)-1;-1-D] are available, the set of patterns defined for aspect ratio W/(H/2) can be used. Otherwise (if at least one of the upper samples is also unavailable), the set of modes defined for the aspect ratio W/H can be used [E004].

The conditions in E001 and E003 check if all samples are available for WAIP mode. If at least one sample is not available, then based on the subsequent conditions, the conventional ECM mode (i.e., the mode defined in Table 1) may not be used. In some embodiments, this first condition is more conservative and only does not use the conventional ECM mode if none of the samples it uses are available. In such embodiments, E001 is written as: ○    If at least one upper sample from ref[W+W/2+(W/2)*D/H; -1-D] up to ref[2*W-1+W*D/H; -1-D] is available (i.e., non-filling sample), then the set of modes defined for the aspect ratio W/H may be used. E003 is written as ○    If at least one left sample from ref[-1-D; H+H/2+(H/2)*D/W] up to ref[-1-D; 2*H-1+H*D/W] is available (i.e., non-filling sample), then the set of modes defined for aspect ratio W/H can be used.

Conditions E002 and E004 check if samples are available for modes that are not normally used with ECM, and only if all samples are available, the modes are added. In some embodiments, this condition is relaxed, and if at least one of the samples is available, the modes are added. In such embodiments, E002 is written as: ○    Otherwise, if at least one left sample from ref[-1-D; 2*H+H*D/W] up to ref[-1-D; H+W/2 +H*D/(W/2)- 1] is available, the set of modes defined for aspect ratio (W/2)/H may be used. Otherwise, (if none of the samples are available), the set of modes defined for aspect ratio W/H may be used. E004 is written as ○    Otherwise, if at least one upper sample from ref[2*W+W*D/H; -1-D] up to ref[W+H/2-1+W*D/(H/2);-1-D] is available, then the set of patterns defined for aspect ratio W/(H/2) can be used. Otherwise, (if none of these samples are available), then the set of patterns defined for aspect ratio W/H can be used.

Some embodiments use a combination of these conditions. Some embodiments use different conditions depending on (but not limited to) block size, sequence size, QP, or neighbor information.

In some embodiments, more patterns are added. In such embodiments, when the set of ECM patterns for W/H is replaced by the set of patterns for (W/2)/H (resp. W/(H/2)), then tests E001 to E004 are performed again as if the width of the block is W/2 (resp. the height is H/2). If a new set of patterns is selected, this can be performed again until it is preferred not to change the set of patterns to be used.

In some embodiments, an existing availability check of neighboring CUs used to construct the MPM list can be used to determine whether a mode should be added. For example, a test for the availability of reference samples is performed at positions 1 and 2, followed by those tests at positions 3 and 4, as described in FIG. 19 . In this case, conditions [E001] to [E004] do not depend on whether W is greater than H. In one example, more modes will be added and no modes will be removed, which can be particularly useful when no signaling is required, for example, if the modes are added for use with TIMD or DIMD.

In one example, if the upper sample from ref[-1; -1] up to ref[W; -1] is available (i.e., non-padded sample up to position 1), then all modes defined as described in Table 1 for aspect ratio 2*W/H may be used in addition to the modes already allowed for this CU. If the upper sample ref[2*W; -1] is also available (i.e., non-padded sample), then all modes defined as described in Table 1 for aspect ratio 4*W/H may be used in addition to the modes already allowed for this CU. If the left sample from ref[-1; -1] up to ref[-1; H] is available (i.e., non-padded sample up to position 2), then all modes defined as described in Table 1 for aspect ratio W/(H*2) may be used in addition to the modes already allowed for this CU. If the upper sample ref[-1; 2*H] is also available (i.e., non-padding sample), then in addition to the modes already allowed for this CU, all modes defined for aspect ratio W/(H*4) as described in Table 1 may be used. Signaling of intra-frame modes

In some embodiments, the number of available IPMs is maintained at 67, with a fixed number of 65 angular IPMs. In such embodiments, no signaling change is required. In other embodiments, the context of CABAC-encoded bins for syntax elements associated with intra-frame mode signaling (e.g., primary MPM flag, secondary MPM flag, or first MPM index flag) may be modified to take into account available modes. For example, one of three context model indices will be selected depending on the following conditions: 1)    The IPMs used are the same as those currently in ECM-6.0. On a PU with W/H = 1, for example, those would be angular modes 2 to 66 as well as planar and DC. 2)    The IPMs used are changed to use IPMs that are typically available for modes with larger W/H ratios. On a PU with W/H = 1, for example, this would mean using angular modes from I to I + 64 (where I > 2) as well as planar and DC. 3)    The used IPM is changed to use an IPM that is normally available for modes with smaller W/H ratios. On a PU with W/H = 1, for example, this would mean using angular modes from I to I + 64 (where I < 2) as well as planar and DC.

In some embodiments, according to Table 128 in the VTM specification text, the flag intra_luma_mpm_flag (which is used to specify whether the intra mode used in the current luma CB is in the MPM list) has the following syntax changes: Syntax elements binIdx 0 1 2 3 4 ＞= 5 intra_subpartitions_split_flag 0 na na na na na intra_luma_mpm_flag[ ][ ] 0 0, 1, 2 (intra_mode_set_diff[][]) na na na na na intra_luma_not_planar_flag[ ][ ] intra_subpartitions_mode_flag na na na na na The value of intra_mode_set_diff is derived as follows: If the set of modes for the current brightness CB with width W and height H (at position posX, posY) is the set designed for blocks with ratio W/H (as defined in Table 1), then set intra_mode_set_diff[posX][posY] to 0. Otherwise, if the set of modes is designed for blocks with ratio strictly greater than W/H, then set intra_mode_set_diff[posX][posY] to 1. Otherwise, set intra_mode_set_diff[posX][posY] to 2.

The context model is initialized as follows, as defined in Table 70 of the VTM specification text. Table 70 – Specification of initValue and shiftIdx for ctxIdx of intra_luma_mp_flag Initialize variables intra_luma_mpm_flag ctxIdx ​ 0 1 2 3 4 5 6 7 8 initValue EP EP EP EP EP EP EP EP EP shiftIdx 0 0 4 0 0 1 0 0 1

In some embodiments, three contexts may be defined depending on the following conditions: 1)    The IPMs used are the same as those currently in ECM-6.0. On a PU with W/H = 1, for example, they would be angle modes 2 to 66 as well as planar and DC. 2)    The IPMs used are changed to use an IPM that is typically available for modes with larger abs(log2(W/H)) ratios. On a PU with W/H ≥ 1 (resp. W/H ≤ 1), this would mean using angle modes from I to I + 64 (where I ＞ J (resp. I ＜ J), J being the smallest angle mode typically available on this PU) as well as planar and DC. 3)    The IPMs used are changed to use an IPM that is typically available for modes with larger abs(log2(W/H)) ratios. On a PU with W/H > 1 (resp. W/H < 1), this would mean using angle modes from I to I+64 (where I < J (resp. I > J), J being the smallest angle mode typically available on this PU) as well as planar and DC.

In some embodiments, the previously mentioned rules may be reduced to using only 2 CABAC contexts (i.e., depending on whether the modes are changed). In some embodiments, the rules may be combined with other information such as, but not limited to, PU size, sequence size, QP, prediction tool used.

In some embodiments, the number of available modes varies from PU to PU, and therefore the signaling changes to account for the different number of modes. In one embodiment, the first MPM list and the secondary MPM list keep the flags of their original number to be encoded, and the remaining IPMs are encoded using truncated binary encoding for N-22 symbols, where N is the number of available modes for this PU (where N = 45 means using the same signaling as in ECM-6.0, N > 45 means there are more available modes than in ECM-6.0, and N < 45 means there are fewer available modes than in ECM-6.0).

In some embodiments, the availability of IPMs is limited to TIMD (Template-based Intra Mode Derivation) and/or DIMD and/or other decoder-side tools to avoid any changes that need to be signaled. In such embodiments, TIMD searches (resp. DIMD searches, and/or other decoder-side tools) may be limited to IPMs that are considered available.

In embodiments where the TIMD can use an extra wide angle, only a subset of patterns are added to reduce the increase in complexity of the search. The extra patterns can be included in the first part of the search, for example, as described below.

Using the values of TIMD for 131 modes (e.g., from Table 1, angular modes from 8 to 72 for W/H=2, which correspond to minOrg = 13 and maxOrg = 141), let W and H be the width and height of the block to be encoded, let minOrg and maxOrg be the minimum and maximum angular intra-frame mode values available for the current block, and let newMin and newMax be the new minimum and maximum angular intra-frame mode values available for the current block, for example, which are determined from E001 to E004 (e.g., if all neighboring samples are available for blocks with W/H=2, then add modes from -4 to 7 and modes from 73 to 78 according to Table 1, which means using the values of TIMD for 131 modes, newMin = -9 and newMax = 153). With a step size of N (e.g., N=5), starting from newMin+1 up to newMax, if the pattern is not between minOrg and maxOrg, it is added to the first part of the TIMD search; otherwise, the pattern is not added.

You can also choose to always add certain patterns to the search. For example, you can choose to always add newMin+1, orgMin-1, orgMax+1, and newMax+1 to the first part of the search. These patterns can be the only ones added in the search to reduce the complexity of the design; or they can be added on top of patterns that have already been added to maximize compression gains. The second part of the TIMD search (the refinement part) can be done as in ECM.

In some embodiments, before decoding the IPM index, if some additional modes are found to be available, an additional flag is decoded to indicate whether to use the original 67 modes allowed for the current block size, or whether to use one of the N additional modes. If one of the N additional modes is used, an additional index is decoded using truncated binary coding for N symbols. Propagation in the Intra-Frame Direction

In ECM, if the neighboring block to the left has W/H = 2 and uses angle mode 67 for prediction, then the mode index used to construct the MPM list for the current block is 2. However, if the current luma block is square, then index 2 corresponds to angle mode 2. Therefore, using the mode index of the neighboring mode to construct the MPM list for the current block may result in adding modes to the MPM list that will not actually be used, and therefore should not be considered the "most likely" to be used to decode the current luma coded block. In addition, in ECM, each index may correspond to two different angle modes (e.g., index 2 is angle mode 2 or angle mode 67, index 3 is angle mode 3 or angle mode 68, etc.), but the two different modes are not 180° opposite.

The two modes are 180° relative meaning that they predict the same directional texture (angle mode 2 and 66 both predict a 45° direction), but from different references (angle mode 2 predicts from lower left to upper right, while angle mode 66 predicts from upper right to lower left).

In some embodiments, the MPM list is built from the actual mode used by the neighboring blocks instead of the index used. For example, when building the MPM list and the neighboring mode is not available, the mode is replaced by the corresponding mode at 180°, that is, if the mode IPM is below 34, the mode is replaced by IPM + 64, otherwise the mode is replaced by IPM - 64. In ECM up to ECM-6.0, this is always possible because there is always a span of 180° angle modes.

This may not be the case in an encoder or decoder that removes some modes but does not add others (such as the encoders or decoders described in some of the aforementioned embodiments). If during MPM list construction a mode that is not available in the current luma CB should be added, it is replaced by the closest available mode. In some embodiments, a mode is chosen that always has a span of 65 angular modes to ensure that any angle is available and it is always possible to replace a mode with its 180° counterpart. Beyond wide-angle in-frame modes

The selection of available intra-frame prediction modes based on the availability of neighboring decoded reference samples can be extended to no longer involve the rules in Table 1. This means that for a given block, depending on the availability of its neighboring decoded reference samples, its available intra-frame prediction mode set for its effective ratio W/H (see Table 1) will no longer be replaced by a different available intra-frame prediction mode set associated with a width-to-height ratio close to its effective ratio W/H. Given the availability of neighboring decoded reference samples for a block, a given number of intra-frame prediction modes can be removed.

For example, the rule that may suppress the intra-frame prediction mode may be as follows. block, if it is located on the upper right side If all decoded reference samples are unavailable, the last Positive vertical intraframe prediction modes are not allowed (e.g. )."finally" A positive vertical intra-frame prediction mode is one with the maximum absolute value of the angle relative to the vertical axis. positive vertical intra-frame prediction mode. According to ECM nomenclature, the "last" The positive vertical intra-frame prediction mode is the one with the largest index If the H decoded reference samples at the lower left are not available, the previous Positive horizontal intraframe prediction modes are not allowed (e.g. )."forward" A positive horizontal intra-frame prediction mode is one with the maximum absolute value of the angle relative to the vertical axis. According to ECM nomenclature, "front" The positive horizontal intra-frame prediction mode is the one with the smallest index positive horizontal intra-frame prediction mode. In case of wide-angle intra-frame prediction, these indices can take negative values.

As an example, in a given In the case of a luma coding block (CB) belonging to an intra slice in ECM-5.0, this rule can be illustrated in FIG. 12. For this given luma CB (1201) within an intra slice in ECM-5.0, the disallowed intra prediction mode is identified from the segmentation history of (1201). Here, W=16 and H=8. The indices 0, 1, 2, and 3 indicate the order for encoding/decoding the first four luma CBs belonging to the first 64×64 luma CB (1200) generated from the QT splitting of the luma CTB under consideration.

In this example, it must be noted that for this given A luma CB can fully specify the availability of its neighboring decoded reference samples at coding time before any bits of the partition of its parent luma Coding Tree Block (CTB) are written to the bitstream (i.e., before any bits associated with intra-frame prediction within its parent luma CTB are written). A luma CB can fully specify the availability of its neighboring decoded reference samples at decoding time after reading the bits of its parent luma CTB segmentation from the bitstream (i.e., before reading any bits associated with intraframe prediction within this parent luma CTB). In the following example, in ECM-5.0, the CTU size is set to (as in VVC) to obtain an instance that is more equivalent to its version in VVC.

On the encoder side, the given The brightness CTB is split into four parts by a quad-tree (QT). Brightness CB. For example, consider the first Luma CB (1200). Let us characterize the split at a given depth by (typeSplit, idxChild), "typeSplit" refers to the type of split in {Quad Tree (QT), Binary-Tree Horizontal (BT_H), Binary-Tree Vertical (BT_V), Ternary-Tree Horizontal (TT_H), Ternary-Tree Vertical (TT_V)}, and "idxChild" represents the index of the coding order of the considered child CB resulting from this split. Then, the partition of the considered luma CB (1201) is completely described by its split tree {(QT, 0), (QT, 0), (BT_V, 1), (TT_H, 2)}, as shown in Figure 12A. From this split tree, we can see that the The decoded references are not available and the It becomes apparent that none of the decoded reference samples are available.

These two parts of unavailable decoded reference samples are summarized as (1202) in Figure 12B. For example, this can be indicated by the flags "is_above_right_full" at 0 and "is_below_left_full" at 0 attached to (1201), respectively. The positive vertical intra-frame prediction mode is not allowed, (1204) and (1203) respectively represent the direction of the intra-frame prediction mode with the smallest index and the direction of the intra-frame prediction mode with the largest index in this set of not allowed modes. A positive horizontal intra-frame prediction mode is not allowed, (1205) and (1206) respectively represent the direction of the intra-frame prediction mode with the smallest index and the direction of the intra-frame prediction mode with the largest index in this set of not allowed modes, as shown in FIG. 12C.

Finally, the signaling (1201) of the index of the intra prediction mode selected for prediction is adapted to account for the removed intra prediction mode. For example, if the selected intra prediction mode is not a template-based intra prediction (TMP), decoder-side intra mode derivative (DIMD), template-based intra mode derivative (TIMD), or matrix-based intra prediction (MIP) mode, does not use MRL, and is not an MPM, then its index is expressed in code length N - - It should be noted that since ECM-5.0 contains 67 conventional intra-frame prediction modes (65 directions, planes, and DC), 6 primary MPMs, and 16 secondary MPMs, the index of a conventional intra-frame prediction mode that is not one of the MPMs can have N = 45 possibilities.

The procedure on the decoder side follows the procedure on the encoder side, except that the index of the intra prediction mode selected for prediction is signaled (1201). In view of the above example, if the selected intra prediction mode is not TMP, DIMD, TIMD, or MIP mode, MRL is not used, and it is not MPM, then its index is used for 45- - The binary code is decoded by truncating the possible symbols.

In given In the case where the luma coding block (CB) belongs to an intra-slice in ECM-5.0, another example of this embodiment can be illustrated in FIG. 13. For a given CB in an intra-slice in ECM-5.0 Luma CB (1301), identifying disallowed intra prediction modes from the segmentation history of (1301). Here, W=8 and H=16. Indices 0 to 7 indicate the order for encoding/decoding the first eight luma CBs belonging to the first 64×64 luma CB (1300) generated from the QT splitting of the luma CTB under consideration.

On the encoder side, the given The brightness CTB is split into 4 parts by QT Brightness CB. For example, consider the first Luminance CB (1300). The partition of the considered luminance CB (1301) is completely specified by its splitting tree {(QT, 0), (QT, 1), (BT_H, 1), (BT_V, 0), (BT_V, 1)}, as shown in Figure 13A.

From this splitting tree, the fact that the W decoded references to the upper right of (1301) are all available and the H decoded reference samples to the lower left of (1301) are all unavailable can be directly inferred, as shown in (1302) in Figure 13B. For example, this can be indicated by the flags "is_above_right_full" at 1 and "is_below_left_full" at 0 attached to (1301), respectively. positive horizontal intra-frame prediction modes are not allowed, (1303) and (1304) represent the direction of the smallest indexed intra-frame prediction mode and the direction of the largest indexed intra-frame prediction mode in this set of disallowed modes, respectively, as shown in FIG. 13C. Finally, the signaling (1301) of the index of the intra-frame prediction mode selected for prediction is adapted to account for the removed intra-frame prediction mode. For example, if the selected intra-frame prediction mode is not a TMP, DIMD, TIMD, or MIP mode, MRL is not used, and it is not an MPM, then its index is used for 45 - The possible symbols are encoded as truncated binary codes.

The procedure on the decoder side follows that on the encoder side, except that the index of the intra-frame mode selected for prediction is signaled (1301). In view of the above example, if the selected intra-frame prediction mode is not TMP, DIMD, TIMD, or MIP mode, MRL is not used, and it is not MPM, then its index is used for 45- The binary code is decoded by truncating the possible symbols.

The above example can be adapted for any other block in another channel/slice. Furthermore, the rule for suppressing intra-frame prediction mode depending on the availability of neighboring decoded reference samples for the current block can be modified directly. For example, this rule can become "for a given Block, if located at the rightmost If all decoded reference samples are unavailable, the last A positive vertical in-frame prediction mode is not allowed. If it is located at the bottom left If all decoded reference samples are unavailable, the previous Positive horizontal intraframe prediction mode is not allowed.

For a given Block, the workflow of encoding the index of the selected intra prediction mode on the encoder side following this embodiment can be summarized by Figure 14. In this embodiment, the rule (which indicates the conditional relationship between the availability of neighboring reference samples for a given block and which intra prediction modes to remove for the given block) is known on the encoder side. Specifically, for the current block, the encoder identifies (1410) unavailable reference samples from the segmentation history of the current block. For example, for the segmentation history of {(QT, 0), (QT, 0), (BT_V, 1), (TT_H, 2)}, in Figure 12, is_above_right_full = false and is_below_left_full = false. Based on the identification of unavailable reference samples and the rules for removing intra-frame prediction modes, the encoder removes (1420) some intra-frame prediction modes that are not applicable to the current block. For example, for is_above_right_full = False and is_below_left_full = False, the last Positive vertical frame prediction mode and front A positive horizontal intra-frame prediction mode is not allowed in Figure 12.

Taking into account the removed modes, the encoder adapts (1430) the signaling of the index of the intraframe prediction mode selected to predict the current block. In general, the total number of available intraframe prediction modes is adjusted by reducing the amount of removed intraframe prediction modes. For example, if the selected intraframe prediction mode is not a TMP, DIMD, TIMD, or MIP mode, MRL is not used, and it is not MPM, then in FIG. 12 its index is expressed in code length 45. - The coded index is then written (1440) into the bitstream.

For this Block, the workflow of decoding the index of the selected intra-frame prediction mode on the decoder side following this embodiment can be summarized in Figure 15. The steps (1510, 1520, 1530, 1540) at the decoder side correspond to those at the encoder side.

These two figures show the workflow for only a single block. When multiple blocks are considered, the order of the steps in Figures 14 and 15 depends on the codec concerned.

In Figures 12 to 15, for a given Block, from the segmentation history of this block, unavailable decoded reference samples are identified. In other embodiments, a function that searches for decoded blocks around the current block can be used to identify the unavailability of decoded reference samples.

For example, as shown in FIG. 16, for a given CB (1602), function "getCURestricted" takes a pixel position "pos" (e.g., "posAR" (1603) or "posBL" (1604)), a given The coding unit (CU) "curCu" of CB (1602), and the channel type "chType" of (1602) are used to return a pointer to the decoded CB containing the pixel at "pos". If the pixel at "pos" does not belong to any CB or it belongs to a CB that has not yet been decoded, "getCURestricted" may return a pointer NULL, for example, "nullptr" in C++. In FIG. 16 , CBs (1600), (1601), and (1602) are generated from the last two split BT_V and BT_H in the current state of encoding/decoding. For example, in FIG. 16 , because "posAR" belongs to a CB that has not yet been decoded, "getCURestricted(posAR, curCu, chType)" returns "nullptr".

For a given Block, the workflow of encoding the index of the selected intra-frame prediction mode on the encoder side according to this embodiment can be summarized by FIG. 17. Specifically, for the current block, the encoder identifies (1710) unavailable reference samples from the segmentation history of the current block. For example, in FIG. 16, if getCURestricted(posAR, curCu, chType) = nullptr, then is_above_right_full = false; if getCURestricted(posBL, curCu, chType) = nullptr, then is_below_left_full = false. Based on the identification of unavailable reference samples and the rules for removing intra-frame prediction modes, the encoder removes (1720) some intra-frame prediction modes (these intra-frame prediction modes are not available for the current block). For example, for is_above_right_full = False and is_below_left_full = False, then Positive vertical frame prediction mode and front Positive horizontal intraframe prediction mode is not allowed.

Taking into account the removed mode, the encoder adapts (1730) the signaling of the index of the intra-frame prediction mode selected to predict the current block. For example, if the selected intra-frame prediction mode is not TMP, DIMD, TIMD, or MIP mode, MRL is not used, and it is not MPM, then its index is expressed in code length 45- - The coded index is then written (1740) into the bitstream.

For this Block, the workflow of decoding the index of the selected intra-frame prediction mode on the decoder side following this embodiment can be summarized in Figure 18. The steps (1810, 1820, 1830, 1840) on the decoder side correspond to those on the encoder side.

In another embodiment, for a given block, instead of removing a given number of intra-frame prediction modes depending on the availability of its neighboring decoded reference samples, its MPM list may be reordered so that for some intra-frame prediction modes that intensively use unavailable decoded reference samples for prediction and whose indexes are in this MPM list, their indices are moved towards the end of this MPM list. Those intra-frame prediction modes whose indices are moved towards the end of the MPM list of the given block are considered to be relatively less likely to be selected as intra-frame prediction modes for predicting the given block.

In ECM-5.0, the current An example of brightness CB is shown in FIG19. Specifically, FIG19 shows that for a given Creation of a common list of 22 MPMs for this luminance CB under the luminance CB. The first 6 MPMs in the common list of MPMs correspond to the primary MPM list, and the last 16 MPMs in the common list of MPMs generate the secondary MPM list.

In Figure 19, the plane mode is first added to the general list of MPM (1900). Then, the index of the intra-frame prediction mode selected for predicting the left, top, bottom left, top right, and top left brightness CB is added to the general list of MPM (1901 to 1905). Then, the index of the two intra-frame prediction modes derived from DIMD for the current brightness CB is added to the general list of MPM (1906, 1907). Then, if the current second MPM is neither plane nor DC, the index of its eight adjacent angle intra-frame prediction modes is placed in the general list of MPM (1908). Then, if the current third MPM is neither plane nor DC, the index of its eight adjacent angle intra-frame prediction modes is placed in the general list of MPM (1909).

After possibly adding the indices of more angular intra-frame prediction modes according to (1910), the index of the default mode is inserted into the general list of MPMs (1911) to reach 22 MPMs. It should be noted that each of the above insertions applies under the condition that there are no redundancies in the general list of MPMs. This means that for the index of the current intra-frame prediction to be inserted into the general list of MPMs, if this index already exists in this list, the insertion is skipped.

FIG. 20 illustrates the creation of a generic list of 22 MPMs for the same current brightness CB according to an embodiment. The creation of a generic list of 22 MPMs for the luminance CB follows the workflow in Figure 19, except that a reordering may be introduced. For example, the function The index of the candidate intra-frame prediction mode to be put into the general list of MPMs may be used as the first argument, and the array "res" of reserved mode indices may be used as the second argument. Conditional on the availability of decoded reference samples of luminance CB, the candidate intra-frame prediction model is "valid", then The index of the candidate intra-frame prediction mode can be put into the general list of MPM. Otherwise, The index of the prediction mode within this frame can be added to "res".

First, the planar mode is added to the general list of MPM (2000). Then, under the validity condition defined by f, the index of the intra-frame prediction mode selected for predicting the left, top, bottom left, top right, and top left brightness CB is added to the general list of MPM (2001 to 2005). Then, all intra-frame prediction mode indices stored in "res" are added to the general list of MPM (2006). Then, the indices of the two intra-frame prediction modes derived from DIMD for the current brightness CB are added to the general list of MPM (2007, 2008). The last steps (2009), (2010), (2011) and (2012) follow (1908), (1909), (1910) and (1911) in Figure 19 respectively.

In another embodiment, The index of any candidate intra-frame prediction mode that may be applied (or not applied) to be possibly added to the general list of MPMs for the current brightness CB. Furthermore, the step of placing all intra-frame prediction mode indices stored in "res" into the general list of MPMs for the current brightness CB may occur at any time during the creation of the general list of MPMs.

For example, in one embodiment shown in FIG. 21 , after possibly placing the indices of two intra-frame prediction modes derived from DIMD for the current brightness CB into the general list of the MPM ( 2106 , 2107 ), the addition of all intra-frame prediction modes stored in “res” occurs ( 2108 ).

For example, in another embodiment shown in FIG. 22, Under the conditions of defined validity, the addition of all intra-frame prediction modes stored in "res" occurs (2208), after possibly placing the indices of the two intra-frame prediction modes derived from the DIMD for the current brightness CB into the general list of the MPM (2206, 2207).

In one embodiment, for a frame indexed by an intra-frame prediction mode as the first argument, The validity conditions defined can be: if the Upper right side of the block If all decoded reference samples are unavailable, the last The positive vertical intraframe prediction mode is invalid (for example, ). If you are currently If the H decoded reference samples on the lower left side of the block are not available, then The positive level intraframe prediction model is invalid (e.g. ).

In another embodiment, for a frame indexed by an intra-frame prediction mode as a first argument, The validity conditions defined can be: if the The rightmost upper right corner of the block If all decoded reference samples are unavailable, then the last The positive vertical intraframe prediction mode is invalid (for example, ). If you are currently The lowest one on the lower left side of the block If all decoded reference samples are unavailable, the previous The positive level intraframe prediction model is invalid (e.g. ).

In yet another embodiment, for a frame indexed by an intra-frame prediction mode as a first argument, The validity conditions defined can be: if the If the decoded reference samples above the block (including the upper left, top, and top right) are not available, the last The vertical intraframe prediction mode is invalid (for example, ). If you are currently If the decoded reference samples on the left side of the block (including the upper left, left, and lower left) are not available, then The horizontal intraframe prediction model is invalid (for example, ).

In yet another embodiment, for a frame indexed by an intra-frame prediction mode as a first argument, The validity of the definition may depend on the current The composition of several conditions for different availability states of decoded reference samples of a block. As an example, if the decoded reference samples above (including the upper left, above, and upper right) the current block are all unavailable, then the last The vertical intraframe prediction mode is invalid (for example, ). Otherwise, check the following condition. If the If all decoded reference samples are unavailable, the last The positive vertical intraframe prediction mode is invalid (for example, As another example, if the decoded reference samples on the left side (including the upper left, left, and lower left) of the current block are all unavailable, then the previous The horizontal intraframe prediction model is invalid (for example, ). Otherwise, check the following condition. If the If all decoded reference samples are unavailable, the previous The positive level intraframe prediction model is invalid (e.g. ).

Various methods are described herein, and each of the methods includes one or more steps or actions for implementing the described method. Unless proper operation of the method requires a specific order of steps or actions, the order and/or use of specific steps and/or actions may be modified or combined. Additionally, terms such as "first," "second," and the like may be used in various embodiments to modify elements, components, steps, operations, and the like, such as "first decoding" and "second decoding." Unless specifically required, the use of such terms does not imply an ordering of the modified operations. So in this example, the first decoding need not be performed before the second decoding, and may, for example, occur before, during, or during a time period overlapping with the second decoding.

The various methods and other aspects described in this application may be used to modify modules, such as the intra-frame prediction module (260, 360) of the video encoder 200 and decoder 300 shown in FIG. 2 and FIG. 3. In addition, the present aspects are not limited to ECM, VVC, or HEVC, and may be applied, for example, to other standards and recommendations, and any extensions of such standards and recommendations. Unless otherwise indicated or technically excluded, the aspects described in this application may be used individually or in combination.

Various numerical values are used in this application. Specific values are used for various purposes, and the described embodiments are not limited to these specific values.

Various embodiments relate to decoding. As used in this application, "decoding" may encompass all or part of a process performed, for example, on a received encoded sequence to produce a final output suitable for a display. In various embodiments, such a process includes one or more of the processes typically performed by a decoder (e.g., entropy decoding, inverse quantization, inverse transform, and differential decoding). Regardless of whether the phrase "decoding process" is intended to refer specifically to a subset of operations or more generally, the decoding process will be clear based on the context of the particular description and is believed to be well understood by one of ordinary skill in the art.

Various embodiments involve encoding. In a manner similar to the above discussion of "decoding", "encoding" as used in this application may cover all or part of the process performed on an input video sequence to produce an encoded bit stream, for example.

The embodiments and aspects described herein may be implemented, for example, as methods or procedures, apparatus, software programs, data flows, or signals. Even if discussed only in the context of a single form of an embodiment (e.g., discussed only as a method), the embodiments of the features discussed may also be implemented in other forms (e.g., apparatus or program). An apparatus may be implemented, for example, in appropriate hardware, software, and firmware. A method may be implemented, for example, in a processor device (e.g., a processor), which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes a communication device, such as a computer, a cellular phone, a portable/personal digital assistant ("PDA"), and other devices that facilitate the communication of information between end users.

Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearance of the phrase "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations appearing throughout this application are not necessarily all references to the same embodiment.

Additionally, the application may be associated with "determining" various information. Determining information may include, for example, one or more of evaluating information, calculating information, predicting information, or retrieving information from memory.

Further, the application may be related to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or evaluating information.

Additionally, this application may be related to "receiving" various items of information. Receiving is intended to be a broad term, as is "accessing." Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from memory). Further, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or evaluating information, generally involves "receiving" in some way or another.

It should be understood that the use of “/”, “and/or”, and “at least one of” in situations such as “A/B”, “A and/or B”, and “at least one of A and B” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the case of "A, B, and/or C" and "at least one of A, B, and C", such phrases are intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to as many items as listed, as is clear to one of ordinary skill in the art.

Furthermore, as used herein, the term "signal" refers in particular to indicating something to a corresponding decoder. For example, in some embodiments, the encoder signals a quantization matrix for dequantization. In this way, in one embodiment, the same parameters are used on both the encoder side and the decoder side. Thus, for example, a encoder may transmit (explicitly signal) a specific parameter to a decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling may be used without transmitting (implicitly signaling) to allow only the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding transmitting any actual functionality. It should be understood that signaling can be achieved in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. Although the foregoing is about the verb form of the word "signal", the word "signal" may also be used as a noun in this article.

As will be apparent to one of ordinary skill in the art, embodiments may generate various signals formatted to carry information that may, for example, be stored or transmitted. The information may include, for example, instructions for executing a method, or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the optical spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted via a variety of different wires or links. The signal may be stored on a processor-readable medium.

100: system 105: block 110: processor 115: connection configuration 120: memory 130: encoder/decoder module; encoder/decoder 140: storage device 150: communication interface 160: interface; display interface 170,180: interface 165: display 175: speaker 185: peripheral device 190: communication channel 200: video encoder; encoder 201: pre-coding process 202: segmentation 205: decision 210: subtraction 225: conversion 230: quantization 240: dequantization 245: entropy coding 250: inverse conversion 255: combination 260: intra-frame prediction; intra-frame prediction module 265: in-loop filter 270: compensation 275: motion estimation 280: reference image buffer 300: video decoder; decoder 335: segmentation 340: dequantization 350: inverse conversion 355: combination 360: intra-frame prediction; intra-frame prediction module 365: in-loop filter 370: acquisition 375: motion compensation prediction 380: reference image buffer 385: post-decoding processing 610,620,630,640: steps 1200: First 64×64 luminance CB 1201: Luminance CB; Considered luminance CB 1202: These two parts of the decoded reference sample cannot be used 1203: Direction of the intra-frame prediction mode of the maximum index 1204: Direction of the intra-frame prediction mode of the minimum index 1205: Direction of the intra-frame prediction mode of the minimum index 1206: Direction of the intra-frame prediction mode of the maximum index 1300: First 64×64 luminance CB 1301: W×H luminance CB; Considered luminance CB 1303: Direction of the intra-frame prediction mode of the minimum index 1304: Direction of the intra-frame prediction mode of the maximum index 1410,1420,1430,1440: Steps 1510,1520,1530,1540: steps 1600,1601: CB 1602: W×H CB；CB 1710,1720,1730,1740: steps 1810,1820,1830,1840: steps 1900,1901,1902,1903,1904,1905,1906,1907,1908,1909,1910,1911: steps 2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012: steps 2106,2107,2108: Steps 2206,2207,2208: Steps

[FIG. 1] shows a block diagram of a system in which the present embodiment can be implemented. [FIG. 2] shows a block diagram of an embodiment of a video encoder. [FIG. 3] shows a block diagram of an embodiment of a video decoder. [FIG. 4] shows reference samples for intra-frame prediction. [FIG. 5A] and [FIG. 5B] show reference sample replacement for intra-frame prediction. [FIG. 6] shows a procedure for reference sample replacement for intra-frame prediction. [FIG. 7A] shows intra-frame prediction directions in HEVC, [FIG. 7B] shows intra-frame prediction directions in VVC, and [FIG. 7C] shows horizontal and vertical, positive and negative intra-frame prediction modes. [FIG. 8A], [FIG. 8B], and [FIG. 8C] show wide-angle intra-frame prediction. [FIG. 9] illustrates all available intra prediction directions in VVC. [FIG. 10] illustrates planar mode. [FIG. 11] illustrates reference samples for each intra prediction mode (IPM) on a PU with aspect ratio W/H=4. [FIG. 12A] illustrates a CB in an intra slice, [FIG. 12B] illustrates unavailable reference samples, and [FIG. 12C] illustrates a removed intra prediction mode. [FIG. 13A] illustrates another CB in an intra slice, [FIG. 13B] illustrates unavailable reference samples, and [FIG. 13C] illustrates a removed intra prediction mode. [FIG. 14] illustrates a workflow for signaling an index of an intra prediction mode selected to predict a current WxH block on the encoder side according to an embodiment. [FIG. 15] illustrates a workflow for decoding an index of an intra-frame prediction mode selected to predict a current WxH block on a decoder side according to one embodiment. [FIG. 16] illustrates identifying unavailable decoded reference samples around the current block using a search for decoded blocks around the current block. [FIG. 17] illustrates a workflow for signaling an index of an intra-frame prediction mode selected to predict a current WxH block on an encoder side according to another embodiment. [FIG. 18] illustrates a workflow for decoding an index of an intra-frame prediction mode selected to predict a current WxH block on a decoder side according to another embodiment. [FIG. 19] illustrates the creation of a generic list of 22 MPMs for a current luma CB in an ECM. [FIG. 20] illustrates a modified creation of a generic list of 22 MPMs for the current brightness CB in an ECM according to one embodiment. [FIG. 21] illustrates a modified creation of a generic list of 22 MPMs for the current brightness CB in an ECM according to another embodiment. [FIG. 22] illustrates a modified creation of a generic list of 22 MPMs for the current brightness CB in an ECM according to another embodiment.

100:System

105: Block

110: Processor

115: Connection configuration

120: Memory

130: Encoder/decoder module; encoder/decoder

140: Storage device

150: Communication interface

160:Interface; Display interface

170,180: Interface

165: Display

175: Speaker

185: Peripheral devices

190: Communication channel

Claims (22)

A method for video decoding, comprising: For a block to be decoded in an image, identifying the availability of one or more reference samples; Obtaining a set of intra-frame prediction modes for the block in response to the availability of the one or more reference samples; Obtaining an intra-frame prediction mode from the set of intra-frame prediction modes; and Performing intra-frame prediction of the block to be decoded based on the intra-frame prediction mode of the block to form a prediction block for the block. A method for video coding, comprising: For a block to be coded in an image, identifying the availability of one or more reference samples; Responsibly obtaining a set of intra-frame prediction modes for the block; Selecting an intra-frame prediction mode from the set of intra-frame prediction modes; and Based on the intra-frame prediction mode of the block, performing intra-frame prediction of the block to be coded to form a prediction block for the block. The method of claim 1 or 2, wherein the availability of the one or more reference samples is identified based on a segmentation history of the block. A method as claimed in any one of claims 1 to 3, wherein obtaining a set of intra-frame prediction models comprises: obtaining a first set of intra-frame prediction models; and adjusting the first set of intra-frame prediction models to the set of intra-frame prediction models in response to the availability of the one or more reference samples. A method as in any one of claims 1 to 4, wherein the first set of intra-frame prediction models is obtained based on an aspect ratio of the block. The method of claim 5, wherein the first set of intra-frame prediction models is obtained based on an aspect ratio different from the aspect ratio of the block. The method of claim 6, wherein the different aspect ratio is half or twice the aspect ratio of the block. The method of any of claims 1 to 7, wherein the adjusting comprises: Removing a plurality of vertical positive in-frame prediction modes in response to one or more upper right reference samples being unavailable. The method of any of claims 1 to 8, wherein the adjusting comprises: Removing a plurality of horizontal positive in-frame prediction modes in response to one or more lower left reference samples being unavailable. A method as claimed in any one of claims 1 to 9, wherein the adjusting comprises: Adding a plurality of vertical positive in-frame prediction modes in response to one or more top right reference samples being available. A method as claimed in any one of claims 1 to 10, wherein the adjusting comprises: Adding a plurality of horizontal positive intra-frame prediction modes in response to one or more lower left reference samples being available. A method as in any one of claims 1 to 11, wherein an index corresponding to the intra-frame prediction mode is signaled in response to a number of intra-frame prediction modes in the set of intra-frame prediction modes. The method of claim 12, in response to the intra-frame prediction mode belonging to a set of remaining modes, the set of remaining modes excluding a most probable mode (MPM), and wherein the index is encoded using a truncated binary encoding for one of N - M symbols, where N is the number of intra-frame prediction modes in the set of intra-frame prediction modes, and M is a number of MPMs. A method as in any of claims 1 to 13, wherein a context index for a syntax element depends on a prediction pattern within the group frame. The method of claim 14, wherein the syntax element is used to signal the intra-frame prediction mode. A device for video decoding, comprising one or more processors, wherein the one or more processors are configured to: identify the availability of one or more reference samples for a block to be decoded in an image; obtain a set of intra-frame prediction modes for the block in response to the availability of the one or more reference samples; obtain an intra-frame prediction mode from the set of intra-frame prediction modes; and perform intra-frame prediction of the block to be decoded based on the intra-frame prediction mode of the block to form a prediction block for the block. An apparatus for video encoding, comprising one or more processors, wherein the one or more processors are configured to: identify the availability of one or more reference samples for a block to be encoded in an image; obtain a set of intra-frame prediction modes for the block in response to the availability of the one or more reference samples; select an intra-frame prediction mode from the set of intra-frame prediction modes; and perform intra-frame prediction of the block to be encoded based on the intra-frame prediction mode of the block to form a prediction block for the block. As in claim 16 or 17, wherein the availability of the one or more reference samples is identified based on a segmentation history of the block. A device as claimed in any one of claims 16 to 18, wherein the one or more processors are configured to obtain a set of intra-frame prediction modes by: obtaining a first set of intra-frame prediction modes; and adjusting the first set of intra-frame prediction modes to the set of intra-frame prediction modes in response to the availability of the one or more reference samples. A device as in any of claims 16 to 19, wherein the first set of intra-frame prediction patterns is obtained based on an aspect ratio of the block. A signal comprising video data, formed by performing a method as in any one of claims 2 to 15. A computer-readable storage medium having stored thereon instructions for encoding or decoding a video according to the method of any one of claims 1 to 15.

Images