TW201939954A - Apparatus and method for palette decoding - Google Patents

Abstract

A palette decoding apparatus includes a palette color storage device which stores palette colors, a color index storage device which stores color indices of pixels, and a palette value processing circuit which generates a palette value for each pixel by reading data from the color index storage device and the palette color storage device. A frame is divided into first coding units, and each first coding unit is sub-divided into one or more second coding units. Before a palette value of a last pixel in a first coding unit is generated by the palette value processing circuit, a palette value of a non-last pixel in the first coding unit is generated by the palette value processing circuit and used by a reconstruction circuit of the video decoder.

Description

Device and method for color palette decoding

cross reference:

This application claims priority from US Provisional Application 62 / 639,025, filed March 6, 2018, the contents of which are incorporated herein by reference.

The present invention relates to video decoding, and more specifically, to a device and method for palette decoding.

Traditional video codec standards usually use block-based codec technology to take advantage of spatial and temporal redundancy. For example, the basic method is to divide the source frame into multiple blocks (or codec units), perform intra / inter prediction on each block, transform the residuals of each block, and perform quantization and entropy coding. In addition, a reconstructed frame is generated to provide reference pixel data for encoding and decoding subsequent blocks. For some video codec standards, a loop filter can be used to enhance the image quality of the reconstructed frame. The video decoder is used to perform the inverse operation of the video encoding operation performed by the video encoder.

Screen content can be computer-generated content including text, graphics, and animation. For example, screen content can be used in applications such as desktop sharing, video conferencing, social networking, and distance education. Because the proposed image content encoding and decoding technology cannot provide the best encoding and decoding efficiency for screen content, the encoding and decoding of screen content should be different from the encoding and decoding of photographic content. When encoding screen content at the video encoder instead of using it for captured image content by using different codecs (for example, palette codec), an appropriate decoding scheme is required in the video decoder (For example, palette decoding) to reconstruct screen content.

It is an object of the claimed invention to provide an apparatus and method for palette decoding.

According to a first aspect of the invention, an exemplary video decoder is disclosed. An exemplary video decoder includes a palette decoding device having a palette color storage device, a color index storage device, and a palette value processing circuit. The palette color storage device is configured to store palette colors decoded from a bit stream. The color index storage device is arranged to store a color index of a pixel, wherein the color index is decoded from a bit stream. The palette value processing circuit is configured to search the palette color stored in the palette color storage device by reading the color index of each pixel from the color index storage device, and set the each color by the palette color. The palette value of each pixel generates a palette value for each pixel of a plurality of pixels, where the palette color storage device is indexed by the color index of each pixel. A frame is divided into a plurality of first codec units, and each first codec unit is subdivided into one or more second codec units. Before the palette value processing circuit generates the palette value of the last pixel in the first codec unit, the palette value processing circuit generates the palette value of the non-last pixel in the first codec unit, and The video decoder's reconstruction circuit is used.

According to a second aspect of the present invention, an exemplary video decoding method is disclosed. An exemplary video decoding method includes: storing a palette color decoded from a bit stream into a palette color storage device; storing a color index of a pixel into a color index storage device, wherein the color index is decoded from the bit stream ; By reading the color index of each pixel from the color index storage device, searching for the palette color stored in the palette color storage device, and setting the palette of each pixel by the palette color Value, a palette value is generated for each pixel, where the palette color storage device is indexed by the color index of each pixel. A frame is divided into a plurality of first codec units, and each first codec unit is subdivided into one or more second codec units. Before generating the palette value of the last pixel in the first codec unit, a palette value of the non-last pixel in the first codec unit is generated, and the palette value is used by reconstructing the non-last pixel.

According to a third aspect of the invention, an exemplary video decoder is disclosed. An exemplary video decoder includes a palette decoding device having a palette color storage device, a color index storage device, and a palette value processing circuit. The palette color storage device is configured to store palette colors decoded from a bit stream. The color index storage device is configured to store a color index of a pixel, wherein the color index is decoded from a bit stream. The palette value processing circuit is configured to search a palette color indexed by the color index of each pixel stored in the palette color storage device by reading the color index of each pixel from a color index storage device, The palette value of each pixel is set by a palette color, and a palette value is generated for each pixel of a plurality of pixels. A frame is divided into a plurality of first coding units, and each of the first coding units is subdivided into one or more second coding units. All palette colors required until the palette decoding of the first group are stored in the palette color storage device and the color indexes of all pixels of the first group are stored in the color index storage device. Only the board value processing circuit starts to generate the palette value of the first pixel in the first group of the at least one first codec unit. Processing time for generating the palette value of the pixels in the first group according to the data read from the palette color storage device and the color index storage device and decoding the palette of the second group of at least one first codec unit The processing time for writing the required palette color into the palette color storage device and writing the color index of the second group of pixels into the color index storage device overlaps.

These and other objects of the present invention will no doubt become apparent to those having ordinary knowledge in the art after reading the following detailed description of the preferred embodiments shown in the various drawings and illustrations.

Certain terms are used throughout the following description and scope of patent applications, which refer to specific components. As one of ordinary skill in the art will appreciate, electronic device manufacturers may refer to a component by different names. This document is not intended to distinguish between components with different names but different functions. In the following description and the scope of the patent application, the terms "including" and "comprising" are used in an open-ended fashion and should therefore be interpreted to mean "including but not limited to ..." Moreover, the term "coupled" is intended to mean an indirect or direct electrical connection. Thus, if one device is coupled to another device, the connection may be through a direct electrical connection, or through an indirect electrical connection via the other device and the connection.

Palette coding takes advantage of the fact that there are few unique colors in the screen content and attempts to send a palette of these unique colors. The invention proposes a high-performance and low-cost palette decoding scheme in a video decoder. FIG. 1 is a diagram showing a palette decoding device according to an embodiment of the present invention. The palette decoding device 100 may be implemented in a video decoder such as an AV1 decoder. The palette decoding device 100 includes a palette value processing circuit 102, a color index storage device 104, and a palette color storage device 106. Any of the color index storage device 104 and the palette color storage device 106 may be implemented by: On-chip memory such as static random access memory (SRAM), Off-chip such as dynamic random access memory (DRAM) Memory, or a combination thereof.

The entropy decoder 10 in the video decoder receives a bit stream BS generated from the video encoder. The entropy decoder 10 decodes the bit stream BS to obtain the transmitted palette-related information, and provides the decoded palette-related information to the palette decoding device 100. For example, the palette-related information derived from the bit stream BS may include palette size data D_IN1, palette color data D_IN2, and color index data D_IN3. In this embodiment, the palette color storage device 106 is arranged to store the palette colors decoded from the bit stream BS, and the color index storage device 104 is arranged to store the color indexes of the pixels decoded from the bit stream BS, And the palette value processing circuit 102 reads the color index of the pixel from the color index storage device 104, searches for the palette color stored in the palette color storage device 106 (indexed by the color index of the pixel), and The palette color found in the palette color storage device 106 sets the palette value of the pixels to produce the palette value of each pixel. The palette value processing circuit 102 is further configured to generate a palette value output D_OUT to the reconstruction circuit 20 of the video decoder, wherein the palette value output D_OUT includes a palette value required for reconstructing pixels.

In order to achieve high-performance and low-cost palette decoding, the palette decoding device 100 is designed to perform palette decoding in a manner based on a small-size codec unit instead of a large-size codec unit. For example, a frame may be divided into multiple mode info (abbreviated as MI) units, and each MI unit may be divided into one or more transform units (abbreviated as TU). Before determining a palette value of a pixel in another small-size codec unit (for example, another TU), the palette decoding device 100 may start outputting pixels in a small-size codec unit (for example, one TU). For the reconstruction at the reconstruction circuit 20. In other words, before the palette value processing circuit 102 generates the palette value of the last pixel in the large-size codec unit (for example, one MI unit), the large-size codec unit (for example, one MI unit) The palette value of the non-last pixel is generated by the palette value processing circuit 102 and used by the reconstruction circuit 20 of the video decoder. In this way, the palette decoding device 100 does not need a large-sized buffer to keep the palette values of all pixels in one large-sized codec unit (for example, one MI unit), thereby relaxing the buffer requirements. In addition, the reconstruction circuit 20 may start to reconstruct a large-size codec unit (for example, one MI units) to improve overall decoding performance.

FIG. 2 is a flowchart illustrating a palette decoding process involved in reconstructing a pixel value of a pixel in one MI unit of a frame according to an embodiment of the present invention. Assuming the results are essentially the same, the steps need not be performed in the exact order shown in Figure 2. In step 202, the palette decoding device 100 obtains palette size data D_IN1 from the entropy decoder 10. In step 204, the palette decoding device 100 obtains the palette color data D_IN2 from the entropy decoder 10. In step 206, the palette decoding device 100 obtains the color index data D_IN3 from the entropy decoder 10. In step 208, decoding of the residual data is started at the video decoder. In step 210, the palette decoding device 100 checks whether the MI unit uses the palette mode by referring to syntax elements via bit stream BS signaling. If the MI unit does not use the palette mode, the palette decoding process of the MI unit is suspended. If the MI unit uses the palette mode, the flow proceeds to step 212. The palette decoding device 100 determines the palette value of the pixels in the MI unit according to the information given by the palette size data D_IN1, the palette color data D_IN2, and the color index data D_IN3. In step 214, the reconstruction circuit 20 reconstructs the pixels using the color palette values of the pixels in the MI unit.

As described above, the palette decoding device 100 is designed to perform palette decoding in a manner based on a small-sized codec unit to achieve high-performance and low-cost palette decoding. For example, the palette value processing circuit 102 performs step 212 to calculate a palette value of a pixel in a TU-based manner. FIG. 3 is a flowchart showing a TU-based palette value calculation process involved in a palette decoding process according to an embodiment of the present invention. Assuming the results are essentially the same, the steps need not be performed in the exact order shown in Figure 3. In step 302, the palette value processing circuit 102 checks whether there are more than "N" color indexes in the color index storage device 104, where N is an integer and M is the number of all palette values to be calculated in the current TU. , 1≤N≤M. The flow does not proceed to step 304 until the conditions checked in step 302 are satisfied. In step 304, the palette value processing circuit 102 obtains an “O” color index from the color index storage device 104 and obtains the corresponding palette color from the palette color storage device 106, where O is an integer and 1 ≦ O ≦ N. In addition, the reconstruction circuit 200 obtains corresponding residual data. In step 306, the palette value processing circuit 102 calculates an "O" palette value based on the "O" color index and the corresponding palette color. In addition, the reconstruction circuit 200 calculates a reconstruction result based on the palette value determined by the palette value processing circuit 102. In step 308, the palette value processing circuit 102 checks whether the calculated palette value count reaches M. When the calculated palette value count reaches M, this means that the palette value processing circuit 102 has determined all the palette values related to the current TU. Therefore, the palette value calculation processing of the current TU is completed. When the calculated palette value count has not reached M, the flow proceeds to step 302. Further details of the proposed codec unit-based palette decoding scheme (eg, a TU-based palette decoding scheme) are described below.

As described above, the palette color storage device 106 is arranged to store the palette colors decoded from the bit stream BS. In this embodiment, the palette value processing circuit 102 calculates a palette value of a pixel in a TU-based manner. When one MI unit uses the palette mode, the palette decoding of each TU included in the MI unit shares the same palette color (ie, base color) used for palette-encoding the MI unit. FIG. 4 is a diagram showing a plurality of palette color tables stored in the palette color storage device 106 according to an embodiment of the present invention. A MI unit's palette coding can establish a luminance palette color table TB_Y in the Y plane, a chromaticity palette color table TB_U in the U plane, and a chromaticity palette color table TB_V in the V plane. Brightness palette color table TB_Y with palette size N + 1 has 8-bit palette colors Y ₀ -Y _N indexed by color indexes (palette index) 0-N respectively, where 2≤N + 1≤8. In this example, the chromaticity palette color tables TB_U and TB_V may have the same palette size M + 1 but have different palette colors, where 2 ≦ M + 1 ≦ 8. The chromaticity palette color table TB_U having a palette size M has 8-bit palette colors U ₀ -U _M indexed by color indexes (palette index) 0-M, respectively. The chromaticity palette color table TB_V having a palette size M has 8-bit palette colors V ₀ -V _M indexed by color indexes (palette index) 0-M, respectively. The palette colors Y ₀ -Y _N , U ₀ -U _M , V ₀ -V _M and the palette sizes N + 1 and M + 1 of an MI unit are encoded and transmitted via the bit stream BS. Therefore, after decoding the palette colors Y ₀ -Y _N , U ₀ -U _M , V ₀ -V _M and the palette sizes N + 1 and M + 1 for one MI unit from the bit stream BS, Obtain the same luma palette color table TB_Y, chroma palette color table TB_U, and chroma palette color table TB_V used by the palette encoding at the video encoder and store them at the video decoder The color palette is stored in the device 106.

As described above, the color index storage device 104 is arranged to store a color index decoded from the bit stream BS. Regarding the palette encoding at the video encoder, the Y-channel value of the pixel is palette-encoded by using one of the color indexes (palette index) 0-N representing one of the palette colors Y ₀ -Y _N , By using one of the color indexes (palette index) 0-M representing one of the palette colors U ₀ -U _M to palette-encode the U-channel value of the pixel, and by using the representative palette color V Color index (palette index) of one of _0- V _M One of 0-M color-codes the V-channel value of the pixel. For each pixel included in one TU encoded in the palette mode, the YUV color index of the pixel is encoded and transmitted via the bit stream BS. Therefore, after the YUV color index of a pixel is decoded from the bit stream BS, the YUV color index of the pixel is obtained and stored in the color index storage device 104 at the video decoder.

Depending on the palette mode, pixels in one TU can be predicted at the video encoder in wavefront order to allow parallel computation. Therefore, the Y color index of pixels in the same TU can be encoded and transmitted in wavefront order, the U color index of pixels in the same TU can be encoded and transmitted in wavefront order, and the wave can be encoded and transmitted in wavefront order. The preceding sequence encodes and transmits the V color indexes of pixels in the same TU. In this embodiment, the entropy decoder 10 may decode and output the Y color index of pixels in the same TU in wavefront order, may decode and output the U color index of pixels in the same TU in wavefront order, and may The previous order decodes and outputs the V color index of pixels in the same TU. FIG. 5 is a diagram illustrating an operation of writing a color index of a pixel into a color index storage device 104 to create a color index map of a TU according to an embodiment of the present invention. Assume that the size of the current TU to be palette-decoded is 8x8. By sequentially writing the Y color indexes of the pixels in the TU in the same write buffer order as the decoding order, a luminance color index map M_CB_Y of the Y plane of the TU is created in the color index storage device 104. As shown in FIG. 5, the write buffer order is a wavefront order, as indicated by reference numerals.

The Y color index of one pixel is used to indicate the palette color in the brightness palette color table TB_Y. Therefore, the Y color index in the luminance color index map M_CB_Y is read from the color index storage device 104 for calculating a palette value for reconstructing the Y channel values of the pixels in the TU. FIG. 6 is a diagram illustrating an operation of reading a color index of a color index map from a color index storage device 104 according to an embodiment of the present invention. Since the Y color index in the luminance color index map M_CB_Y is used to calculate the palette value used to reconstruct the Y channel values of the pixels in the TU, the Y color index in the luminance color index map M_CB_Y is aligned with the residual data The read buffers in the same output order are read from the color index storage device 104. In FIG. 6, the read buffer order is a column-by-column order indicated by a reference numeral.

Similarly, by sequentially writing the U color indexes of the pixels in the TU to the write buffer in the same write buffer order as the decoding order, the chromaticity color index map M_CB_U of the U plane of the TU is created in the color index storage device 104 The U color indexes sorted in the chromaticity color index map M_CB_U are read from the color index storage device 104 in the same read buffer order as the residual data output order. In addition, by sequentially writing the V color indexes of the pixels in the TU in the same write buffer order as the decoding order, a chromaticity color index map M_CB_V of the V plane of the TU is created in the color index storage device 104, and is compared with the residual The reading buffer sequence in the same material output order reads the V color indexes arranged in the luminance color index map M_CB_V from the color index storage device 104. Since ordinary knowledgeable persons in the field can easily understand the details of the write buffer order and the read buffer order of the chroma color index maps M_CB_U and M_CB_V after reading the above paragraphs for the luminance color index map M_CB_Y, for the sake of simplicity See, further description is omitted here.

When the luma palette color table TB_Y and chroma palette color tables TB_U and TB_V in the palette color storage device 106 are available, and the luma color index map M_CB_Y and chroma color index map M_CB_U in the color index storage device 104 When M_CB_V is available, the palette value processing circuit 102 calculates the color palette of the pixels in one TU based on the color index read from the color index storage device 104 and the palette color stored in the palette color storage device 106. Board value.

FIG. 7 is a diagram illustrating a palette value generation process performed by the palette value processing circuit 102 according to the embodiment of the present invention. The palette value processing circuit 102 outputs a control signal CS to the palette color storage device 106 for selecting one of the brightness palette color table TB_Y and the chroma palette color table TB_U and TB_V, and further controls The signal CS is output to a color index storage device 104 for selecting one of a luminance color index map M_CB_Y and a chrominance color index map M_CB_U and M_CB_V, where the control signal CS is used as a plane selection signal. The palette value processing circuit 102 outputs the index address ADR_IDX to the color index storage device 104 for selecting the color index of the pixel from the selected color index map (ie, the color index map selected by the control signal CS), and searches for The selected palette color table (ie, the palette color table selected by the control signal CS) is used for the palette colors indexed by the selected color index stored at the index address ADR_IDX, and is passed through The palette color found in the selected palette color table sets the pixel's palette value PV.

For example, when the Y channel value of the pixel needs to be reconstructed at this time, the palette value processing circuit 102 updates the control signal CS to select the brightness palette color table TB_Y and the brightness color index map M_CB_Y and updates the index address ADR_IDX to select storage The Y color index of the pixel in the brightness color index map M_CB_Y. Therefore, the palette value PV associated with the reconstruction of the Y-channel value of the pixel is determined and output by the palette value processing circuit 102.

As another example, when the U-channel value of the pixel needs to be reconstructed at this time, the palette value processing circuit 102 updates the control signal CS to select the chroma palette color table TB_U and the chroma color index map M_CB_U, and updates the index Address ADR_IDX to select the U color index of the pixels stored in the chromaticity color index map M_CB_U. Therefore, the palette value processing circuit 102 determines and outputs a palette value PV associated with the U-channel value of the reconstructed pixel.

As another example, when the V-channel value of the pixel needs to be reconstructed at this time, the palette value processing circuit 102 updates the control signal CS to select the chroma palette color table TB_V and the chroma color index map M_CB_V, and updates the index address. ADR_IDX selects the V color index of the pixels stored in the chromaticity color index map M_CB_V. Therefore, the palette value PV associated with the V-channel value of the reconstructed pixel is determined and output by the palette value processing circuit 102.

In this embodiment, the palette value processing circuit 102 is arranged to perform a TU-based palette value calculation process. Therefore, the color index storage device 104 may be implemented by a small-sized buffer having a storage capacity sufficient to buffer the color index required for the TU-based palette value calculation process, and / or the palette color storage device 106 may store therethrough A small-sized buffer implementation with sufficient capacity to buffer the palette colors required for TU-based palette value calculation processing.

As shown in FIG. 7, the palette color storage device 106 stores a brightness palette color table TB_Y and a chromaticity palette color table TB_U, TB_V of a single MI unit, where each TU in the same MI unit has a color palette The board decoding shares the same brightness palette color table TB_Y and chroma palette color table TB_U, TB_V. In one exemplary design, the palette color storage device 106 stores the palette colors required for the palette decoding of each TU in the first MI unit of the frame (eg, the color palette by the first MI unit). The luma palette color table TB_Y and chroma palette color table TB_U, TB_V used by the panel mode, and then decode the required palette color by rewriting the palette of each TU in the first MI unit. At least a part (ie, part or all) of the palette color required for palette decoding of each TU in the second MI unit of the frame (for example, the brightness adjustment used by the palette mode of the second MI unit Swatch color table TB_Y and chroma palette color table TB_U, TB_V). Since the storage capacity of the palette color storage device 106 is equal to the maximum size of palette colors required for palette decoding of one MI unit, the limited storage capacity of the palette color storage device 106 is used to buffer the adjustment of one MI unit. The palette color table used by the swatch mode and reused to buffer the palette color table used by the palette mode of another MI unit.

As shown in FIG. 7, the color index storage device 104 stores the luminance color index map M_CB_Y and chrominance color index maps M_CB_U, M_CB_V of a single TU, where the Y color indexes of all pixels in the same TU are arranged in the luminance color index map M_CB_Y The U color indexes of all pixels in the same TU are arranged in the chroma color index map M_CB_U, and the V color indexes of all pixels in the same TU are arranged in the chroma color index map M_CB_V. In one exemplary design, the color index storage device 104 stores the color indexes of the pixels included in the first TU (for example, the luminance color index map M_CB_Y and chrominance color index maps M_CB_U, M_CB_V of the first TU), and Write at least a part (ie, part or all) of the pixel color index included in the first TU to store the color index of the pixels included in the second TU (for example, the luminance color index map M_CB_Y and the chrominance color index of the second TU Figures M_CB_U, M_CB_V). Since the storage capacity of the color index storage device 104 is equal to the maximum size of the color index of all pixels in a TU, the limited storage capacity of the color index storage device 104 is used to buffer the color index of the pixels in one TU and reused to buffer another Color index of a pixel in a TU. In short, the palette value processing circuit 102 calculates the palette value of a pixel in a TU-based manner, thus allowing the storage capacity of the color index storage device 104 to be smaller than the size of the color index of all pixels in a frame.

Regarding the low-cost palette decoding design in the video decoder, the maximum storage capacity of the palette color storage device 106 may be able to buffer the palette colors used only by the palette mode of a single MI unit, and the color index storage device The maximum storage capacity of 104 may be able to buffer the color index of pixels included in only a single TU. However, this is for illustrative purposes only and is not meant to be a limitation on the present invention. There is a trade-off between cost and performance. In an alternative design, the maximum storage capacity of the palette color storage device 106 may be able to buffer the palette colors used by the palette mode of the K MI units (K≥2) included in the frame to be decoded , And / or the maximum storage capacity of the color index storage device 104 may be able to buffer the color indexes of pixels included in the L TUs (L ≥ 2) of the MI unit to be decoded, where the frames to be decoded may be divided into K ' MI units (K '> K), and the MI units to be decoded can be divided into L' number of TUs (L '> L). For example, the palette color storage device 106 may be implemented by a ping-pong buffer, and / or the color index storage device 104 may be implemented by a ping-pong buffer.

The palette value output D_OUT of the palette decoding device 100 includes a palette value for pixel reconstruction. According to the coding and decoding mode of the TU, reconstructing a pixel in the TU may require the palette value of the pixel, or the residual data of the pixel, or both the palette value and the residual data of the pixel. FIG. 8 is a diagram illustrating a first video decoder according to an embodiment of the present invention. For example, the video decoder 800 may be an AV1 decoder. As shown in FIG. 8, the video decoder 800 includes the above-mentioned entropy decoder (represented by “entropy decoding”) 10, a palette decoding device (represented by “palette decoding”) 100, and a reconstruction circuit (represented by “REC ") 20, and also includes residual decoding circuit (represented by" IS / IQ / IT ") 802, adder circuit 804, multiple multiplexer circuits (represented by" MUX ") 806 and 820, residual buffer 808, deblocking filter (represented by "DF") 810, at least one reference frame buffer 812, intra prediction circuit (represented by "IP") 814, motion vector generation circuit (represented by "MV generation") 816 , And a motion compensation circuit (indicated by "MC") 818.

The palette decoding device 100 can generate a palette value PV of each pixel included in one TU to be reconstructed. The residual decoding circuit 802 can generate residual data (RD) for each pixel included in one TU to be reconstructed. For example, the residual decoding circuit 802 may apply inverse scanning (IS), inverse quantization (IQ), and inverse transform (IT) to the entropy decoding result of the bit stream BS to obtain residual data RD. The adder circuit 840 is arranged to add the palette value PV to the residual data RD to generate an adjusted residual data. The multiplexer circuit 806 includes a first input port, a second input port, a third input port, and an output port. The first input port is arranged to receive the output of the palette decoding device 100 as the mode 0 data D_M0 (for example, D_M0 = PV), the second input port is used to receive the output of the adder circuit 804 as the mode 1 data D_M1 (for example, D_M1 = PV + RD), and the third input port is used to receive the output of the residual decoding circuit 816 as the mode 2 data D_M2 (for example, D_M2 = RD), and the output port is arranged to output one of the mode 0 data D_M0, the mode 1 data D_M1, and the mode 2 data D_M1 to the residual buffer 808.

The multiplexer circuit 806 is controlled by a mode selection signal MS generated from the control logic of the palette decoding device 100. In other words, the mode selection signal MS determines which of the output of the palette decoding device 100, the output of the adder circuit 804, and the output of the residual decoding circuit 816 is stored in the residual buffer 808. The reconstructor 20 reads the stored data from the residual buffer 808 for reconstructing pixels in a TU. When the residual data is not needed for reconstructing the TU, the mode selection signal MS is set by mode 0 (which is a "palette value only" mode), so that the multiplexer 806 passes the palette value PV of the pixels in the TU to the residual The difference buffer 808 is used for reconstruction. When palette values and residual data are required for the reconstruction of the TU, the mode selection signal MS is set by mode 1 (which is a "palette value plus residual data" mode), so that the multiplexer 806 passes the combined palette The results of the values PV and the residual data RD are passed to the residual buffer 808 for reconstruction. When the palette value is not required for reconstructing the TU, the mode selection signal MS is set by mode 2 (which is the "residual data only" mode), so that the multiplexer 806 passes the residual data RD to the residual buffer 808 to For refactoring.

FIG. 9 is a flowchart illustrating a control flow related to a TU in an MI unit of a decoded frame according to an embodiment of the present invention. Assuming the results are essentially the same, the steps need not be performed in the exact order shown in Figure 9. In step 902, decoding of the MI unit is started. In step 904, it is checked whether the MI unit uses the palette mode. If the MI unit does not use the palette mode, the proposed TU-based palette decoding process need not be performed, and the flow proceeds to step 918. At step 918, decoding of the current TU in the MI unit is started. In step 920, the multiplexer 806 obtains the residual data RD from the residual decoding circuit 802, and stores the residual data RD in the residual buffer 808 for reconstruction. In step 922, it is checked whether the current TU is the last TU in the MI unit. When the current TU is the last TU in the MI unit, the decoding of the MI unit ends. When the current TU is not the last TU in the MI unit, the flow proceeds to step 918 to process the next TU in the MI unit.

When step 904 instructs the MI unit to use the palette mode, the flow advances to step 906. At step 906, decoding of the current TU in the MI unit begins. In step 908, it is checked whether the current TU uses skip mode. When the current TU uses skip mode, no information is sent from the video encoder to the video decoder. For example, the bitstream BS does not have codec transform coefficients, has no header, and has no intra / inter prediction information for skip mode TU. Therefore, for the current TU being a skip mode TU, the residual data RD is not generated from the residual decoding circuit 802. In step 910, the multiplexer 806 obtains the palette value PV from the palette decoding device 100, and stores the palette value PV in the residual buffer 808 for reconstruction. It should be noted that under the condition that the current TU is a skip mode TU, the residual data of the current TU is not decoded from the bit stream BS, and the residual buffer 808 is used as a palette for buffering the palette values of the current TU buffer. In step 916, it is checked whether the current TU is the last TU in the MI unit. When the current TU is the last TU in the MI unit, the decoding of the MI unit ends. When the current TU is not the last TU in the MI unit, the flow proceeds to step 906 to process the next TU in the MI unit.

When step 908 indicates that the skip mode is not used for the current TU, the residual data RD is generated from the residual decoding circuit 802 for the current TU of the non-skip mode TU. In step 912, the adder circuit 804 obtains the palette value PV from the palette decoding device 100, and obtains the residual data RD from the residual decoding circuit 802. By way of example and not limitation, when all the residual data RD of the current TU is ready, the residual decoding circuit 802 asserts a signal that the residual data is ready, and the palette decoding device 100 does not perform tuning for generating the current TU. The palette value calculation process of the swatch value PV is performed until the residual decoding circuit 802 asserts the residual data ready signal. At step 914, the multiplexer 806 receives the adjusted residual data from the adder circuit 804 and stores the adjusted residual data into the residual buffer 808 for reconstruction. In step 916, it is checked whether the current TU is the last TU in the MI unit. When the current TU is the last TU in the MI unit, the decoding of the MI unit ends. When the current TU is not the last TU in the MI unit, the flow proceeds to step 906 to process the next TU in the MI unit.

The reconstruction circuit 20 reads the stored data in the residual buffer 808 for reconstruction. In addition, when the TU is encoded using the intra mode or the inter mode, the reconstruction circuit 20 also uses a prediction value PRED, where the prediction value PRED may be an intra mode prediction value determined by the intra prediction circuit 814 or a motion vector The inter prediction value determined by the generating circuit 816 and the motion compensation circuit 818. Specifically, when the current TU is encoded using the intra mode, the multiplexer 820 outputs the intra mode prediction value to the reconstruction circuit 20; when the current TU is encoded using the inter mode, the multiplexer 820 encodes the frame The inter-mode prediction value is output to the reconstruction circuit 20. The deblocking filter 810 is used to perform a deblocking process after reconstruction. Therefore, the reconstructed frame generated from the reconstruction circuit 20 is stored in the reference frame buffer 814 through the deblocking filter 810.

The architecture shown in FIG. 8 includes: using the same residual buffer 808 at different timings to store the mode 0 data D_M0 used by reconstructing the skip mode TU in the palette mode MI unit, Mode 1 data D_M1 used by reconstructing the non-skip mode TU in the palette mode MI unit, and Mode 2 data D_M2 used by reconstructing the TU in the non-palette mode MI unit. In this way, the buffering requirements of the video decoder can be relaxed. However, this is for illustrative purposes only and is not meant to be a limitation on the present invention. In fact, any video decoder using the proposed high-performance and low-cost palette decoding device 100 falls within the scope of the present invention. In an alternative design, both a palette buffer and a residual buffer may be implemented in a video decoder using the proposed high-performance and low-cost palette decoding device 100.

FIG. 10 is a diagram illustrating a second video decoder according to an embodiment of the present invention. For example, the video decoder 1000 may be an AV1 decoder. As shown in FIG. 10, the video decoder 800 includes the entropy decoder 10, the palette decoding device 100, the reconstruction circuit 20, the residual decoding circuit 802, the deblocking filter 810, the reference frame buffer 812, and the intra frame. The prediction circuit 814, the motion vector generation circuit 816, the motion compensation circuit 818, and the multiplexer circuit 820 further include a palette buffer 1002, a residual buffer 1004, and a selection circuit (indicated by "SEL") 1006. The palette buffer 1002 stores a palette value PV of pixels generated from the palette decoding device 100. The residual buffer 1004 is used to store residual data RD of pixels generated from the residual decoding circuit 802. The selector circuit 1006 is controlled by a mode selection signal MS generated by the control logic of the palette decoding device 100, and has a first input port, a second input port, and an output port, wherein the first input port is used to receive a buffer from the palette. The palette value PV read by the processor 1002, the second input port is used to receive the residual data RD read from the residual buffer 1004, and the output port is used to output one of the data outputs of the palette buffer 1002 Or two (ie, the palette value PV read from the palette buffer 1002) and outputting the data of the residual buffer 1004 (ie, the residual data RD read from the residual buffer 1004) Output to the reconstruction circuit 20.

In the first case where the current TU to be decoded is the skip mode TU in the palette mode MI unit, the palette value PV of the current TU is generated from the palette decoding device 100 and stored in the palette In the buffer 1002, for the current TU, no residual data is generated from the residual decoding circuit 802. Since the mode selection signal MS is set by mode 0 (which is a "palette value only" mode), the selector circuit 1006 passes the palette value PV read from the palette buffer 1002 to the reconstruction circuit 20, To reconstruct the current TU.

In the second case where the current TU to be decoded is a non-skip mode TU in the palette mode MI unit, the palette value PV of the current TU is generated from the palette decoding device 100 and stored in the color palette In the board buffer 1002, the residual data RD of the current TU is generated from the residual decoding circuit 802 and stored in the residual buffer 1004. Since the mode selection signal MS is set by the mode 1 (which is a "palette value plus residual data" mode), the selector circuit 1006 reads the palette value PV read from the palette buffer 1002 and the buffer from the residual The residual data RD read by the adder 1004 is passed to the reconstruction circuit 20 to reconstruct the current TU, where the sum of the palette value PV of each pixel and the residual data RD is in the reconstruction circuit 20 implemented by the adder circuit. Everywhere.

In the third case where the current TU to be decoded is a TU in a non-palette mode MI unit, the residual data RD of the current TU is generated from the residual decoding circuit 802 and stored in the residual buffer 1004, and For the current TU, no palette value is generated from the palette decoding device 100. Since the mode selection signal MS is set by the mode 2 (which is a "residual data only" mode), the selector circuit 1006 passes the residual data RD read from the residual buffer 1004 to the reconstruction circuit 20 to reconstruct The current TU.

In the above embodiments, the palette decoding device 100 may perform palette decoding in a manner based on a small-size codec unit (for example, TU) instead of a large-size codec unit (for example, MI). Alternatively, the palette decoding device 100 may perform palette decoding in a pipeline based on a large-size codec unit (for example, MI). Compared to palette decoding performed in a large-size codec unit (eg, MI) without a pipeline, palette decoding performed in a pipeline based on a large-size codec unit (eg, MI) may have improved performance. FIG. 11 is a diagram illustrating a pipeline-based palette decoding process according to an embodiment of the present invention. The color index storage device 104 may be configured to have a storage capacity sufficient to buffer the color indexes of all pixels in one MI group composed of one or more MI units. In addition, the palette color storage device 106 may be configured to have a storage capacity sufficient to buffer the palette colors required for palette decoding of one MI group composed of one or more MI units. In addition, all palette colors required until palette decoding of the MI group are stored in the palette color storage device 106 and the color indexes of all pixels of the MI group are stored in the color index storage device 104, and the palette values The processing circuit 102 then starts to generate the palette value of the first pixel in the MI group composed of one or more MI units.

For clarity and simplicity, it is assumed that each MI group has only a single MI unit. As shown in FIG. 11, the palette color required for palette decoding of the first MI group (consisting of the MI unit MI0) is written into the palette color storage device 106 during the first time period T0, and the first The color indexes of all pixels in an MI group (which is composed of the MI unit MI0) are written into the color index storage device 104 during the first time period T0. The palette colors required for palette decoding in the first MI group (which consists of MI unit MI0) are available in the palette color storage device 106, and the first MI group (which consists of MI units After the color indexes of all pixels in MI0) are available in the color index storage device 104, the palette value processing circuit 102 starts to generate the palette values of the pixels in the first MI group (consisting of MI unit MI0). During the second time period T1, the palette value processing circuit 102 generates pixels in the first MI group (consisting of the MI unit MI0) according to the data read from the palette color storage device 106 and the color index storage device 104. The palette value. In addition, the palette color required for palette decoding of the second MI group (composed of MI unit MI1) is written into the palette color storage device 106 during the second time period T1, and the second MI group (which The color indexes of all pixels in the MI unit (MI1) are written into the color index storage device 104 during the second time period T1. Therefore, the palette colors required for the palette decoding of the second MI group (consisting of the MI unit MI1) are written into the palette color storage device 106 and the The processing time for writing the color indexes of all pixels into the color index storage device 104 and generating the first MI group (consisting of the MI unit MI0) from the data read from the palette color storage device 106 and the color index storage device 104 The processing time of the palette values of the pixels overlaps.

Similarly, the palette colors required for palette decoding in the second MI group (which consists of MI unit MI1) are available in the palette color storage device 106, and the second MI group (which consists of MI unit MI1) After the color indexes of all pixels in the composition) are available in the color index storage device 104, the palette value processing circuit 102 starts to generate the palette values of the pixels in the second MI group (consisting of the MI unit MI1). During the third time period T2, the palette value processing circuit 102 generates pixels in the second MI group (consisting of the MI unit MI1) according to the data read from the palette color storage device 106 and the color index storage device 104. The palette value. In addition, the palette colors required for palette decoding of the third MI group (composed of MI unit MI2) are written to the palette color storage device 106 during the third time period T2, and the third MI group (which The color indexes of all pixels in the MI unit (MI2) are written into the color index storage device 104 during the third time period T2. Therefore, the palette color required for the palette decoding of the third MI group (consisting of the MI unit MI2) is written into the palette color storage device 106 and the color indexes of all pixels in the third MI group are written The processing time to the color index storage device 104 and the palette value of the pixels in the second MI group (consisting of the MI unit MI1) are generated based on the data read from the palette color storage device 106 and the color index storage device 104. Processing time overlaps.

It should be noted that the pipeline-based palette decoding process shown in FIG. 11 is for illustrative purposes only, and is not meant to limit the present invention.

Those of ordinary skill in the art will readily observe that many modifications and changes can be made to the apparatus and method while retaining the teachings of the present invention. Accordingly, the above disclosure should be construed as being limited only by the scope and boundaries of the appended claims.
The above description is only a preferred embodiment of the present invention, and all equivalent changes and modifications made in accordance with the scope of patent application of the present invention shall fall within the scope of the present invention.

10‧‧‧ Entropy Decoder

100‧‧‧ palette decoding device

102‧‧‧ Palette value processing circuit

104‧‧‧Color Index Storage Device

106‧‧‧ Palette Color Storage Device

20‧‧‧ Reconstruction circuit

202 ~ 214, 302 ~ 308, 902 ~ 922‧‧‧ steps

800, 1000‧‧‧ video decoder

802‧‧‧ residual decoding circuit

804‧‧‧adder circuit

806, 820‧‧‧ Multiplexer

808‧‧‧ Residual buffer

810‧‧‧ Deblocking Filter

812‧‧‧Reference frame buffer

814‧‧‧ intra prediction circuit

816‧‧‧Motion vector generation circuit

818‧‧‧Motion compensation circuit

1002‧‧‧ Palette Buffer

1004‧‧‧ Residual buffer

1006‧‧‧Selection circuit

FIG. 1 is a diagram showing a palette decoding device according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating a palette decoding process involved in reconstructing a pixel value of a pixel in one MI unit of a frame according to an embodiment of the present invention.

FIG. 3 is a flowchart showing a TU-based palette value calculation process involved in a palette decoding process according to an embodiment of the present invention.

FIG. 4 is a diagram showing a plurality of palette color tables stored in the palette color storage device shown in FIG. 1 according to an embodiment of the present invention.

FIG. 5 is a diagram illustrating an operation of writing a color index of a pixel into the color index storage device shown in FIG. 1 to create a color index map of a TU according to an embodiment of the present invention.

FIG. 6 is a diagram illustrating an operation of reading a color index of a color index map from the color index storage device shown in FIG. 1 according to an embodiment of the present invention.

FIG. 7 is a diagram showing a palette value generation process performed by the palette value processing circuit shown in FIG. 1 according to an embodiment of the present invention.

FIG. 8 is a diagram illustrating a first video decoder according to an embodiment of the present invention.

FIG. 9 is a flowchart illustrating a control flow related to a TU in an MI unit of a decoded frame according to an embodiment of the present invention.

FIG. 10 is a diagram illustrating a second video decoder according to an embodiment of the present invention.

FIG. 11 is a diagram illustrating a pipeline-based palette decoding process according to an embodiment of the present invention.

Claims (22)

A video decoder includes: Palette decoding device, including: Palette color storage device, used to store multiple palette colors decoded from the bitstream; A color index storage device for storing a plurality of color indexes of a plurality of pixels, wherein the plurality of color indexes are decoded from the bit stream; and The palette value processing circuit is configured to generate a palette value for each pixel of the plurality of pixels in the following manner: Read the color index of each pixel from the color index storage device; Searching for a palette color stored in the palette color storage device, the palette color being indexed by the color index of each pixel; and Set the palette value of each pixel by the palette color; The frame is divided into a plurality of first codec units, each of the plurality of first codec units is subdivided into one or more second codec units, and the first value is generated by the palette value processing circuit. Before the palette value of the last pixel in a codec unit, the palette value of the non-last pixel in the first codec unit is generated by the palette value processing circuit and is repeated by the video decoder.建 电路 用。 Use circuit. The video decoder according to item 1 of the patent application scope, wherein the color index storage device stores a plurality of color indexes of a plurality of pixels included in one of the plurality of second codec units, and then includes At least a part of the plurality of color indexes of a plurality of pixels in the one of the plurality of second codec units to store a plurality of pixels included in another one of the plurality of second codec units Multiple color indexes. The video decoder according to item 2 of the patent application scope, wherein a storage capacity of the color index storage device is equal to a maximum size of a color index of all pixels in a second codec unit. The video decoder according to item 2 of the scope of patent application, wherein each first codec unit is a mode information (MI) unit, and each second codec unit is a transform unit (TU). The video decoder according to item 1 of the patent application scope, wherein a storage capacity of the color index storage device is smaller than a size of a color index of all pixels in a first codec unit. The video decoder according to item 1 of the patent application scope, wherein the palette color storage device stores a palette decoding required for each second codec unit in one of the plurality of first codec units. A plurality of palette colors, and then by rewriting at least a portion of the plurality of palette colors required by the palette decoding of each second codec unit of one of the plurality of first codec units, To store a plurality of palette colors required for the palette decoding of each of the second one of the plurality of first codec units. The video decoder according to item 6 of the patent application scope, wherein the storage capacity of the palette color storage device is equal to the maximum size of multiple palette colors required for palette decoding of a first codec unit . The video decoder according to item 6 of the scope of patent application, wherein each first codec unit is a mode information (MI) unit, and each second codec unit is a transform unit (TU). The video decoder according to item 1 of the patent application scope, wherein the palette decoding device generates a palette value for each pixel in one second codec unit, and generates the palette value in another second codec unit. Generates a palette value for each pixel of the; and the video decoder further includes: A residual decoding circuit for decoding the entropy decoding result of the bit stream to generate residual data for each pixel in the other second codec unit, and for each of the second codec unit Pixel generated residual data; An adder circuit, configured to add the palette value of the each pixel in the other second codec unit and the residual data of the each pixel in the other second codec unit, To generate the adjusted residual data of each pixel in the another second codec unit; Storage devices; and Multiplexer circuit having a first input port, a second input port, a third input port, and an output port, wherein the first input port is used to receive the output of the palette decoding device, and the second input port is used to receive The output of the adder circuit, the third input port is used to receive the output of the residual decoding circuit, and the output port is used to output the output of the palette decoding device and the output of the adder circuit respectively at different timings. And the output of the residual decoding circuit to the storage device; The reconstruction circuit reads the stored data from the storage device for reconstruction. The video decoder described in item 1 of the patent application scope further includes: A first storage device, wherein the palette value processing circuit is further configured to store a plurality of palette values into the first storage device; Second storage device; A residual decoding circuit for decoding an entropy decoding result of the bit stream, generating a plurality of residual data, and storing the plurality of residual data in the second storage device; and The selector circuit has a first input port, a second input port, and an output port, wherein the first input port is configured to receive the plurality of palette values read from the first storage device, and the second input port is set To receive a plurality of residual data read from the second storage device, and the output port is arranged to output one or two of the plurality of palette values and the plurality of residual data to the reconstruction Circuit. A video decoding method includes: Store multiple palette colors decoded from a bit stream into a palette color storage device; Storing multiple color indexes of multiple pixels into a color index storage device, wherein the multiple color indexes are decoded from the bit stream; and Generate a palette value for each of the multiple pixels by: Read the color index of each pixel from the color index storage device; Searching for a palette color stored in the palette color storage device, the palette color being indexed by the color index of each pixel; and Set the palette value of each pixel by the palette color; The frame is divided into a plurality of first codec units, each of the plurality of first codec units is subdivided into one or more second codec units, and the last one of the first codec units is generated Before the palette value of the pixel, the palette value of the non-last pixel in the first codec unit is generated and used for reconstruction of the non-last pixel. The video decoding method according to item 11 of the scope of patent application, wherein storing multiple color indexes of multiple pixels into the color index storage device includes: Storing a plurality of color indexes of a plurality of pixels included in one of the plurality of second codecs; and By rewriting at least a part of the plurality of color indexes of a plurality of pixels included in the one of the plurality of second codec units, the one included in the other of the plurality of second codec units is stored Multiple color indexes for multiple pixels. The video decoding method according to item 12 of the scope of patent application, wherein the storage capacity of the color index storage device is equal to the maximum size of the color indexes of all pixels in a second codec unit. The video decoding method according to item 12 of the scope of patent application, wherein each first codec unit is a mode information (MI) unit, and each second codec unit is a transform unit (TU). The video decoding method according to item 11 of the scope of patent application, wherein a storage capacity of the color index storage device is smaller than a size of a color index of all pixels in a first codec unit. The video decoding method according to item 11 of the scope of patent application, wherein storing the plurality of palette colors decoded from the bit stream into the palette color storage device includes: Storing a plurality of palette colors required for palette decoding of each of the plurality of first codec units; and Storing the plurality of first codecs by rewriting at least a part of the plurality of palette colors required for palette decoding of each of the plurality of first codec units A plurality of palette colors required for palette decoding of each of the second codec units in the other of the units. The video decoding method according to item 16 of the patent application scope, wherein the storage capacity of the palette color storage device is equal to the maximum size of multiple palette colors required for palette decoding of a first codec unit . The video decoding method according to item 16 of the scope of patent application, wherein each first codec unit is a mode information (MI) unit, and each second codec unit is a transform unit (TU). The video decoding method according to item 11 of the patent application scope, wherein a palette value is generated for each pixel in one second codec unit, and a tone is generated for each pixel in another second codec unit Color plate value, the video decoding method further includes: Decode the entropy decoding result of the bit stream to generate residual data for each pixel in the other second codec unit, and generate the data for each pixel in yet another second codec unit Residual data Generating the other first codec unit by adding the palette value of each pixel in the other second codec unit to the residual data of each pixel in the other second codec unit The adjusted residual data of each pixel in the two codec units; and A first input including multiple palette values generated for the one second codec unit, a second input including adjusted residual data generated for the other second codec unit at different timings, and A third input including residual data generated for the yet another second codec unit is stored in the same storage device, where the stored data is read from the storage device to reconstruct the pixels. The video decoding method described in item 11 of the patent application scope further includes: Storing multiple palette values in a first storage device; Decoding the entropy decoding result of the bit stream to generate a plurality of residual data, and storing the plurality of residual data in a second storage device; and Perform pixel reconstruction in the following ways: Reading the plurality of palette values from the first storage device; or Reading the plurality of residual data from the second storage device; or The plurality of palette values are read from the first storage device, and the plurality of residual data are read from the second storage device. A video decoder includes: Palette decoding device, including: Palette color storage device, used to store multiple palette colors decoded from the bitstream; A color index storage device for storing a plurality of color indexes of a plurality of pixels, wherein the plurality of color indexes are decoded from the bit stream; and The palette value processing circuit is configured to generate a palette value for each pixel of the plurality of pixels in the following manner: Read the color index of each pixel from the color index storage device; Searching for a palette color stored in the palette color storage device, the palette color being indexed by the color index of each pixel; and Set the palette value of each pixel by the palette color; The frame is divided into multiple first codec units, and each first codec unit is subdivided into one or more second codec units; Wherein, the palette value processing circuit does not start to generate the palette values of the first pixels in the first group of at least one first codec unit until all the palettes required for the palette decoding of the first group Plate colors are stored in the palette color storage device, and color indexes of all pixels of the first group are stored in the color index storage device; and The processing time for generating multiple palette values of multiple pixels in the first group according to data read from the palette color storage device and the color index storage device and converting at least one first codec unit A plurality of palette colors required for palette decoding of the second group are written into the palette color storage device and multiple color indexes of multiple pixels of the second group are written into the color index storage device Processing time overlaps. The video decoder according to item 21 of the scope of patent application, wherein each first codec unit is a mode information (MI) unit, and each second codec unit is a transform unit (TU).