WO2025250144A1

WO2025250144A1 - High-throughput entropy decoding architecture

Info

Publication number: WO2025250144A1
Application number: PCT/US2024/032108
Authority: WO
Inventors: Dilan Dushane MANATUNGA; Xiaopeng Huang; Yang Yang; Injae Yoo; Andrey Ayupov
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2024-05-31
Filing date: 2024-05-31
Publication date: 2025-12-04
Anticipated expiration: 2026-11-30
Also published as: TW202606226A

Abstract

Methods and systems, including computer-readable media, are described for high-throughput data decoding at an integrated circuit. A system receives addresses that specify memory locations storing encoded data at a memory device. Each of the addresses are provided by a respective decoder of the circuit. For each address: the system generates a request for an encoded block of an encoded data stream that includes multiple encoded blocks. Over multiple clock cycles, the system processes each of the requests corresponding to the addresses based on available buffer space for storing encoded data in at least one of the decoders. In response to processing each request, the system retrieves multiple encoded blocks from non-contiguous memory locations of the memory and decodes the encoded blocks based on an interleaved configuration used to generate a corresponding encoded data stream. A portion of the encoded blocks are decoded in parallel across two or more decoders.

Description

HIGH-THROUGHPUT ENTROPY DECODING ARCHITECTURE

BACKGROUND

[0001] This specification generally relates to data decoding and decompression.

[0002] Data compression can include techniques for reducing the number of bits of input data while still preserving at least some of the same information represented by the input data. Compression can be lossless, in which no information is lost during compression, or lossy, in which some information is lost during compression. Data decompression can include techniques for reversing data compression — receiving input compressed data and generating the data in its original uncompressed form.

[0003] Entropy encoding refers to a lossless data compression technique in which values, or ■’codew ords." of a fixed length are replaced with shorter, symbols of varying length. The mapping between codewords and the symbols can be determined based on a probability distribution of the rate of occurrence of the codewords. For example, more frequently occurring codewords can be replaced with shorter symbols, and less frequently occurring codewords can be replaced with longer symbols.

SUMMARY

[0004] This specification describes an architecture for hardware decoding circuitry and corresponding processes that use the decoding circuitry to implement efficient data decoding (or decompression) operations. The decoding operations are executed in support of high- bandwidth data transfers based on block interleaved coding techniques. The decoding circuitry is implemented on a hardware integrated circuit, such as a System-on-Chip (“SoC”) that communicates with an integrated memory device. The decoding circuitry' can be adjacent (or local) to a special-purpose (or host) processor of the SoC to process encoded (or compressed) data received from the integrated memory device.

[0005] The decoding circuitry includes multiple individual decoders that cooperate to implement the block interleaved decoding techniques based on control signals generated at the hardware integrated circuit. The individual decoders are configured for parallel operation, w hich allows for unique implementations of Huffman coding techniques. The concurrent or parallel operation of the individual decoders, coupled w ith Huffman coding, provides efficient decoding (or decompression) of encoded/ compressed data and enables high- throughput data transfers (e.g.. 32 symbols/cycle) between a hardware processor of the SoC (e.g.. a host) and the memory device. [0006] According to an aspect, there is provided a computer-implemented method comprising receiving multiple addresses that specify memory locations that store encoded data at a memory device, each of the multiple addresses being provided by a respective decoder; for each of the multiple addresses: generating, based on the address, a request for an encoded block of an encoded data stream that includes multiple encoded blocks; over a multiple clock cycles, processing each of the requests corresponding to the multiple addresses based on available buffer space for storing encoded data in at least one of multiple decoders; in response to processing each of the requests, retrieving, from the memory device, multiple encoded blocks from non-contiguous memory' locations of the memory' device; decoding the multiple encoded blocks based on an interleaved configuration used to generate one or more decoded data streams, wherein a portion of the multiple encoded blocks are decoded in parallel across two or more decoders.

[0007] Over the multiple clock cycles, processing each of the requests corresponding to the multiple addresses may comprise, at each of the multiple clock cy cles: selecting a decoder from the multiple decoders based on prioritizing for selection a decoder that has greatest available buffer space among the multiple decoders; and processing a request corresponding to one of the multiple of addresses to retrieve an encoded block for the selected decoder.

[0008] The encoded data can include multiple encoded data streams that have been generated using an entropy encoding scheme.

[0009] The entropy encoding scheme can include a Huffman encoding scheme.

[0010] The multiple encoded data streams may each include a different number of encoded blocks, the encoded blocks may have an identical size.

[0011] The available buffer space for storing encoded data may comprise available space of a first-in-first-out (FIFO) buffer of the decoder.

[0012] Decoding the multiple encoded blocks may comprise: generating, by each of the multiple decoders and in parallel with each other, a decoded data stream based at least on some of the multiple encoded blocks that have been retrieved for the decoder.

[0013] The encoded data stream may comprise parameters for a neural network.

[0014] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0015] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0016] The techniques described in this specification can be used to implement high- bandwidth decoding circuitry that efficiently decompresses machine-learning (ML) data values to enable high-throughput data transfers from a memory device to a host processor on the SoC. The data value decoding operations are performed with improved efficiency over prior approaches for on-device data transfer operations.

[0017] For example, the individual decoders of the decoding circuitry can operate in parallel to implement block interleaved operations based on the Huffman coding technique. The decompression operations of the decoding circuitry complements data compression operations performed by corresponding encoder circuitry. These compression and decompression operations are enabled by an encoder/decoder pipeline that reduces the memory space required at the memory device as well as bandwidth requirements for transferring data from the memory device to a special-purpose processor of the SoC.

[0018] The Huffman compression is used to reduce the overall size of data transfers between the SoC and the memory device. This allows for transferring larger quantities of data, which can improve the compute utilization and overall energy consumption at the hardware integrated circuit. Further, the disclosed techniques provide a block-based decompression (decoding) framework that combines hardware and software approaches to enable randomized tensor accesses and low resource overhead for implementing data transfers between devices of a system. The decoding framework can also be used to efficiently implement spill and fill operations as well as compression and decompression of parameter/weight values for ML w orkloads.

[0019] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Fig. 1 is a block diagram of an example system for implementing high-throughput decoding operations. [0021] Fig. 2 shows an example electronic device that includes a system-on-chip (“SoC”) processor and a memory device.

[0022] Fig. 3 shows an example high-throughput decoding architecture.

[0023] Fig. 4 is an example process for implementing high-throughput decoding operations using the system of Fig. 1.

[0024] Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

[0025] Fig. 1 is a block diagram of an example system 100 for implementing high- throughput decoding operations. Decoding throughput refers to the rate at which compressed, e.g., encoded, data can be decompressed, e.g.. decoded. For example, decoding throughput can be measured in terms of bits per second, data blocks/symbols per clock cycle, or both. Generally, a higher decoding throughput means that the time required to decode a predetermined amount of encoded data can be reduced — or, put another way — a greater amount of encoded data can be decoded within a fixed length of time.

[0026] The system 100 includes a decompression engine 110, a compression engine 120, a memory device 130, and a processor 140. Each of the decompression engine 110 and the compression engine 120 is coupled to the memory device 130 and the processor 140. Each of the decompression engine 110 and the compression engine 120 can be implemented according to any combination of software, firmware, and/or hardware.

[0027] The memory device 130 can include multiple memory dies. For example, the memory device 130 can include N memory die, where N is an integer greater than 1. The memory device 130 can be a dynamic random-access memory' (DRAM) or Double Data Rate (DDR) synchronous DRAM (SDRAM). In some implementations, memory device 130 is configured to perform or support various types of Processing-in-Memory (“PiM”) operations, Computing-in-Memory (“CiM”) operations, and memory -near-computing operations f‘MnC operations”). The memory device 130 performs or supports these operations using its multiple PiM compute elements, which are described below with reference to Fig. 2.

[0028] To achieve high-throughput decoding, the decompression engine 110 includes multiple decoders 110a- 11 On that can each operate on a respective one of the multiple encoded data streams 135a-n. Each encoded data stream 135 that is received by a corresponding decoder 110a- 11 On includes multiple encoded data blocks. In some examples, each of the multiple encoded data blocks have the same size, whereas, in other examples, some (or all) of the multiple encoded data blocks are sized differently.

[0029] The decoders 1 10a-l lOn are configured to operate individually or in parallel with each other to generate a respective one of the multiple decoded data streams 145a-n in response to processing a respective one of the multiple encoded data streams 135a-n. The decoded data streams 145a-n can represent streams of data values and/or operands that are processed at the processor 140 to generate a corresponding output.

[0030] In some implementations, the processor 140 is a ML hardware accelerator configured to implement a neural network and the data values of the decoded data streams 145a-n include inputs to be processed through a neural network layer, a corresponding set of weights for the layer, or both. For example, the inputs and weights represent operands for a ML or neural network computation that is performed at the processor 140 to generate a corresponding ML output.

[0031] In the example of Fig. 1, the results of the computation (e.g., the ML output) are represented as data streams for encoding 155a-n, which are then encoded by compression engine 120 to generate encoded data streams 165a-n. In this example, the results of the computation are encoded/ compressed and routed to be stored at memory device 130 using a high-bandwidth/throughput data transfer operation.

[0032] Like the decompression engine 110, the compression engine 120 also includes multiple encoders 120a- 120n that can each operate on a respective one of the multiple data streams for encoding 155a-n. The encoders 120a-120n are configured to operate individually or in parallel with each other to generate a respective one of the multiple encoded data streams 165a-n based on encoding/compression operations performed on a respective one of the multiple data streams for encoding 155a-n. Each encoded data stream 165 that is generated by a corresponding encoder 120a- 120n includes multiple encoded data blocks. In some examples, each of the multiple encoded data blocks have the same size, whereas, in other examples, some (or all) of the multiple encoded data blocks are sized differently.

[0033] The system 100 can be implemented by any combination of hardware, software, and/or firmware. For example, the system 100 can be part of a system-on-a-chip (SoC) that includes multiple components and devices, where each device can be implemented as an integrated circuit(s) configured for installation on a printed circuit board (PCB). The processor 140 is referred to alternatively as an SoC processor 140. The system 100 can be included in a computing device and is configured to interact with other hardware components of the device, including memory, processors, network components, and/or peripherals. In some implementations, the memory device 130 and/or processor 140 can be coupled within system 100 through a physical interface that includes a slot(s) or socket(s).

[0034] Fig. 2 shows an example electronic device 200 that includes a system-on-chip 240 C’SoC 240”) and the memory device 130. The SoC processor 140 can be any processor or processing device of the SoC processor 240 included in electronic device 200.

[0035] In the example of Fig. 2, the SoC 240 includes a central processing unit 204 (‘‘CPU 204”), a memory controller 205. a memory 206. a resource manager 208. and an IP/circuit block 210. In other examples, the SoC 240 can include fewer or more components, e.g., can include no CPU or IP/circuit block. In some implementations, the electronic device 200 can include multiple SoCs 240 and/or SoC processors 140 and descriptions about the SoC 240 can apply equally to each of the multiple SoCs and/or processors that may be included at electronic device 200.

[0036] The CPU 204 can be a general-purpose CPU (e.g., a single or multi-core CPU). The CPU 204 generates one or more indicators, such as an app-launch indicator or a function call that is triggered in response to executing or launching an application at a user device.

For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory and graphics processing resources to render graphical content of the game. The CPU 204 also generates one or more application values, such as pixel values or frame rate. The application values may be associated with a function call, may be descriptive of an event that occurs during execution of the application, or both.

[0037] The memory 206 is a system memory⁷, shared memory⁷, or both. In the example of Fig. 2, memory 206 is depicted external to circuit block 210. However, memory 206 can include portions of memory that are: i) specific to circuit block 210, ii) external to circuit block 210, or iii) both. The memory 206 can be random access memory of the SoC 240, such as static random access memory⁷ (SRAM), dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0038] In some implementations, aspects of memory 206 are configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more processors of the circuit block 210. Memory 206 can also include various other types of memory⁷, such as high bandwidth memory⁷ (HBM), narrow memory⁷ (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0039] The resource manager 208 is implemented in hardware and software. Aspects of the resource manager 208 can be also implemented as firmware of an SoC processor 140 or firmware of a device of the SoC 240, such as a ML processor, the memory controller 205, or the CPU 204. The resource manager 208 includes control logic implemented in hardware, software, or both. For example, the resource manager 208 can include resources such as flipflops, registers, buffers, etc. that are implemented in hardware and control logic (e.g., programmed code) that is implemented in software. In some implementations, the resource manager 208 is a decoding/encoding resource manager that generates control signals for executing decompression and compression operations of system 100.

[0040] The circuit block 210 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 210 can include an image signal processor (ISP) 212, a host processing unit (HPU) 214, a digital signal processor (DSP) 216, and a graphics processing unit (GPU) 218. The circuit block 210 is referred to alternatively as an IP block 210, where the IP block can include one or more proprietary hardware elements. For example, each of the ISP 212, HPU 214, DSP 216, and GPU 218 can be a respective proprietary' IP block (or IP device) of a particular entity or device manufacturer.

[0041] The HPU 214 can be a special-purpose processor, such as ML hardware accelerator, neural processor unit, neural network (or ML) processor, or another applicationspecific processor. In some implementations, the HPU 214 is a neural network tensor processor (e.g.. a tensor processing unit (TPU)) with an integrated circuit architecture that is optimized for power-efficient execution of ML computations using tensor constructs such as multi-dimensional matrices/tensors.

[0042] One or more aspects of the resource manager 208 can be implemented as a software routine (or module) of the CPU 204, which uses one or more hardware resources of the CPU 204, such as registers, buffers, etc. The CPU 204 can be configured as an instruction and vector data processing engine that processes data obtained from memory device 130 and/or from a system memory of the SoC 202, such as memory^ 206. In some implementations, each processor, e.g., ISP 212, DSP 216, HPU 214, GPU 218, of the SoC 240 includes multiple cores.

[0043] The CPU 204 and/or the resource manager 208 can generate control signals to control and/or manage the operations of one or more other components of the electronic device 200, such as decompression engine 110 and compression engine 120. The control signals are routed at the electronic device 200 using an example bus 222 (“SoC bus 222’') of the SoC 240. The control signals can include commands, requests, data, instructions, or combination of these. [0044] In some implementations, the resource manager 208 cooperates with the CPU 204 and memory controller 205 to dynamically control and manage one or more PiM operations. CiM operations, or both. The operations involve accessing and processing data stored locally at the memory device 130. The PiM and CiM operations are executed locally at the memory device 130 based on control signaling generated at the SoC 240. The control signals are received at the memory’ device 130 and processed locally to trigger or initiate PiM/CiM operations using PiM/CiM compute elements of the memory device 130. The operations can be executed in support of heterogeneous computations that involve two or more processing units of the IP block 210, the CPU 204, or both.

[0045] The memory device 130 can include an example decompression engine 110 that is used to decode or decompress encoded data values that are received at the memory device 130 via an example encoded data stream 250. The encoded data stream 250 can include and/or correspond to some (or all) of the encoded data streams 135a-n.

[0046] The SoC 240 cooperates with the memory device 130 to perform computations using data (e g., encoded data) obtained from memory banks of the memory device 130. The computations can be for operations or workloads that involve one or more of the processors at IP block 210. Additionally, the computations can be for a heterogenous operation that spans multiple processors of IP block 210, multiple IP blocks 210, or both. In some implementations, the heterogeneous operation includes a host device, e g., HPU 214. and the memory device 130 cooperating to execute an example ML workload. In at least one example, the memory device 130 may be external to the SoC 240, whereas in another example the memory' device 130 may be internal to the SoC 240.

[0047] In the example of Fig. 2, the SoC 240 is an integrated circuit of an example user/client device 230, consumer electronic device, or mobile device, where each of these devices can include items such as a smartphone 230a, tablet 230b, laptop 230c, smartwatch or wearable device 230d. The device 230 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer. In some implementations, the SoC 240 and memory device 130 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0048] In the example of Fig. 1, the decompression engine 110 and compression engine

120 is show n as being collocated with processing resources of the IP block 210. However, in other examples, the decompression engine 110 and compression engine 120 can be disposed at other locations of the SoC 240. The decompression engine 110 and the compression engine 120 can be implemented as a set of integrated circuits, or discrete hardware modules. connected to an SoC processor 140 and memory' device 130. For example, the decompression engine 110 and compression engine 120 can be coupled to the memory device 130 through one or more memory channels, communication buses, or circuit interconnects in electronic device 200 or client device 230.

[0049] The system 100 can communicate data to and from the memory device 130 and the SoC processor 140, through the decompression engine 110 and the compression engine 120. The memory device 130 can store data that includes compressed/encoded data values of encoded data streams 165a-n. The compressed data can be data that has been processed and compressed by the compression engine 120 or another component of the system 100, such as a pre-processing engine.

[0050] Multiple encoded data streams 135a-n can be sent from the memory device 130 to the SoC 240 for decoding at the decompression engine 110, and subsequent processing by the SoC processor 140. As indicated above, the processor 140 can be any processor or processing device of the IP block 210 or the SoC 240. The decompression engine 110 receives and processes the multiple encoded data streams 135a-n to generate, using techniques that will be described further below, uncompressed data in the form of multiple decoded data streams 145a-n. A host processor 140 of the SoC 240 receives the multiple decoded data streams 145a-n and performs one or more operations on the multiple decoded data streams 145a-n. [0051] For example, the SoC processor 140 can perform logical or arithmetic operations on the multiple decoded data streams 145a-n or data derived (generated) from the multiple decoded data streams 145a-n, and generate output data. Optionally, the output data, which is similarly in the form of multiple data streams 155a-n, can be sent back to the compression engine 120. The compression engine 120 can compress the multiple data streams 155a-n to generate multiple encoded data streams 165a-n. The memory device 130 can receive and store the multiple encoded data streams 165a-n or data derived (generated) from the multiple encoded data streams 165a-n, which in turn can be sent back for subsequent processing by way of the decompression engine 110.

[0052] The data sent by the memory device 130 can be data for processing any of a variety of different workloads. A workload refers to a computerized task performed by the system 100 on some data. In some implementations, the workload that the system 100 is configured to perform can be to train or execute a neural network. Thus, for example, the data sent by the memory' device 130 can include data defining values associated with the neural network, such as model parameters (e.g., weights and, optionally, bias values for each of multiple layers) of the neural network. As another example, the data can include inputs to the neural network, which will be processed in accordance with the model parameters to generate neural network outputs.

[0053] The memory device 130 can store encoded data blocks/symbols that correspond to model parameters that have been compressed (or encoded) by the compression engine 120.

In these implementations, the data values that are compressed represent model parameters for a trained neural network, which generally remain unchanged when the trained neural network processes inputs to compute an inference.

[0054] During execution of the neural network, the memory device 130 can pass the compressed model parameters that are represented in the form of multiple encoded data streams 135a-n to the decompression engine 110, and the decompression engine 110 can generate uncompressed model parameters that are represented in the form of multiple decoded data streams 145a-n to provide to the SoC processor 140, which then uses as input for performing some operations related to the neural network.

[0055] Each encoded data stream, in turn, includes multiple encoded data blocks. Each of the multiple encoded data blocks has the same size. In some implementations, each of the multiple encoded data blocks can be thirty-two bytes (32 B) in size. Likewise, each decoded data stream, in turn, includes multiple decoded data blocks. Each of the multiple decoded data blocks has the same size. In some implementations, each of the multiple decoded data blocks can be thirty-two bytes (32 B) in size.

[0056] The data can be stored and sent by the memory device 130 according to a variety of different formats. In examples in which the w orkload for the system 100 is to execute or train a neural netw ork, data can be stored and sent as tensors. A tensor is a multi-dimensional array. For example, a zero-dimensional tensor is a scalar value, a one-dimensional tensor is a vector, and a two-dimensional tensor is a matrix.

[0057] Elements of a tensor can be, for example values for different model parameters for a given layer of a neural network, or input or output betw een layers of a neural network or of the neural network. Elements for a tensor, or the data in general, can be in one or more of a variety of different formats, including floating point formats, such as 8. 16, 32, or 64-bit floats, and fixed-precision formats, such as 4, 8. 16. 32. and 64-bit integers. While tensors are provided as one example of a format in which data is transferred, in general any of a variety of different data structures, formats, or representations of data, can be used.

[0058] In these implementations, the SoC processor 140 can perform one or more operations related to executing the neural network on the uncompressed model parameters that are represented in the form of multiple decoded data streams 145a-n, for example matrix multiplication or computing the output to an activation function, and generate, as the output of the one or more operations performed by the neural network, network outputs or intermediate outputs that are represented in the form of multiple data streams 155a-n. An intermediate output of a neural network refers to an output generated by one or more intermediate layers of the neural network, i.e., layers following the input layer but preceding the output layer.

[0059] Optionally, the multiple data streams 155a-n can be sent back to the compression engine 120, which is configured to compress the multiple data streams 155a-n to generate multiple encoded data streams 165a-n. The memory' device 130 can receive and store the multiple encoded data streams 165a-n, which in turn can be sent back for subsequent processing, e.g., for backpropagation computation (in cases where the workload is to train a neural network) or for processing by a subsequent layer of the neural network (in cases where the workload is to execute a neural network), by way of the decompression engine 110.

[0060] The decompression engine 110 and compression engine 120 can implement any of a variety of different techniques for compression and decompression, respectively. For example, the compression engine 120 can be configured to perform an entropy encoding to compress incoming data from the SoC processor 140. Examples of entropy encoding that can be used include Huffman encoding, arithmetic encoding, and asymmetric numeral systems (ANS) encoding, although any of a variety of different entropy encoding techniques can be applied. The decompression engine 110 can be configured to perform an entropy’ decoding (e.g., Huffman decoding, arithmetic decoding, and ANS decoding) that corresponds to the entropy encoding used in the compression engine 120 to decompress incoming data from the memory' device 130.

[0061] Generally, the compression engine 120 can take advantage of the distribution of repeated values in the data executed as part of a workload. In implementations where the workload is to train or execute a neural network, the system 100 can be configured to generate entropy encodings based on the probability distribution of values of the model parameters of the neural network, and can further generate code tables for compressing neural network model parameters on a per-layer basis.

[0062] Fig. 1 shows that the compression engine 120 implements multiple encoders 120a- n for parallel execution of an entropy encoding for input data from the memory' device 130. Likewise, the decompression engine 110 implements multiple decoders 11 Oa-n for parallel execution of an entropy decoding for input data from the SoC processor 140. [0063] More specifically, each of the multiple encoders 120a-n can be configured to execute, in parallel with others of the multiple encoders 120a-n, an entropy encoding of a corresponding one of the multiple data streams 155a-n to generate a corresponding one of the multiple encoded data streams 165a-n. For example, a first encoder 120a included in the compression engine 120 can execute an entropy encoding of a first data stream 155a received from the SoC processor 140 to generate a first encoded data stream 165a.

[0064] Each of the multiple decoders 1 lOa-n can be configured to execute, in parallel with others of the multiple decoders 1 lOa-n, an entropy decoding of a corresponding one of the multiple encoded data streams 135a-n to generate a corresponding one of the multiple decoded data streams 145a-n. For example, a first decoder 110a included in the decompression engine 110 can execute an entropy decoding of a first encoded data stream 135a received from the memory device 130 to generate a first decoded data stream 145a. [0065] Encoding and decoding data in parallel can improve the overall operational efficiency of the system 100, e.g., the encoding (or decoding) power consumption, encoding (or decoding) throughput, or both of the system 100, at least because a greater amount of data can be encoded within a fixed length of time (or, analogously, a greater amount of encoded data can be decoded within a fixed length of time). Encoding and decoding data in parallel allows more data to be transferred between the memory' device 130 or the SoC processor 140. As an example for illustration, if the compression engine 120 compresses data with a compression ratio of 1.5. then a transfer bandwidth saturated at 1 gigabyte/second could transfer up to 1 .5 gigabytes per second of data, by first compressing the data. Storing data in a compressed format in the memory' device 130 can also improve the total storage capacity of the memory' device 130.

[0066] For many workloads, such as training or executing a neural network, however, the throughput bottleneck is not at the memory device 130 or the SoC processor 140, but in the mismatch between the (relative lower) decoding bandwidth and the (relative higher) transfer bandwidth between the memory device 130 or the SoC processor 140. In the case of Huffman encoding, for example, due to the prefix code nature, Huffman decoding is typically a low- bandwidth operation (e.g.. one byte per clock cycle). This can negatively impact the performance of system 100 when Huffman coding is used for some workloads which rely on high-throughput transfers (e.g., thirty⁷ -two bytes per clock cycle) to achieve the target performance.

[0067] Whilst this mismatch could be alleviated by having multiple decoders, e.g.. 32 decoders in the example above, to execute in parallel, there remains yet another challenge in this parallel decoding scheme: although each decoder could potentially output data at a constant rate (e.g., one decoded data block in the size of 32 B per clock cycle), the amount of encoded data required to output the block depends on the compression ratio of the encoded data stream. This can further cause throughput issues, since in order to ensure that the system 100 meets the throughput requirements, each decoder will ideally have the encoded data that it needs to generate a given decoded data block.

[0068] For example, the decompression engine 110 fetches encoded data blocks from the encoded data streams in a round robin ordering. In some instances, an encoded data stream 135a can have a higher compression ratio, while another encoded data stream 135b has a lower compression ratio. If this occurs, then system 100 may experience a performance impact based on differences in the amount of time (or latency) required to fetch data compressed via a high compression ratio relative to data compressed via a low compression ratio.

[0069] For example, decoder 110a can operate on encoded data stream 135a and require an additional encoded data block, however, due to the round robin ordering, the decompression engine 110 will proceed to fetch an encoded data block from encoded data stream 135b for decoder 110b at a given clock cycle. If decoder 110a requires additional time or latency to fetch the high compression ratio data, then decoder 110a will not have enough encoded data blocks for entropy decoding. This can cause delays that impact (or degrade) performance of block interleaved decoding operations at system 100. For example, some delays can cause up to 31 clock cycles to elapse before the additional encoded data block is fetched from encoded data stream 135a for decoder 110a. As indicated above, this can limit or degrade the decoding bandwidth of system 100.

[0070] Aspects of the disclosed techniques can be used to address these performance challenges. More specifically, the techniques provide an innovative decoding architecture that enables the space and bandw idth savings of compression mechanisms like Huffman encoding. For example, the techniques provide a high-throughput decoding architecture that implements an application-specific decompression engine to achieve the data throughput requirements for efficient execution of large ML workloads. The ML workloads can be for generative Artificial Intelligence (“genAI”) applications, such as large language models ("LLMs") or generative computer vision models. Thus, the techniques address the aforementioned challenges, as discussed further below;

[0071] Fig. 3 shows an example high-throughput decoding architecture 300. The components illustrated in Fig. 3 can be implemented by any combination of hardware, software, and/or firmware. For example, the high-throughput decoding architecture 300 can be implemented using a set of integrated circuits that can represent aspects of the decompression engine 110.

[0072] The high-throughput decoding architecture 300 includes a request generator 302, a response router 304, an output selector 306, a decoder selector 308, and multiple decoders 310 (decoder0-decoder31). Each of the multiple decoders 310 can correspond to at least one of the decoders 1 lOa-n described above with reference to Fig. 1. The multiple decoders 310 also correspond respectively to multiple encoded data streams. Decoder 310-0 (decoderO) includes an encoded data buffer 314, a decoded data buffer 316, and an output block counter 318. The descriptions for decoderO apply to each other of the multiple decoders (decoderl- decoder31) included in the high-throughput decoding architecture 300.

[0073] At any given clock cycle, the request generator 302 generates read requests based on the addresses that specify memory locations that store encoded data blocks at the memory device 130. In some implementations, the request generator 302 receives multiple addresses from multiple decoders in parallel, e.g., during a single clock cycle. The request generator 302 is configured to generate multiple read requests based on address information for the multiple addresses received during the clock cycle. For example, the request generator 302 can receive an address from decoder 310-0 in parallel with receiving an address from decoder 310-1 and/or decoder 310-31. The request generator 302 can generate a read request for the address received from decoder 310-0 in parallel with generating a read request for the addresses received from decoder 310-1 and/or decoder 310-31.

[0074] The addresses can be generated by the multiple decoders in a variety of ways. In the example of Fig. 3, decoderO includes a next request address generator 312 that can generate addresses. In some implementations, the next request address 312 is generated based on i) a base memory address indicating an initial data encoded data block in memoiy device 130 that will be the start of an encoded data stream and ii) offset values indicating the (relative) positions of other encoded data blocks in an encoded data stream that corresponds to decoderO. The offset values can be viewed as step distances, which are used to determine memory addresses for encoded data blocks that correspond to the offset values.

[0075] For example, to generate the address for a memory location that stores the next encoded data block in an encoded data stream, the next request address generator 312 in decoderO can compute a product between the offset value for the next encoded data block and the size of the encoded data block. In some implementations, the next request address generator 312 adds the product to a base address for a base memoiy location at the memory device 130, where the base memory location stores the first encoded data block in the encoded data stream.

[0076] The request generator 302 can generate multiple read requests during a single clock cycle. In some implementations, only one of those read requests will be processed by the high-throughput decoding architecture 300 during the clock cycle. That is, only a single encoded data block will be fetched from the memory device to give to a corresponding one of the multiple decoders at each clock cycle. The manner in which the requests are selected by the high-throughput decoding architecture 300 at each clock cycle is explained in more detail below.

[0077] DecoderO is configured to feed a received encoded data block to its encoded data buffer 314 for storage. In some implementations, the decoder 310-0 performs this feed operation in response to receiving an encoded data block from an encoded data stream that: i) corresponds to decoderO and ii) includes multiple encoded data blocks. In the example of Fig. 3, each of the encoded data blocks is 32B (32 bytes). The encoded data buffer 314 can be a first-in-first-out (FIFO) buffer, a circular or ring buffer, or the like, in various implementations. The encoded data buffer 314 can store a threshold or maximum number of encoded data blocks. In some implementations, the threshold storage capacity of the buffer 314 is dynamically adjustable via software or firmware controls of the SoC 240.

[0078] At any given clock cycle, decoderO retrieves one or more encoded data blocks from the encoded data buffer 314 to execute an entropy decoding to generate a decoded data block that have a fixed size (32 B in the example of Fig. 3). The entropy decoding executed by decoderO can correspond to the entropy encoding used to generate the encoded data blocks. The entropy decoding can be, for example, Huffman decoding, arithmetic decoding, or ANS decoding. The rate of this entropy decoding limits the bit rate of data that can pass from the memory device, through the decompression engine, to the SoC processor.

[0079] Having generate a decoded data block, decoderO then supplies the decoded data block to the decoded data buffer 316 for storage. The decoded data buffer 316 can take the form of a first-in-first-out (FIFO) buffer, a circular or ring buffer, or the like, in various implementations.

[0080] The output block counter 318 generates a count of the decoded data blocks as they are forwarded to the decoded data buffer 316 for storage. This count can be used by decoderO to determine when decoderO can cease its execution of the entropy decoding. For example, decoderO can be configured to cease its execution when the total number of decoded data blocks counted by the output block counter 318 reaches a predetermined number. [0081] The output selector 306 cooperates with the decoder selector 308 to select between the multiple decoders to output a decoded data block that has been generated by the selected decoder. In some implementations, the output selector 306 can loop through the multiple decoders across multiple clock cycles and, at each clock cycle, output a decoded data block (if available) that has been generated by one of the multiple decoders.

[0082] For example, the output selector 306 can be implemented as a multiplexer, e.g.. a 32: 1 multiplexer in the example of Fig. 3. that is configured to selectively enable output of the decoded data blocks from the decoded data buffers of the multiple decoders. The multiplexer receives an output signal of the decoder selector 308 as the multiplexer selector input. As a result, the multiplexer selects one of the multiple decoders and outputs a decoded data block from the decoded data buffer of the selected decoder.

[0083] Although the compression engine is configured to compress each of the multiple data streams using the same entropy encoding algorithm, the compression ratios of the multiple data streams might vary from one data stream to another (e.g., depending on the probability distribution of the rate of occurrence of the codewords). Accordingly, the amount of encoded data required to output each decoded data block in a decoded data stream may vary from one data stream to another. For example, at any given clock cycle, generating each decoded data block in a first decoded data stream may require processing two encoded data blocks by a decoder that corresponds to the first decoded data stream, while generating each decoded data block in a second decoded data stream may require processing only one encoded data block by a decoder that corresponds to the second decoded data stream.

[0084] To accommodate this difference in the compression ratios across the multiple data streams, at any given clock cycle, the high-throughput decoding architecture 300 adopts a free-space based arbitration scheme to select between the multiple read requests that have been generated by the request generator 302 during the given clock cycle based on the multiple addresses provided by the multiple decoders (decoder0-decoder31) included in the high-throughput decoding architecture 300. That is, the high-throughput decoding architecture 300 determines which one of the multiple read requests should be processed at any given clock cycle based on available buffer space for storing the encoded data blocks at each of the multiple decoders, so as to ensure that all of the encoded data buffers included in the multiple decoders are at least partially filled at the given clock cycle, i.e., none of the encoded data buffers would remain empty at the next clock cycle.

[0085] In particular, the high-throughput decoding architecture 300 prioritizes for selection of read requests generated for decoders that have the least filled encoded data buffers, i.e., decoders that have the greatest amount of free space in their encoded data buffers, over read requests generated for decoders that have the most filled encoded data buffers, i.e., decoders that have the least amount of free space in their encoded data buffers. [0086] Stated differently, the high-throughput decoding architecture 300 prioritizes for selection of read requests generated for decoders that store the lowest numbers of encoded data blocks in their encoded data buffers. Under this free-space based arbitration scheme, the high-throughput decoding architecture 300 is able to ensure that, at any given clock cycle, a decoder need not have to wait for encoded data block(s) to be fetched from the memory device in order to output a decoded data block during the given clock cycle.

[0087] Optionally, in some implementations, the high-throughput decoding architecture 300 determines which one of the multiple read requests should be processed at any given clock cycle based additionally on the respective IDs associated with the multiple decoders. For example, when lower IDs indicate higher levels of decoding priority, the high-throughput decoding architecture 300 can additionally prioritize for selection of read requests generated for decoders with lower IDs over read requests generated for decoders with higher IDs. [0088] Suppose that, in the example of Fig. 3, at a given clock cycle, the multiple decoders (decod er0-decoder31) each have an encoded data buffer that has the capacity to store a total of five encoded data blocks, and decoderO has one encoded data block stored in its encoded data buffer, while other decoders all have two encoded data blocks stored in their respective encoded data buffers. The request generator 302 generates a read request for each of the multiple decoders. At the given clock cycle, the high-throughput decoding architecture 300 can prioritize for selection of the read request generated for decoderO because it has the least filled encoded data buffer among all decoders and, correspondingly, process the read request to fetch an encoded data block from an encoded data stream that corresponds to decoderO for storage at its encoded data buffer.

[0089] Further, in implementations where the selection of the read requests is based additionally on the respective IDs of the decoders, suppose both decoderO and decoderl have one encoded data block stored in their respective encoded data buffers, while other decoders all have two encoded data blocks stored in their respective encoded data buffers. Then, at the given clock cycle, the high-throughput decoding architecture 300 can prioritize for selection of the read request generated for decoderO because it i) has the least filled encoded data buffer among all decoders and ii) has a lower ID than decoderl which indicates it has a higher level of decoding priority. [0090] By adopting such a free-space based arbitration scheme to select between the multiple read requests at any given clock cycle, the high-throughput decoding architecture 300 is able to rate-match the decoders included in the decompression engine to improve the overall performance of the system, e.g., to reduce decoding latency and increase decoding throughput. For certain workloads, such as executing dense neural networks with millions, or even billions, of model parameters, the increased throughput and reduced latency in transferring data needed by these workloads can be significant.

[0091] Fig. 4 is an example process 400 for implementing high-throughput decoding operations using the system of Fig. 1. Process 400 is also implemented or executed at the system 100 using at least the decompression engine 110 and/or compression engine 120 described above with reference to Fig. 1. Hence, descriptions of process 400 will reference the above-mentioned computing resources of system 100, device 200, and architecture 300. In some examples, the steps or actions of process 400 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non- transitory machine-readable storage device and is executable by one or more of the processors or other resources described in this specification.

[0092] Referring again to process 400, the system 100 receives addresses that specify memory’ locations that store encoded data at a memory device (402). Each of the addresses are provided by a respective decoder of multiple decoders that are included in a decompression engine of the system 100. Each decoder can correspond to an encoded data stream. Each encoded data stream includes multiple encoded data blocks. Each of the multiple encoded data blocks has the same size.

[0093] Each encoded data stream can be stored in a respective portion of the memory' device. For example, each encoded data stream can be stored in multiple contiguous memory’ locations beginning from a respective base memory address at the memory device (that indicates the start of the encoded data stream). Thus, different encoded data blocks from different encoded data streams w ill be stored at non-contiguous memory’ locations of the memory device (but a sequence of encoded data blocks from the same encoded data stream can generally be stored adjacent to each other, i.e., at contiguous memory locations of the memory device).

[0094] In some implementations, the compression ratios of the multiple data streams might vary from one data stream to another (e.g., depending on the probability distribution of the rate of occurrence of the codewords) and, correspondingly, the encoded data streams may not have the same length. As such, the multiple encoded data streams may each include a different number of encoded data blocks. For example, a first encoded data stream that has a relatively higher compression ratio will include a smaller number of data blocks than a second encoded data stream that has a relatively lower compression ratio.

[0095] For each of the addresses, the system 100 generates, based on the address, a read request for accessing (reading) an encoded data block of an encoded data stream that includes multiple encoded blocks (404). The system 100 can generate multiple read requests in parallel, e.g.. during and within a single clock cycle. That is, the system 100 can receive multiple addresses from the multiple decoders during a single clock cycle, and generate multiple read requests for the multiple addresses that have been received during the clock cycle.

[0096] Over multiple clock cycles, the system 100 processes each of the read requests corresponding to the addresses based on available buffer space for storing encoded data blocks in the encoded data buffer of each of the multiple decoders that are included in the decompression engine of the system 100 (406). During a single clock cycle, although the system 100 can generate multiple read requests, only one of those read requests will be processed by the system 100 to access an encoded data block stored in the memory device. [0097] In particular, the system 100 adopts a free-space based arbitration scheme to, at any given clock cycle, select between the multiple read requests that have been generated for the multiple decoders during the given clock cycle. That is, at any given clock cycle, the system 100 selects a decoder from the multiple decoders based on prioritizing for selection a decoder that has greatest available buffer space in its encoded data buffer among the multiple decoders that are included in the decompression engine of the system 100, and then processes a read request corresponding to the address provided by the selected decoder to retrieve an encoded block for the selected decoder. For example, at a given clock cycle, the system 100 can process a read request generated for a decoder that has the least filled encoded data buffer among the multiple decoders.

[0098] Over multiple clock cy cles, and in response to processing a respective one of the read requests at each of the multiple clock cycles, system 100 retrieves multiple encoded data blocks from the memory device (408) and stores the retrieved encoded data blocks in the encoded data buffers of the multiple decoders. For example, the system 100 can store encoded data blocks that are retrieved in response to processing the read requests generated for a particular decoder over multiple clock cycles into the encoded data buffer of the particular decoder, e.g., in the order that they are retrieved. As mentioned above, the system 100 can generally retrieve different encoded data blocks from different encoded data streams at non-contiguous memory locations of the memory device.

[0099] The system 100 decodes the encoded blocks using the multiple decoders to generate multiple decoded data streams (410). In some implementations, the system 100 can generate a decoded data stream corresponding to each encoded data stream, and the number of the decoded data streams can be the same as the number of the encoded data streams. Like the encoded data streams, each decoded data stream includes multiple decoded data blocks. Each of the multiple decoded data blocks has the same size.

[00100] In particular, each of the multiple decoders are configured to execute an entropy decoding on the encoded data blocks stored in the encoded data buffer of the decoder in parallel with others of the multiple decoders, such that at least some of the encoded data blocks are decoded in parallel across two or more decoders. Thus, for example, up to N decoded data blocks can be generated across a total of N decoders at every clock cycle.

[00101] Each of the multiple decoders that are included in the decompression engine of the system 100 can be configured to execute any entropy decoding that corresponds to an entropy encoding used to generate the encoded data blocks. The entropy decoding can be, for example, Huffman decoding, arithmetic decoding, or ANS decoding.

[00102] In some implementations, the multiple encoded data blocks within each encoded data stream can be interleaved (i.e., reordered or shuffled) based on an interleaved configuration. An interleaved configuration defines a mapping or a function that specifies how the encoded data blocks in the encoded data stream should be reordered. In these implementations, each of the multiple decoders can be configured to operate based on (an inverse of) the interleaved configuration to generate the decoded data blocks.

[00103] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.

[00104] Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[00105] The term “computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[00106] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[00107] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[00108] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (General purpose graphics processing unit).

[00109] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. [00110] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[00111] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[00112] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application serv er, or that includes a front end component, e.g.. a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[00113] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[00114] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[00115] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[00116] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

What is claimed is:

1. A method, comprising: receiving a plurality of addresses that specify memory locations that store encoded data at a memory device, each of the plurality of addresses being provided by a respective decoder; for each of the plurality of addresses: generating, based on the address, a request for an encoded block of an encoded data stream that includes a plurality of encoded blocks; over a plurality' of clock cycles, processing each of the requests corresponding to the plurality of addresses based on available buffer space for storing encoded data in at least one of a plurality of decoders; in response to processing each of the requests, retrieving, from the memory device, a plurality of encoded blocks from non-contiguous memory locations of the memory device; decoding the plurality of encoded blocks based on an interleaved configuration used to generate one or more decoded data streams, wherein a portion of the plurality of encoded blocks are decoded in parallel across two or more decoders.

2. The method of claim 1, wherein over the plurality' of clock cycles, processing each of the requests corresponding to the plurality of addresses comprises, at each of the plurality' of clock cycles: selecting a decoder from the plurality of decoders by prioritizing selection of a decoder that has a greatest available buffer space among the plurality of decoders; and processing a request corresponding to one of the plurality of addresses to retrieve an encoded block for the selected decoder.

3. The method of any one of claims 1-2, wherein the encoded data comprises a plurality of encoded data streams that have been generated using an entropy encoding scheme.

4. The method of claim 3, wherein the entropy encoding scheme comprises a Huffman encoding scheme.

5. The method of any one of claims 3-4, wherein the plurality of encoded data streams each include a different number of encoded blocks, the encoded blocks having an identical size.

6. The method of any one of claims 1-5, wherein the available buffer space for storing encoded data comprises available space of a first-in-first-out (FIFO) buffer of the decoder.

7. The method of any one of claims 1-6, wherein decoding the plurality of encoded blocks comprises: generating, by each of the plurality of decoders and in parallel with each other, a decoded data stream based at least on some of the plurality of encoded blocks that have been retrieved for the decoder.

8. The method of any one of claims 1-7, wherein the encoded data stream comprises parameters for a neural network.

9. A system comprising: one or more processing devices; and one or more non-transitory machine-readable storage devices for storing instructions that are executable by the one or more processing devices to cause performance of operations comprising: receiving a plurality⁷ of addresses that specify memory⁷ locations that store encoded data at a memory device, each of the plurality⁷ of addresses being provided by a respective decoder; for each of the plurality⁷ of addresses: generating, based on the address, a request for an encoded block of an encoded data stream that includes a plurality of encoded blocks; over a plurality of clock cycles, processing each of the requests corresponding to the plurality of addresses based on available buffer space for storing encoded data in at least one of a plurality of decoders; in response to processing each of the requests, retrieving, from the memory⁷ device, a plurality⁷ of encoded blocks from non-contiguous memory locations of the memory device; decoding the plurality of encoded blocks based on an interleaved configuration used to generate one or more decoded data streams, wherein a portion of the plurality of encoded blocks are decoded in parallel across two or more decoders.

10. The system of claim 9, wherein over the plurality⁷ of clock cycles, processing each of the requests corresponding to the plurality of addresses comprises, at each of the plurality of clock cycles: selecting a decoder from the plurality of decoders based on prioritizing for selection a decoder that has greatest available buffer space among the plurality⁷ of decoders; and processing a request corresponding to one of the plurality of addresses to retrieve an encoded block for the selected decoder.

11. The system of any one of claims 9-10, wherein the encoded data comprises a plurality of encoded data streams that have been generated using an entropy encoding scheme.

12. The system of claim 11, wherein the entropy encoding scheme comprises a Huffman encoding scheme.

13. The system of any one of claims 11-12, wherein the plurality of encoded data streams each include a different number of encoded blocks, the encoded blocks having an identical size.

14. The system of any one of claims 9-13, wherein the available buffer space for storing encoded data comprises available space of a first-in-first-out (FIFO) buffer of the decoder.

15. The system of any one of claims 9-14, wherein decoding the plurality⁷ of encoded blocks comprises: generating, by each of the plurality of decoders and in parallel with each other, a decoded data stream based at least on some of the plurality of encoded blocks that have been retrieved for the decoder.

16. The system of any one of claims 9-15, wherein the encoded data stream comprises parameters for a neural network.

17. A non-transitory machine-readable storage device storing instructions that are executable by a processing device to cause performance of operations comprising: receiving a plurality of addresses that specify memory locations that store encoded data at a memon device, each of the plurality of addresses being provided by a respective decoder; for each of the plurality of addresses: generating, based on the address, a request for an encoded block of an encoded data stream that includes a plurality of encoded blocks; over a plurality of clock cycles, processing each of the requests corresponding to the plurality of addresses based on available buffer space for storing encoded data in at least one of a plurality of decoders; in response to processing each of the requests, retrieving, from the memory device, a plurality of encoded blocks from non-contiguous memory locations of the memory device; decoding the plurality of encoded blocks based on an interleaved configuration used to generate one or more decoded data streams, wherein a portion of the plurality of encoded blocks are decoded in parallel across two or more decoders.

18. The non-transitory machine-readable storage device of claim 17, wherein over the plurality of clock cycles, processing each of the requests corresponding to the plurality of addresses comprises, at each of the plurality of clock cycles: selecting a decoder from the plurality of decoders based on prioritizing for selection a decoder that has greatest available buffer space among the plurality of decoders; and processing a request corresponding to one of the plurality of addresses to retrieve an encoded block for the selected decoder.

19. The non-transitory machine-readable storage device of any one of claims 17-18, wherein the encoded data comprises a plurality⁷ of encoded data streams that have been generate using an entropy encoding scheme.

20. The non-transitory machine-readable storage device of claim 19, wherein the entropy encoding scheme comprises a Huffman encoding scheme.