TW202414200A - Reducing memory bank conflicts in a hardware accelerator - Google Patents

Abstract

Methods and systems, including computer-readable media, are described for reducing or preventing memory bank conflicts in a hardware accelerator to allow for concurrent access of memory banks at a hardware accelerator. A compute tile of the hardware accelerator receives requests that are used to access a tile memory of the accelerator. For each of the requests: a logical address represented by a sequence of bits is identified in the request and a first subset of bits is obtained from the sequence. An identifier is generated based on a bank generation function that uses the first subset of bits. The identifier identifies a particular bank among physical memory banks of the tile memory. Each request is processed using the respective bank identifier that is generated for that request. Multiple distinct memory banks are accessed concurrently during the same clock cycle in response to processing the requests.

Description

Reduce memory bank conflicts in hardware accelerators

This specification generally relates to the operation of memory in a hardware integrated circuit.

A neural network is a machine learning model that uses one or more layers of nodes to produce an output (e.g., a classification) for a received input. Some neural networks include one or more hidden layers in addition to an output layer. Some neural networks may be convolutional neural networks (CNNs) configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing. Different types of neural network architectures can be used to perform a variety of tasks related to classification or pattern recognition, prediction involving data modeling, and information clustering.

A neural network layer may have a corresponding set of parameters or weights. The weights are used to process inputs (e.g., a batch of inputs) through the neural network layer to produce a corresponding output of the layer for use in computing a neural network inference. A batch of inputs and a core set may be represented as a tensor (i.e., a multidimensional array) of inputs and weights. A hardware accelerator is a dedicated integrated circuit for implementing a neural network. The circuit includes memory with locations corresponding to elements of a tensor that can be traversed or accessed using the control logic of the circuit.

This document describes techniques for reducing (or preventing) memory bank conflicts at the on-chip memory of a hardware accelerator to allow simultaneous access to their physical memory banks.

A computing chip of a hardware accelerator receives a request for accessing a chip memory of the accelerator. For each of the requests: i) identifying a logical address represented by a bit sequence in the request; ii) obtaining a first bit subset from the sequence; and iii) generating an identifier based on a library generation function using the first bit subset.

The identifier identifies a particular bank of physical memory banks of chip memory. Each request is processed using a respective bank identifier ("bank ID") generated for the request. In response to processing the request, multiple distinct memory banks are accessed simultaneously during the same clock cycle (eg, a single clock cycle).

An aspect of the subject matter described in the specification may be embodied in a computer-implemented method for simultaneously accessing a memory bank of a hardware accelerator. The method includes receiving a plurality of requests, wherein each request is for accessing a chip memory of the hardware accelerator. For each of the plurality of requests, the method includes: identifying a respective logical address represented by a bit sequence in the request; obtaining a first bit subset from the bit sequence; and generating a respective bank identifier identifying a particular bank among a plurality of physical memory banks of the chip memory based on a bank generation function using the first bit subset. The method further includes: i) processing each of the plurality of requests using the respective bank identifier generated for the request; and ii) in response to processing each of the plurality of requests, simultaneously accessing a plurality of different physical memory banks of the chip memory during a clock cycle.

These and other embodiments may each include one or more of the following features as desired. For example, in some embodiments, accessing the multiple different physical memory banks simultaneously includes accessing the multiple different physical memory banks during a single clock cycle for a specific stride value. The specific stride value may be a memory access stride equal to a difference between specific rows of the same physical memory bank.

In some embodiments, accessing the multiple different physical memory libraries simultaneously includes: accessing the multiple different physical memory libraries without a library conflict. In one aspect, the library conflict is when two or more requesters request access to the same physical memory library of the chip memory during the same clock cycle. Each physical memory library may include multiple columns and the method includes: for each of the multiple requests: i) obtaining a second bit subset from the bit sequence; ii) providing the second bit subset as an input to the library generation function; and iii) based on applying the library generation function to the second bit subset, generating a respective column identifier identifying a specific column in the specific library among the multiple physical memory libraries of the chip memory.

In some implementations, each row of the plurality of rows comprises a width of 16 bytes; a partition of the chip memory comprises 32 physical memory banks; and the memory access stride is equal to: row_width * num_banks. Obtaining a first bit subset from the bit sequence may comprise: obtaining two or more bits among the least significant bits (LSBs) in the bit sequence.

Other implementations of this aspect and other aspects include corresponding systems, devices, and computer programs encoded on computer storage devices that are configured to perform the actions of the method. A system of one or more computers can be so configured by software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform the actions during operation. One or more computer programs can be so configured by having instructions that, when executed by a data processing device, cause the device to perform the actions.

The subject matter described in this specification may be implemented in specific embodiments to achieve one or more of the following advantages: A bank generation function is disclosed that can be used to process multiple requests for memory resources of a chip memory without triggering a bank conflict.

The library generation function is configured to generate a respective library ID for each request in a request group so that no two requests will need to access the same physical memory library of chip memory. The library generation function provides an access pattern that allows each request in the group to be processed in parallel during the same clock cycle (e.g., a single clock cycle). The access pattern provided by the library generation function allows multiple requests to be processed simultaneously regardless of the stride value used for memory access.

Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages of the subject matter will be apparent from the description, drawings, and the scope of the invention application.

FIG1A is a block diagram of an exemplary computing system 100 for implementing a neural network model at a hardware integrated circuit (such as a machine learning hardware accelerator). The computing system 100 includes one or more computing chips 101, a host 120, and a higher-level controller 125 ("controller 125"). As described in more detail below, the host 120 and the controller 125 cooperate to provide data sets and instructions to the one or more computing chips 101 of the system 100.

In some embodiments, the host 120 and the controller 125 are the same device. The host 120 and the controller 125 may also perform different functions but be integrated into a single device package. For example, the host 120 and the controller 125 may form a central processing unit (CPU) including multiple computing chips 101 that interacts or cooperates with a hardware accelerator. In some embodiments, the host 120, the controller 125, and the multiple computing chips 101 are included or formed on a single integrated circuit die. For example, the host 120, the controller 125, and the multiple computing chips 101 may form a dedicated system-on-chip (SoC) optimized to execute a neural network model for processing machine learning workloads.

Each computing chip block 101 generally includes a controller 103, which provides one or more control signals 105 to cause an input vector 102 to be stored (or activated) at a memory location of a first memory 108 ("memory 108") or accessed from the memory location. Similarly, the controller 103 may also provide one or more control signals 105 to cause weights (or parameters) of a matrix structure of weights 104 to be stored at a memory location of a second memory 110 ("memory 110") or accessed from the memory location. In some embodiments, the input vector 102 is obtained from an input tensor, and the matrix structure of weights is obtained from a parameter tensor. Each of the input tensor and the parameter tensor can be a multi-dimensional data structure, such as a multi-dimensional matrix or tensor. This is described in more detail below with reference to FIG. 6 .

Each memory location of the memory 108, 110 may be identified by a corresponding memory address (e.g., a logical address having a corresponding mapping to a physical row of a physical memory bank of the memory). Referring to the example of FIG. 1B , a computing chip block 101 may derive a set of consecutive addresses (e.g., virtual/logical addresses) from a group of requests 130. For example, the set of consecutive addresses may be derived with reference to a logical memory corresponding to the physical memory 108 of the computing chip block 101. This is also described below with reference to the embodiments of FIGS. 4 and 5 .

The logical memory has a plurality of logical ports 135 and each port may be connected to or associated with a different requestor requesting access to physical resources of the memory 108. To process a given access request, for each port, the chip block 101 (or its controller 103) determines a library to which the request is to be routed based on an address in the request. For each library, the computing chip block 101 may include an arbitrator 140 that arbitrates access from multiple ports to the library (for example) based on a unique library generation function that is configured to mitigate some (or all) requests to be routed to the same physical memory library. The logical memory, the ports of the logical memory, and the arbitrator may be implemented in software, hardware, or both. In some implementations, the logic memory and its ports and the arbiter are controlled based on control signals generated by the controller 103.

Each of memories 108, 110 may be implemented as a series of physical libraries, units, or any other related storage media or devices. Each of memories 108, 110 may include one or more registers, buffers, or both. In some embodiments, memory 108 is an input/activation memory and memory 110 is a parameter memory. In some other embodiments, inputs or activations are stored in memory 108, memory 110, or both; and weights are stored in memory 108, memory 110, or both. For example, inputs and weights may be transmitted between memory 108 and memory 110 to facilitate a particular neural network operation. In some implementations, each of memory 108 and memory 110 is referred to as on-chip memory.

Each computing chip block 101 also includes an input enable bus 106, an output enable bus 107, and an operation unit 112 including one or more hardware multiply-accumulate circuits (MACs) in each cell 114 a/b/c. The controller 103 can generate control signals 105 to obtain the operation elements stored at the memory of the computing chip block 101. For example, the controller 103 can generate control signals 105 to obtain: i) an example input vector 102 stored at the memory 108; and ii) the weight 104 stored at the memory 110. Each input obtained from memory 108 is provided to input enable bus 106 for routing (e.g., direct routing) to a computation cell 114a/b/c in computation unit 112. Similarly, each weight obtained from memory 110 is routed to a cell 114a/b/c in computation unit 112.

As described below, each cell 114a/b/c performs an operation that generates a partial sum or accumulated value used to generate an output of a given neural network layer. An activation function can be applied to a set of outputs to generate a set of output activations for the neural network layer. In some embodiments, the outputs or output activations are routed via the output activation bus 107 for storage and/or transmission. For example, a set of output activations can be transmitted from a first computing chip block 101 to a second, different computing chip block 101 to be processed at the second computing chip block 101 as input activations for a different layer of the neural network.

In general, each computing chip block 101 and system 100 may include additional hardware structures to perform operations associated with multi-dimensional data structures (such as tensors, matrices and/or data arrays). In some embodiments, inputs for an input vector (or tensor) 102 and weights 104 for a parameter tensor may be preloaded into the memory 108, 110 of the computing chip block 101. The inputs and weights are received as sets of data values from a host 120 (e.g., an external host) via a host interface or from a higher-level control (such as controller 125) to a particular computing chip block 101.

Each of the computing chip block 101 and the controller 103 may include one or more processors, processing devices, and various types of memory. In some embodiments, the processor of the computing chip block 101 and the controller 103 includes one or more devices, such as a microprocessor or central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a combination of different processors. Each of the computing chip block 101 and the controller 103 may also include other computing and storage resources (such as buffers, registers, control circuit systems, etc.). These resources work together to provide additional processing options for performing one or more of the determinations and calculations described in this specification.

In some implementations, the processing unit(s) of the controller 103 executes programmed instructions stored in the memory to cause the controller 103 and the computing chip block 101 to perform one or more functions described in this specification. The memory of the controller 103 may include one or more non-transitory machine-readable storage media. The non-transitory machine-readable storage medium may include solid-state memory, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information or instructions.

System 100 receives instructions defining a particular computational operation to be performed by a computing chip block 101. In some implementations, a host may generate sets of parameters (i.e., weights) and corresponding inputs for processing at a neural network layer. For example, a host may generate sets of compressed parameters (CSPs) and corresponding mapping vectors (e.g., a non-zero mapping (NZM)) for a given operation that maps a neural network input to a non-zero parameter in a set of CSPs. Host 120 may send the parameters to a computing chip block 101 via a host interface for further processing at the chip block. Controller 103 may execute programmed instructions to analyze a data stream associated with received weights and inputs including compressed parameters and corresponding mapping vectors.

Controller 103 causes the input of data streams and the storage of weights at compute chip block 101. For example, controller 103 may store the mapping vectors and compressed sparse parameters in local chip block memory of compute chip block 101. This is described in more detail below. Controller 103 may also analyze the input data stream to detect an operation code ("operation code/opcode"). System 100 may support various types of operation codes, such as operation code types indicating vector matrix multiplication, element-by-element vector operations, and whether a given operation uses compressed sparse parameters and uncompressed parameters/weights.

Based on one or more opcodes, the controller 103 may initiate or execute a library generation function (described below) to arbitrate requests for access to a chip memory of a computing chip 101. For example, the controller 103 utilizes the library generation function to arbitrate two or more requests so that different physical libraries for chip memory process each of the two or more requests. The controller 103 may employ a predetermined library selection scheme that is programmed or coded at the controller 103 before performing inference decisions at the computing chip 101.

In some implementations, a given computational operation involves multiple requestors/accessors each requiring access to resources of memory 108. For example, a computational workload executed at compute die 101 may trigger memory access requests due to tensor traversal operations requiring read and write access to respective address locations of memory 108. As described below, these address locations may correspond to elements of an input tensor processed as part of the workload.

In addition to tensor read/write operations, processing a workload may also involve processing read or write access requests to: i) move data (e.g., parameters) from memory 108 (narrow) to memory 110 (wide); and ii) move data from memory 110 (wide) to memory 108 (narrow). In some cases, to process an example workload, a first computing chip block 101 arbitrates and executes access requests (e.g., read/write requests) to memory 108, where the requests are based on external data communications originating from outside the first computing chip block 101.

These different types of access requests correspond to one or more computation threads that all need to access the same physical memory 108. In some cases, multiple requests may correspond to a single computation thread (or clock cycle). The disclosed library generation function can be used to handle multiple requests for memory resources of a chip memory without triggering a library conflict. For example, the library generation function is configured to generate a separate library ID for each request in a request group so that no two requests will need to access the same physical memory library of the chip memory.

In some implementations, based on the bank identifier returned as an output of the bank generation function, the controller 103 is able to arbitrate requests by processing two or more requests for different physical memory banks of the die memory during the same clock cycle (e.g., a single clock cycle). The bank generation function allows the controller 103 to achieve these advantages even for particularly problematic strides (e.g., memory accesses where the stride varies depending on the number of banks and the number of bytes in a row of banks). This is described in more detail below.

In addition, based on the opcode, the controller 103 may activate dedicated data path logic associated with one or more computational cells 114 a/b/c to perform operations (e.g., coefficient operations) using parameters and inputs/activations and corresponding mapping vectors for mapping the inputs/activations to a subset of parameters. As used herein, sparse operations include neural network operations performed on a neural network layer using non-zero weight values in a set of compressed sparse parameters that are generated from a set of weights for the neural network layer.

In some implementations, the opcode indicates details about the operation associated with the inputs and weights of a given layer, such as the sparsity of one or more parameter tensors associated with the layer. The controller 103: i) detects the opcode, including any relevant tensor sparsity information; ii) uses local read logic to obtain parameters from chip memory (e.g., memory 108 or 110) based on the opcode; and iii) writes or routes those parameters to cells 114 a/b/c of the computing chip 101. The controller 103 may also analyze an example data stream and based on the analysis, generate a set of compressed sparse parameters and a corresponding mapping vector that maps discrete inputs of an input vector to respective non-zero weight values in the compressed sparse parameters. To the extent that operations and/or procedures for generating compressed sparse parameters and corresponding mapping vectors are described with reference to the controller 103, each of those operations and procedures may also be performed by the host 120, the controller 125, or both.

In some implementations, performing some (or all) operations (such as analyzing tensor indexes), performing direct memory access (DMA) operations to read address space in system memory (e.g., SRAM, DRAM, etc.) to obtain input and weight values, generating compressed sparse parameters, and generating corresponding mapping vectors at the host 120 allows for reduced processing time at each computing chip block 101 and improved data processing throughput at the system 100. For example, performing these operations at the host 120 using the controller 125 allows a set of compressed parameters to be sent to a given chip block computing 101, which reduces the size and amount of data routed at the system 100.

2 shows an example processing pipeline 200 for routing inputs and outputs between a memory and computational cells in a hardware integrated circuit. In general, pipeline 200 uses input bus 106 to route inputs obtained from a memory location in memory 108 to one or more computational cells 114 and uses output bus 107 to route outputs resulting from multiplications performed at one or more computational cells 114 to a memory location in memory 108.

Pipeline 200 utilizes a hardware architecture in which input bus 106 is coupled (e.g., directly coupled) to each of a plurality of hardware operation cell groups of dedicated integrated circuits. System 100 may provide a first operator from a location in memory 108 and a second operator from a location in memory 110. Operator 204 is used for an operation performed at cell 114 and cell 114 produces an output corresponding to a result of the operation. In some embodiments, the operation is used for a machine learning operation, such as for processing inputs through a neural network layer of an artificial neural network.

In this embodiment, a computing chip block 101 may provide a first operator corresponding to an input or activation (e.g., a0, a1, a2, etc.) of an input feature map to a subset of cells 114. For example, a respective input of an input vector 102 is provided to each MAC in the subset via the input bus 106 of the computing chip block 101. The system 100 may perform this broadcast operation across multiple computing chips 101 to calculate a product of a given neural network layer using respective input groups and corresponding weights at each computing chip block 101. At a given computing chip block 101, the product is calculated by multiplying a respective input (e.g., a1) at each MAC in the subset with the corresponding weight (e.g., w1) using the multiplication circuitry of the MAC.

The system 100 may generate an output of a layer based on the accumulation of a plurality of respective products operated at each MAC of a cell 114a/b/c in a subset of cells 114a/b/c of an operation unit 112. As explained below with reference to FIG. 6 , a multiplication operation performed within an operation chip block 101 may involve: i) a first operand (e.g., an input or activation) stored at a memory location of the memory 108, which corresponds to a respective element of an input tensor; and ii) a second operand (e.g., a weight) stored at a memory location of the memory 110, which corresponds to a respective element of a parameter tensor.

In the example of FIG. 2 , a shift register 202 may provide shift functionality, wherein an input of an operator 204 is broadcast onto the input bus 106 and routed to one or more MACs of the cell 114. In some implementations, the shift register 202 enables one or more input broadcast modes at a computing die 101. For example, the shift register 202 may be used to broadcast inputs sequentially (one by one) from the memory 108 (a first broadcast mode), broadcast inputs simultaneously (e.g., in parallel) from the memory 108 (a second broadcast mode), or broadcast inputs using some combination of these broadcast modes. The shift register 202 may be an integrated function of the memory 108 and may be implemented in hardware, software, or both.

In some implementations, the weight (w3) of the operator 206 may have a weight value of zero. When the controller 103 determines that the weight (w3) has a value of zero, to save processing resources, a multiplication between an input (a2) and the weight (w3) may be skipped so that those operators are not routed to or consumed by a cell 114 a/b/c. The determination to skip the particular multiplication operation may be based on a mapping vector that maps discrete inputs (a n ) of an input vector to individual weights (w n ) of a parameter tensor, as described above.

3 illustrates a traversal table 300 showing examples of memory/data traversals for different functions 302, 304 used to generate addresses for accessing a physical memory bank of the memory 108. Specifically, a first memory traversal is shown for a first function 302, and a second, different memory traversal is shown for a second function 304. In some implementations, the memory 108 includes one or more memory partitions.

3, a partition of memory 108 includes 32 physical memory banks 312, shown as bank 0 through bank 31. Each bank may include a number of rows, e.g., 16 rows, 24 rows, etc. In some implementations, each row is 16 bytes (16B) wide, so that a computing chip 101 accesses data from memory 108 in 16B chunks (e.g., 128 bits). In some implementations, memory 108 may include more or fewer physical memory banks and each row of a bank may have a width greater than 16B or less than 16B (e.g., 1B).

The traversal table 300 includes a first address 306, a second address 308, and a third address 310. The memory traversal distance between each of the first address 306, the second address 308, and the third address 310 can be based on a stride value. In the example of Figure 3, the stride value is 512. However, in other examples, the stride value can be different. Therefore, other stride values are within the scope of the present invention. When the computing system 100 performs a given task or machine learning workload, a stride operation may be required. The stride operation can be based on a stride parameter or value. A given stride value can be programmable at the system 100. For example, the system 100 can program a stride for a given reasoning based on a compiler of the system 100 that is known to be specific to a specific problematic stride for a specific reasoning operation.

In some implementations, the stride value is determined based on a hardware configuration of memory 108, the type of machine learning operation being performed, or both. For example, computing chip block 101 of system 100 may be used to implement a neural network (e.g., a convolutional neural network) tuned or used for compression and/or recognition of image and video content. A stride may be a component of the neural network. In this example, the machine learning operation involves processing an image through a filter of a layer of the neural network according to a set of weights/parameters corresponding to the layer.

Referring to a hardware configuration of physical rows and banks of memory, image or pixel values associated with an image can be stored across physical rows and banks of memory 108. In this example, a stride is a component or parameter of a filter (or kernel) of a neural network. The stride is used to modify the amount by which the filter moves over an image or video. For example, when the stride is set to 1, the computing chip block 101 moves (several) filters over the area one pixel (or input) at a time. Similarly, when the stride is 2, the computing chip block 101 moves (several) filters over the area two pixels at a time.

Thus, filters may be shifted based on a step value of a layer, and in some implementations, the system 100 may repeat this process across multiple computing chips 101 of different layers until inputs of different regions of an image have a corresponding dot product. Moving a filter on inputs of a region of an image based on a step value may include fetching, obtaining, or otherwise accessing inputs from various locations in the memory 108 according to the step value.

As mentioned above, images or pixel values associated with images are stored across physical rows and banks of memory 108. The example of FIG. 3 may be described with reference to a stride value at 512. In some examples, 512 represents a common stride associated with a particular image processing operation. Depending on the type of operation, the computing chips 101 of the system 100 may process requests for accessing memory 108 according to a series of stride values. For example, a first chip 101 may process requests based on a first stride value, while a second, different chip 101 may process requests based on a second, different stride value.

As shown at traversal table 300, when the bank generation function 302 is used to generate a bank ID for processing a request, an access stride of 512 is repeated at the same physical memory bank 314 (e.g., bank 0) of the chip memory. This indicates a bank conflict. As used in this document, a bank conflict occurs when two or more requesters request access to the same physical memory bank of the chip memory during the same clock cycle.

4 , the bank generation function 304 is configured so that processing requests for accessing physical rows in a memory bank of the memory 108 uniformly or at least substantially results in accessing different physical memory banks of the memory 108. More specifically, for each request in a group of requests to access the memory 108, the bank generation function 304 uses bits from an address in the request to generate a bank ID that results in accessing different physical memory banks when, for example, the group of requests is processed in parallel during a single clock cycle.

3, the bank generation function 304 allows two or more requests to be processed based on an access pattern that causes the computing chip 101 to access different physical memory banks of the memory 108. The bank generation function 304 is configured to generate the bank ID and row ID that allow this access pattern regardless of the stride value used for the memory access.

In some implementations, based on the library ID generated by the library generation function 304: i) a first request causes the computing chip block 101 to access 16B in a row of a first physical memory library 316 (e.g., library "0"); ii) a second request causes the computing chip block 101 to access 16B in a row of a second, different physical memory library 318 (e.g., library "4"); and iii) a third request causes the computing chip block 101 to access 16B in a row of a third, different physical memory library 320 (e.g., library "8").

Each of the first, second, and third requests may be a different request, may be from a different requestor, or both. Based on the repository ID and/or access pattern enabled by the repository generation function 304, there is no or substantially no instance in which two or more requestors request access to the same physical memory repository of the chip memory during the same clock cycle. Therefore, each of the first, second, and third requests may be processed simultaneously during the same clock cycle (e.g., a single clock cycle) without a repository conflict occurring at the memory 108. Furthermore, each of the first, second, and third requests may be processed simultaneously during the same clock cycle (e.g., a single clock cycle) for a range of stride values (e.g., a stride of 512) without a repository conflict occurring.

In some embodiments, the computing chip 101 is operable to route and store outputs so that the memory 108 (e.g., an activation memory) does not experience a bank conflict when obtaining the stored output value as an input activation of a second, different neural network layer. This is described in the following paragraphs with reference to activation values, but is applicable to other data types/values that can be written to (or stored in) a physical location of the memory 108.

The computing chip block 101 may include a nonlinear unit that applies activation functions to accumulated values resulting from operations performed at the computing unit 112. In one example, the nonlinear unit may be a hardware circuit included in the multiplication and addition circuits of the computing chip block 101. In another example, the nonlinear unit is included at the computing chip block 101 but is external to the computing unit 112.

The nonlinear unit applies its activation function to produce a set of activated values. The activated values may be the output of a machine learning workload. For example, the activated values may be the output of a first neural network layer routed to and stored at a memory bank of memory 108. These activated values (e.g., outputs of a first layer) may be retrieved from the memory bank of memory 108 and provided as input activations for processing by a second, different neural network layer.

The computing die 101 may issue access requests (e.g., write requests) to the memory 108 to store outputs such as activation values at physical rows and banks of the memory 108. In some examples, these requests may be routed to a partition of the memory 108 used as an activation memory for storing activations. In some implementations, the bank generation function 304 may be used to process these write access requests so that the system 100 does not experience a bank conflict at a particular computing die 101 when the output/activation value is stored at the memory 108 of the die.

The library generation function 304 can also be used to reduce or prevent library conflicts when processing access requests (e.g., read requests) to retrieve or extract output values. For example, the output value can be extracted to be provided as an input activation to a second, different neural network layer.

4 shows an example library generation function 304 for reducing memory bank conflicts at a hardware integrated circuit. As described above, the library generation function 304 is configured so that requests for accessing physical rows in a memory bank of the memory 108 consistently or at least substantially result in accessing different physical memory banks of the memory 108.

For example, given a request group to access memory 108, a controller 103 of computing chip 101 may derive a set of addresses from the request group. In some cases, controller 103 may derive a set of consecutive addresses (e.g., virtual/logical addresses) from the request group. Given this set of consecutive addresses, library generation function 304 is configured to generate a set of corresponding library identifiers ("library IDs"). For example, library generation function 304 generates a separate library ID for each request in the request group.

The system 100 or a computing chip 101 can use the bank ID set to process the request group in one or more clock cycles or in the same clock cycle without a bank conflict occurring at the memory 108. In other words, based on the bank ID set generated by the bank generation function 304, a computing chip 101 can process each request in the request group at the same time (e.g., in parallel) and no two requests will need to access the same physical memory bank of the chip memory 108 during the same clock cycle.

In the example of FIG4 , the library generation function 304 generates a respective library ID for each request based on an algorithm 402. The algorithm 402 generates a library ID based on an exemplary sequence of address bits 404 included in a corresponding request for accessing the memory 108. For example, the computing chip 101 uses the sequence of address bits 404 to obtain a respective value of one or more variables provided as input to the algorithm 402. The input variables may correspond to different portions of the sequence of address bits 404.

A first variable A may be obtained from a first portion of bits, a second variable B may be obtained from a second portion of bits, a third variable row_id may be obtained from a third portion of bits, and a fourth variable byte_in_row may be obtained from a fourth portion of bits. For example, the sequence of bits 404 may be an input address divided into two parts: i) bytes_in_row (the number of bytes in the memory row); and ii) a row_address. Using the library generation function 304, row_address may be divided into two parts: i) a row_id (e.g., the row_id in each library); and ii) a learned library id defined as variable A. The least significant bit (LSB) in row_id may be defined as variable B.

4, for a given bit sequence 404, one or more bits in one portion of the sequence 404 may overlap with one or more bits in another portion of the sequence. In some implementations, an input variable may correspond to the least significant bit (lsb) in a request or the most significant bit (msb) in a request.

In some implementations, the library generation function 304 is configured to perform at least a first set of operations 412, a second set of operations 414, and a third operation 416. The first and/or second set of operations may be used to create specific input variables, such as row_address, row_id, A, and B. In some cases, row_address is used to extract specific bits in the sequence 404, such as bits 4 to 15, and row_id is used to extract specific other bits, such as bits 8 to 15. In some implementations, the variables may be used to extract other ranges or combinations of bits.

The third operation (or set of operations) 416 may be used to rotate a bit vector based on a shift parameter or operation such as rotation_banking_shift. The bit vector may be derived from an address bit sequence in a request to access memory 108 based on the operation performed using the library generation function 304. The bit vector may be derived from the msb in the address bit sequence or the lsb in the address bit sequence. For example, the bit vector may be a combination of variables A and B and the library generation function 304 may be configured to apply a rotation_banking_shift operation to this combination of variables.

The rotation_banking_shift operation may be applied with reference to a number of banks in the memory of a computing chip 101. For example, the rotation_banking_shift operation may be used to: i) shift an msb bit vector by an amount specified by a shift value of the operation; or ii) shift an lsb bit vector by an amount specified by a shift value of the operation. The system 100 performs the shift operation so that a bank ID of a new bank will belong to a particular group. Thus, the third operation 416 may generate a rotated bank ID based on the number of banks and any row, bank, and byte attributes associated with variables A and/or B.

In some implementations, the system 100 determines a minimum number of bits used to define an MSB or LSB. For example, the system 100 may determine the minimum number of bits based on the number of physical memory banks in a given partition of the memory 108. For example, if there are 32 physical memory banks, the system 100 may determine that a minimum number of 5 bits is required to represent 32 numbers or 32 column IDs.

In some embodiments, the library generation function 304 is configured or coded as shown at algorithm 402 in the example of FIG4. In some other embodiments, a modified library generation function may be configured or coded (using an alternative command) to generate a set of library IDs that can be processed simultaneously at a computing die 101 without triggering a library conflict at memory 108. For example, the modified library generation function may be configured or coded for a die memory that includes more or fewer physical memory libraries and more or fewer rows relative to memory 108.

The bank generation function 304 may be an algorithm that receives one or more inputs and generates one or more outputs. For example, as discussed above, the inputs may be a respective address of one or more rows ("row_address"), the number of rows in each bank ("rows_per_bank"), the least significant bit (lsb) in a request, a minimum number of bits of the lsb, the most significant bit (msb) in a request, a minimum number of bits of the msb, a bank rotation parameter, or a shift parameter ("msb_shift_minus_1"). The outputs may be respective bank IDs of physical memory banks of the memory 108 and respective row IDs of physical rows in a physical memory bank of the memory 108. More or fewer inputs are within the scope of the present invention and may be used in conjunction with the bank generation function 304.

FIG5 is an example process 500 for reducing memory bank conflicts in a chip memory of a hardware accelerator or dedicated hardware integrated circuit. In some implementations, the process 500 is performed during operations for a neural network machine learning model implemented on a hardware accelerator. For example, the operations may be performed to process a neural network input (such as an image or speech utterance) using a dedicated neural network processor. Such a processor may be represented by the integrated circuit or system 100 described with reference to FIG1A.

For example, the hardware integrated circuit may be configured to implement a CNN comprising a plurality of neural network layers. In some cases, the neural network layers may include a group of convolutional layers. The input may be an example image as described above, including various other types of digital images or related graphical data. In at least one example, the integrated circuit may implement an RNN for processing input derived from a speech or other audio content. In some cases, process 500 is part of a technique that allows improvements in latency and throughput to be achieved in accelerating neural network operations used to produce image or speech processing outputs relative to other data processing techniques.

Process 500 may be implemented or executed using the system 100 described above. Therefore, the description of process 500 may refer to the above-described computing resources of system 100. In some examples, the steps or actions of process 500 are implemented by programmed firmware instructions, software instructions, or both. Instructions of various types may be stored in a non-transitory machine-readable storage device and may be executed by one or more processors of the devices and resources described in this document.

In some implementations, the steps of process 500 are performed at a hardware circuit to generate a layer output for a neural network layer. The output may be part of a computation of a machine learning task or reasoning workload to generate an image processing or image recognition output. As indicated above, the integrated circuit may be a dedicated neural network processor or a hardware machine learning accelerator configured to accelerate computations used to generate various types of data processing outputs.

Referring again to process 500, system 100 receives a plurality of requests for accessing memory resources of the system (502). For example, each of the plurality of requests may be for accessing a first, die-block memory of a base circuit or a hardware accelerator. In some implementations, system 100 processes the plurality of requests across a plurality of die-blocks. For example, system 100 may process a respective subset of requests at each computing die-block 101 of a base circuit.

For each request, the system 100 may identify a respective logical address represented by a bit sequence in the request (504). As described above, the memory 108 may be row-addressable such that a row of a memory bank may be identified or accessed based on an address in a corresponding request. Generally, each request specifies a logical address that may be mapped to a corresponding physical address. For example, a compiler of the system 100 may determine a mapping of a set of logical addresses to a set of corresponding physical addresses, where the physical address specifies a physical location in memory (e.g., a location in a bank in the first memory 108 or a location in a row in a bank in the first memory 108).

For each request, the system 100 may obtain a first bit subset from the bit sequence (506). For example, a logical address may be specified by a data structure comprising a sequence of bits (e.g., 8 bits, 16 bits, etc.). In some embodiments, a controller 103 of a computing chip 101 scans each data structure representing a logical address specified by a request. In response to the bits formed from the scanned data structure, the controller identifies or determines a bit subset that can be used as an input to a library generation function. In some embodiments, the controller determines the bit subset based on a configuration of the chip memory 108.

For each request, the system 100 may generate a respective library identifier ("library ID") (508) based on a library generation function 304 using the first subset of bits. Each respective library ID identifies a particular physical library of the plurality of physical memory libraries of the memory 108. As described above, each physical memory library may include a plurality of columns. For each request, the computing chip 101 may: i) obtain a second subset of bits from the bit sequence; ii) provide the second subset of bits as an input to the library generation function; and iii) based on applying the library generation function to the second subset of bits, generate a respective column ID identifying a particular column in a particular library of the physical memory libraries of the chip memory.

The system 100 is configured to process the requests using the respective bank ID generated for each request (510). In some implementations, a computing chip 101 processes multiple requests (e.g., simultaneously), and in response to processing the requests, accesses two or more physical memory banks in parallel without a bank conflict (512). For example, a controller 103 of the computing chip 101 generates control signals to access a physical bank or row of memory 108 to fetch an input vector 102 from an address location of the first memory 108.

In some implementations, in response to processing each of the requests to fetch one or more input vectors, the computing chip 101 may simultaneously access multiple distinct physical memory banks of the chip memory during a single clock cycle. The input vector 102 may correspond to an input feature map of an image and may be a matrix structure of neural network inputs (such as activations generated by a previous neural network layer).

As discussed above, the bank generation function 304 allows two or more requests to be processed based on an access pattern that causes the computing chip 101 to access different physical memory banks of the memory 108. The bank generation function 304 is configured to generate bank IDs and row IDs that allow this access pattern regardless of the stride value used for the memory access.

Notably, the bank generation function can provide this access pattern even when a memory access stride is equal to a difference between specific rows of the same physical memory bank. For example, each row of a physical memory bank may have a width of 16 bytes and a partition of chip memory may have 32 physical memory banks. In this example, the bank generation function can provide this access pattern even when the memory access stride is equal to row_width * num_banks (i.e., stride = 512).

6 shows an example of a tensor or multidimensional matrix 600 including an input tensor 604, a transformation of a parameter tensor 606, and an output tensor 608. In the example of FIG6 , each of the tensors 600 includes respective elements, where each element may correspond to a respective data value (or operand) for an operation performed at a given layer of a neural network.

For example, each input of the input tensor 604 may correspond to a respective element along a given dimension of the input tensor 604, each weight of the parameter tensor 606 may correspond to a respective element along a given dimension of the parameter tensor 606, and each output value or activation in a set of outputs may correspond to a respective element along a given dimension of the output tensor 608. Relatedly, each element may correspond to a respective memory location or address in a memory of a compute die 101 assigned to operate on one or more dimensions of a given tensor 604, 606, 608.

Operations performed at a given neural network layer may include multiplying an input/activation tensor 604 with a parameter/weight tensor 606 on one or more processor clock cycles to produce layer outputs, which may include output activations. Multiplying an activation tensor 604 with a weight tensor 606 includes multiplying an activation from an element of tensor 604 with a weight from an element of tensor 606 to produce one or more partial sums. The example tensors 606 of FIG. 6 may be unmodified parameter tensors, modified parameter tensors, or a combination thereof. In some embodiments, each parameter tensor 606 corresponds to a modified parameter tensor that includes a non-zero CSP value derived based on a particular sparsity exploitation technique described above.

The processor core of the system 100 can operate on: i) a scalar corresponding to a discrete element in a multidimensional tensor 604, 606; ii) a vector (e.g., input vector 102) containing the values of multiple discrete elements 609 along the same or different dimensions of a multidimensional tensor 604, 606; or iii) a combination of these. Depending on the dimensionality of the tensor, a discrete element 609 or each of multiple discrete elements 609 in a multidimensional tensor can be represented using X, Y coordinates (2D) or using X, Y, Z coordinates (3D).

The system 100 may compute multiple partial sums corresponding to products generated by multiplying a batch of inputs by corresponding weight values. As mentioned above, the system 100 may perform an accumulation (e.g., partial sum) of products over many clock cycles. For example, the accumulation of products may be performed in a random access memory, shared memory, or high-speed scratch memory of one or more computing chips based on the techniques described in this document. In some embodiments, an input weight multiplication may be written as a sum of products of each weight element multiplied by a discrete input of an input vector 102 (e.g., a column or slice of the input tensor 604). This row or slice can represent a given dimension, such as a first dimension 610 of the input tensor 604 or a second, different dimension 615 of the input tensor 604.

In some implementations, an example set of operations may be used to compute an output of a convolutional neural network layer. The operation of the CNN layer may involve performing a 2D spatial convolution between a 3D input tensor 604 and at least one 3D filter (weight tensor 606). For example, convolving a 3D filter 606 over the 3D input tensor 604 may produce a 2D spatial plane 620 or 625. The operation may involve computing the sum of dot products of a particular dimension of an input volume including the input vector 102.

For example, spatial plane 620 may include output values from sums of products of input operations along dimension 610, while spatial plane 625 may include output values from sums of products of input operations along dimension 615. The sums of products operations used to generate output values in each of spatial planes 620 and 625 may be: i) performed at operation cells 114 a/b/c; ii) performed directly at memory 110 using an arithmetic operator coupled to a shared memory bank of memory 110; iii) or both. In some embodiments, the reduction operation may be simplified and performed directly at a memory cell (or location) of memory 110 using various techniques for reducing the accumulated values.

Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device.

Alternatively or in addition, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to appropriate receiver equipment for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The term "computing system" encompasses all kinds of equipment, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. Equipment may include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, equipment may also include program code that establishes an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

A computer program (which may also be called or described as a program, software, a software application, a module, a software module, a script or code) may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

A computer program may (but need not) correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in a plurality of coordinated files (e.g., files that store one or more modules, subroutines or portions of program code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

The procedures and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The procedures and logic flows may also be performed by, and the apparatus may be implemented as, a dedicated logic circuit system, such as, for example, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

A computer suitable for the execution of a computer program includes (for example, may be based on) a general purpose microprocessor or a special purpose microprocessor or both or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or be operably coupled to receive data from or transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), etc.

Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, by way of example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM optical disks. The processor and memory may be supplemented by or incorporated in dedicated logic circuitry.

To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., an LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. Additionally, a computer may interact with a user by sending files to and receiving files from a device used by the user; for example, by sending web pages to a web browser in response to requests received from the web browser on a user's client device.

Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an embodiment of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN") and a wide area network ("WAN"), such as the Internet.

A computing system may include clients and servers. A client and server are generally remote from each other and usually interact through a communication network. The relationship of client and server occurs by virtue of computer programs running on the respective computers and having a client-server relationship with each other.

Although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or claimable content, but rather as describing features that may be specific to a particular embodiment of the invention. Specific features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. In addition, although features may be described above as functioning in a particular combination and even initially claimed as such, one or more features from a claimed combination may be exempted from that combination in some cases, and the claimed combination may be related to a subcombination or a variation of a subcombination.

Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in sequential order or that all depicted operations be performed to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may be generally integrated together in a single software product or packaged in multiple software products.

Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the specific order shown, or sequential order, to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

100: Computing system 101: Computing chip 102: Input vector 103: Controller 104: Weight 105: Control signal 106: Input enable bus 107: Output enable bus 108: First memory/chip memory 110: Second memory 112: Computing unit 114: Computing cell 114a: Computing cell 114b: Computing cell 114c: Computing cell 120: Host 125: Higher-level controller 130: Request 135: Logic port 140: Arbiter 200: Processing pipeline 202: Shift register 204: Operator 206: Operator 300: Traversal table 302: First function/library generation function 304: Second function/library generation function 306: First address 308: Second address 310: Third address 312: Physical memory library 314: Physical memory library 316: First physical memory library 318: Second, different physical memory library 320: Third, different physical memory library 402: Algorithm 404: Address bit/bit sequence 412: First set of operations 414: Second set of operations 416: Third operation 500: Program 502: Step 504: Step 506: Step 508: Step 510: Step 512: Step 600: Tensor or multidimensional matrix 604: Input tensor/starting tensor 606: Parameter tensor/weight tensor/3D filter 608: Output tensor 609: Discrete elements 610: First dimension 615: Second different dimension 620: 2D spatial plane 625: 2D spatial plane

FIG1A is a block diagram of an example computing system for implementing a neural network machine learning model.

FIG1B is a block diagram of an example computing system for implementing a neural network machine learning model.

FIG. 2 shows an example processing pipeline for routing inputs and outputs between a memory and computational cells in a hardware integrated circuit.

FIG3 shows an example of a memory traversal for different banking generation functions.

FIG4 shows an example library generation function for reducing memory library conflicts at a hardware integrated circuit.

FIG. 5 is an example process for reducing memory bank conflicts in a hardware integrated circuit.

FIG6 shows an example of an input tensor, a parameter tensor, and an output tensor.

Like reference numerals and names in the various drawings indicate like elements.

100: Computing system

101: Computing chip block

102: Input vector

103: Controller

104: Weight

105: Control signal

106: Input start bus

107: Output start bus

108: First memory/chip memory

110: Second memory

112: Arithmetic unit

114a: Operation Cell

114b: Operation Cell

114c: Operation Cell

120: Host

125: Higher-end controller

Claims (20)

A computer-implemented method for simultaneously accessing a memory library of a hardware accelerator, the method comprising: Receiving a plurality of requests, each of the plurality of requests being for accessing a chip memory of the hardware accelerator; For each of the plurality of requests: Identifying a respective logical address represented by a bit sequence in the request; Obtaining a first bit subset from the bit sequence; and Based on a library generation function using the first bit subset, generating a respective library identifier identifying a particular library among a plurality of physical memory libraries of the chip memory; Processing each of the plurality of requests using the respective library identifier generated for the request; and In response to processing each of the plurality of requests, multiple different physical memory banks of the chip block memory are simultaneously accessed during a clock cycle. The method of claim 1, wherein simultaneously accessing the multiple different entity memory libraries comprises: For a specific stride value, accessing the multiple different entity memory libraries during a single clock cycle. The method of claim 2, wherein the specific stride value is equal to a memory access stride of a difference between specific rows of the same physical memory bank. The method of claim 3, wherein simultaneously accessing the multiple different physical memory libraries comprises: Accessing the multiple different physical memory libraries without any library conflict. A method as claimed in claim 4, wherein a bank conflict occurs when two or more requesters request access to the same physical memory bank of the chip block memory during the same clock cycle. The method of claim 5, wherein each physical memory library includes a plurality of columns and the method includes: For each of the plurality of requests: Obtaining a second bit subset from the bit sequence; Providing the second bit subset as an input to the library generation function; and Based on applying the library generation function to the second bit subset, generating a respective column identifier for a specific column in the specific library of the plurality of physical memory libraries of the chip block memory. The method of claim 6, wherein: each of the plurality of rows comprises a width of 16 bytes; a partition of the chip block memory comprises 32 physical memory banks; and the memory access stride is equal to: row_width * num_banks. The method of claim 1, wherein obtaining a first bit subset from the bit sequence comprises: Obtaining two or more bits from the least significant bits (LSBs) in the bit sequence. A system comprising: a hardware accelerator; a processing device; and a non-transitory machine-readable storage medium for storing instructions executable by the processing device to cause execution of operations including: receiving a plurality of requests, each of the plurality of requests for accessing a chip memory of the hardware accelerator; for each of the plurality of requests: identifying a respective logical address represented by a bit sequence in the request; obtaining a first bit subset from the bit sequence; and generating a respective library identifier identifying a particular library among a plurality of physical memory libraries of the chip memory based on a library generation function using the first bit subset; Processing each of the plurality of requests using the respective repository identifier generated for the request; and In response to processing each of the plurality of requests, simultaneously accessing multiple distinct physical memory repositories of the chip block memory during a single clock cycle. A system as claimed in claim 9, wherein simultaneously accessing the multiple different entity memory libraries comprises: For a specific stride value, accessing the multiple different entity memory libraries during a single clock cycle. A system as claimed in claim 10, wherein the particular stride value is equal to a memory access stride that is a difference between particular rows of the same physical memory bank. A system as claimed in claim 11, wherein simultaneously accessing the multiple different physical memory libraries comprises: Accessing the multiple different physical memory libraries without any library conflict. A system as claimed in claim 12, wherein a bank conflict occurs when two or more requestors request access to the same physical memory bank of the chip memory during the same clock cycle. The system of claim 13, wherein each physical memory library includes a plurality of rows and the operations further include: For each of the plurality of requests: Obtaining a second bit subset from the bit sequence; Providing the second bit subset as an input to the library generation function; and Based on applying the library generation function to the second bit subset, generating a respective row identifier for a specific row in the specific library of the plurality of physical memory libraries of the chip block memory. The system of claim 14, wherein: each row of the plurality of rows comprises a width of 16 bytes; a partition of the chip block memory comprises 32 physical memory banks; and the memory access stride is equal to: row_width * num_banks. The system of claim 9, wherein obtaining a first bit subset from the bit sequence comprises: obtaining two or more bits from the least significant bits (LSBs) in the bit sequence. A non-transitory machine-readable storage medium for storing instructions executable by a processing device of a hardware accelerator to cause execution of operations including the following: Receiving a plurality of requests, each of the plurality of requests for accessing a chip memory of the hardware accelerator; For each of the plurality of requests: Identifying a respective logical address represented by a bit sequence in the request; Obtaining a first bit subset from the bit sequence; and Based on a library generation function using the first bit subset, generating a respective library identifier identifying a particular library among a plurality of physical memory libraries of the chip memory; Processing each of the plurality of requests using the respective library identifier generated for the request; and In response to processing each of the plurality of requests, multiple different physical memory banks of the chip block memory are simultaneously accessed during a single clock cycle. A non-transitory machine-readable storage medium as claimed in claim 18, wherein accessing the multiple different physical memory libraries simultaneously comprises: For a specific stride value, accessing the multiple different physical memory libraries during a single clock cycle. A non-transitory machine-readable storage medium as in claim 18, wherein the specific stride value is a memory access stride equal to a difference between specific rows of the same physical memory bank. The non-transitory machine-readable storage medium of claim 19, wherein simultaneously accessing the multiple different physical memory libraries comprises: Accessing the multiple different physical memory libraries without any library conflict.