DE102020211815A1 - storage unit - Google Patents

Abstract

The invention relates to a memory unit (200) with one or more memory banks (102), each memory bank (102) having one or more memory cell arrays (104) with rows and columns of memory cells (106), each memory cell array (104) having one Row decoding unit (116), a column decoding unit (118) and a primary sense amplifier (114) are assigned, each memory bank (102) being assigned a secondary sense amplifier (120), the memory unit (200) being set up such that by means of the row decoding unit ( 116) and the column decoding unit (118) memory cells (106) can be selected, that for reading out the memory cells (106) a signal can be amplified by means of the primary sense amplifier (114), that bits stored in a row of memory cells (106) go into the primary sense amplifier (114) are chargeable, and that a signal obtained from the primary sense amplifier (114) is readable by means of the secondary sense amplifier (120) can be amplified, each memory cell array (104) being assigned a first processing unit (122), a second processing unit (124) and an adding unit (126), the memory unit (200) being set up in such a way that in a row of memory cells (106) stored bits are loadable into the first processing unit (122), that bits present in the primary sense amplifier (114) and bits present in the first processing unit (122) are loadable into the second processing unit (124) and processable therein, and that bits processed in the second processing unit (124) can be loaded into the adding unit (126) and processed therein, as well as a use thereof and a method herewith.

Description

The present invention relates to a memory unit with one or more memory banks, use of such a memory unit when processing an artificial neural network, a method for processing an artificial neural network using the memory unit, and a computing unit and a computer program for its implementation.

Background of the Invention

The implementation or processing of (artificial) neural networks, in particular so-called deep neural networks, with dedicated accelerators (to speed up the calculation, typically with external memory, on-chip buffer memory, register files and processing engine) requires a large external memory throughput. The reason for this is that due to the limited on-chip memory available in such accelerators, the results of intermediate layers are generally placed in external memory. Since off-chip memory accesses are costly in terms of power and performance, recent work has focused on offloading processing from a dedicated accelerator to external memory itself.

The main focus here was to shift the inference phase (and not the training phase) of neural networks to the external memory. In addition, due to their high density and low cost, dynamic random access memories (so-called DRAMs) are the means of choice for implementing the external memory of a system.

Previous work on this so-called in-DRAM inference followed one of two directions. The first direction relies on the concept of activating multiple rows simultaneously to implement basic logic operations, such as in "Q. Deng, L. Jiang, Y. Zhang, M. Zhang and J. Yang, "DrAcc: a DRAM based Accelerator for Accurate CNN Inference," 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC), San Francisco, CA, 2018, pp. 1-6.”, “J. Sim, H. Seol and L. Kim, "NID: Processing Binary Convolutional Neural Network in Commodity DRAM," 2018 IEEEIACM International Conference on Computer-Aided Design (ICCAD), San Diego, CA, 2018, pp. 1-8." , and "S. Angizi, Z. He and D. Fan, "DIMA: A Depthwise CNN In-Memory Accelerator," 2018 IEEEIACM International Conference on Computer-Aided Design (ICCAD), San Diego, CA, 2018, pp. 1-8." As promising as this concept is, it suffers from high power consumption due to the numerous multi-row activations. In addition, a process variation (to manufacture a memory chip) can greatly affect the logical operation result and consequently the model accuracy.

The second direction can be found in “C. Sudarshan et al., "An In-DRAM Neural Network Processing Engine," 2019 IEEE International Symposium on Circuits and Systems (ISCAS), Sapporo, Japan, 2019, pp. 1-5." can be observed, after which attempts have been made to overcome the problems by bringing the calculation closer to the data in the DRAM.

Disclosure of Invention

According to the invention, a memory unit with one or more memory banks, use of such a memory unit when processing a neural network, a method for processing a neural network using the memory unit and a computing unit and a computer program for its implementation with the features of the independent patent claims are proposed. Advantageous configurations are the subject of the dependent claims and the following description.

The invention relates to a memory device, particularly a memory device such as a DRAM and a modification thereof. DRAM or "Dynamic Random Access Memory" refers to a technology for an electronic memory module with random access (Random Access Memory, RAM), which is used in computers, for example, but is also used in other electronic devices. The storing element, a memory cell, is a (small) capacitor (also called storage capacitor), which is either charged or discharged. It is accessible via a switching transistor and either read out or written with new content.

For example, the memory cells are arranged and wired on the surface of a die, and are typically organized into an appropriate hierarchical structure. While the internal structure can be manufacturer-specific, the logical structure visible from the outside is usually standardized. This ensures that chips from different manufacturers and different sizes can always be addressed using the same scheme.

A memory unit on which the invention is based has one or more memory banks, there being typically a plurality of banks, in particular a large number thereof. Each memory bank in turn has one or more Memory cell fields (“array”) with rows (or rows, also word lines) and columns (also bit lines) of memory cells. Here, too, it is typical for a memory bank to have a number of memory cell arrays, in particular in the form of a 2D matrix. In addition, each bit of each word is stored in particular in the mentioned capacitor, a storage capacitor.

A row decoding unit (row selection logic), a column decoding unit (column selection logic) and a primary sensing amplifier are assigned to each memory cell array. In addition, each memory bank is assigned a secondary sensing amplifier. These can also be provided jointly for a number of memory cell arrays.

The memory unit is designed in a specific way in which the components mentioned interact, and is set up for this purpose in such a way that memory cells can be selected or controlled by means of the row decoding unit and the column decoding unit. The row decoding unit (or row decoding unit) decodes an incoming activation command and correspondingly confirms a row or word line. Activating such a wordline completes the circuit between storage capacitors associated with a series of primary sense amplifiers and the memory bank's columns or bitline. The column decode unit decodes read or write commands and connects bit lines to the secondary sense amplifier.

In addition, the memory unit is set up so that, in order to read out the memory cells, a signal can be amplified by means of the primary sense amplifier, that bits stored in a row of memory cells can be loaded into the primary sense amplifier, and that a signal obtained from the primary sense amplifier can be amplified by means of the secondary sense amplifier is.

Primary sense amplifiers are required because the bits or information in the memory cell arrays are stored in the small capacitors. Consequently, special structures, i.e. the primary sense amplifiers, are required to detect the presence or absence of a charge in such (small) capacitors. Reading data from a memory cell array into a primary sense amplifier is a very slow process. It is important to note that major or major modifications to the primary sense amplifiers found in conventional DRAM chips are undesirable because they can affect the density, cost, and speed of a device.

The secondary sense amps are required because the output driver of the primary sense amps is not powerful enough to drive a data bus of the DRAM chip (which has a large capacitive load associated with it). Note that reading data from the secondary sense amp is a fast process. In addition, large changes in this block are much less critical than changes in the architecture of the primary sense amps.

In the aforementioned work on processing neural networks in DRAM, the focus was on so-called binary or binarized neural networks, which still cannot achieve high levels of accuracy in many applications. Such binary neural networks have weights that can only have two values, zero or one. In order to perform in-DRAM calculations with higher accuracy, in “Q. Deng, Y. Zhang, M. Zhang and J. Yang, "LAcc: Exploiting Lookup Table-based Fast and Accurate Vector Multiplication in DRAM-based CNN Accelerator," 2019 56th ACM/IEEE Design Automation Conference (DAC), Las Vegas, NV, USA, 2019, pp. 1-6.” proposed a multiplierless accelerator that depends on look-up tables in DRAM. However, such a design cannot use the enormous potential of DRAM in terms of parallelization, since it does not use the potential of parallelization of memory banks.

As has now been recognized and as will become clear later, this is a limiting factor since inference involves memory operations (the output of hidden layers of the neural network) that cannot be performed in parallel with computation unless parallelization of memory banks is exploited. In addition, the conventional DRAM structure needs to be modified significantly in terms of capacitive unit and circuitry, which limits the applicability of the mentioned approach.

With this in mind, however, it is also desirable to be able to perform multi-bit precision inference in DRAM without affecting the already highly efficient internal structure of a DRAM chip.

To this end, a modification or expansion of the memory unit described above, in particular a DRAM chip, is proposed so that inference (processing) with a plurality of bit precisions (and not just a binary neural network) can also be carried out or undertaken. The use of multiplierless logic is of some importance to this approach since the multi-bit multiplication logic requires a large (on-chip) area. The Integr Integration of such logic into a DRAM chip would therefore require modification of the microarchitecture of a DRAM chip that is already highly optimized for low cost and high density.

In the proposed memory unit, a first processing unit, a second processing unit and an adder unit are now assigned to each memory cell array, ie in particular also applied to the chip. The memory unit is set up in such a way that bits stored in a row of memory cells can be loaded into the first processing unit, that bits present in the primary sense amplifier and bits present in the first processing unit can be loaded into the second processing unit and processed there, and that in the bits processed by the second processing unit can be loaded into the adding unit and processed therein.

Regarding neural networks, the following observations could be made: The main operation performed by such networks is multiplication and accumulation. Each output element of a network layer is obtained by multiplying a set of input elements by a set of corresponding weights, i.e. a kernel. The result of these multiplications is then added (accumulation) and then fed into a non-linear function, e.g neurons). The output of the nonlinear function represents an output element of a tensor. In addition, the arithmetic multiplication operation can be replaced by a shift operation.

With the extensions or modifications mentioned, a neural network can be processed as follows: The first processing unit serves as a weight latch. The idea here is that the weights of a neural network kernel are stored in the rows of a memory cell array. Such weights should first be loaded into the first processing unit by executing a corresponding activation command.

The second processing unit is for a shifter that is specifically designed for both the first processing unit and the primary sense amplifier. Each second processing unit performs the multiplication of an enable (i.e., an input) and a weight, receiving the appropriate bits from the primary sense amplifier and the first processing unit. This gives - after processing - a weighted activation as a result.

In particular, the adder unit includes a series of 4-operand adder trees. In particular, each adder tree therefore sums four weighted activations (the bits processed in the second processing unit). Larger adder trees are not practical because, as previously mentioned, large changes in the primary sense amplifier can significantly affect the density, cost, and speed of the memory device.

The processing of a layer of a neural network using the memory unit mentioned is carried out in particular as follows: The weights and activations (input elements) that are to be multiplied are stored (or placed) in different rows of a common or the same memory bank. In response to an activation command (here the command name does not refer to the concept of activation in a neural network but to DRAM), the weights are loaded into the first processing unit.

Upon another activation command, the activations (input elements) are loaded into the primary sense amp. Once the activations are in the primary sense amplifier and the weights are in the first processing unit, a shift operation can be performed by the second processing unit and the result fed into the appropriate adder trees or adder unit.

The addition unit or the four operands per addition tree are generally not sufficient to generate an element of an output tensor. Therefore, in particular, further accumulation operations (with adder units) are implemented near the secondary sense amplifiers. Other operations such as max pooling and/or a non-linearity function such as ReLU can also be implemented at the secondary sense amp level. Again, the secondary sense amp is not as critical as the primary sense amp, so fitting complex logic is usually not a problem.

Adding the shifting logic (second processing unit) and required accumulation logic (adder unit) to a DRAM chip not only allows for massively parallel execution of convolution operations (in a convolutional neural network) but also reduces the number of Requests (memory requests or memory requests ments) between a chip containing processing logic and its external memory (which has been a bottleneck until now).

In addition, the reduced number of memory requests directly translates to reduced power consumption because the DRAM data bus lines (which are high capacitance) do not have to be switched as often. As a result, the system's teraops/watt metric is significantly improved. These hardware improvements are achieved while maintaining high inference accuracy due to the full bit precision multiplier approach.

This proposed memory unit also enables various other aspects such as optimizing the refresh intervals within the DRAM since the inference is done in the DRAM. Also these in-DRAM techniques can be extended to other neural networks than just convolutional neural networks. The proposed structure can also be extended to other storage technologies and is also suitable for so-called hybrid storage cubes and high-bandwidth storage.

Different types of parallelism are possible with the proposed memory unit for processing neural networks: the parallel multiplication and accumulation operations within the same secondary sense amplifier; within the same memory bank there are multiple memory cell arrays that operate in parallel, and within the memory unit there are multiple memory banks that operate independently of each other. At the same time, the algorithm for computing the output feature map of a neural network layer, i.e. the totality of the output values or output elements computed in a layer, offers four preferred ways to exploit the mentioned parallelism.

One way to calculate a layer of a neural network, in particular a convolutional one, i.e. to calculate the output elements or the output feature map from the input elements and the weights, is to divide the calculation process into four loops, such as in “Y . Ma, Y. Cao, S. Vrudhula, and J.-S. Seo, "Optimizing loop operation and dataflow in FPGA acceleration of deep convolutional neural networks," in Proc. ACM/SIGDA Int. Symp. Field-Program. Gate Arrays (FPGA), Feb. 2017, pp. 45-54.”” mentioned.

The first loop covers the kernels or weights (with x and y dimensions of the input elements), the second loop the number of input elements (or input feature map), the third loop the x and y dimensions of output items (or output feature map) and the fourth loop the number of output items. These loops can now be distributed particularly well in the proposed memory unit.

For example, the first and second loops within a memory cell array can be fully unwound (unwound here means that the different iterations of these loops are executed in parallel). Similarly, different arrays of memory cells can fully unwind the fourth loop (this is achieved by forcing that weights belonging to the same kernel are always stored in a single array of memory cells). And further, the third loop can be handled between different memory banks, i.e. each memory bank produces output elements belonging to a different y-dimension of the output tensor of the layer to be processed.

However, this example schema can be modified depending on the layer to be processed and the specifications of the neural network. For example, in a scenario with a layer with a large number of cores and a small input feature map (i.e. few input elements), the first and second loops can be mapped to different memory banks, while the third loop can be mapped to a memory cell array.

Another challenge is to avoid the computation stalling when storing elements of the output tensor (i.e. output elements) of a layer. In particular, elements of the output tensor are calculated in logic located near the second sense amplifier. After they have been calculated, they have to be stored again in the corresponding memory cell array. This is necessary because the output elements of one layer generally become the input elements of another layer (an exception is the last layer of a mesh).

Since the path between the computational logic near the second sense amp where output elements are generated and the memory cell array where output elements are stored traverses a primary sense amp, the memory operation would prevent the primary sense amp from performing multiplications (by shifting) and accumulations (through the 4-operand adder tree).

A parallelization distribution in which the memory banks are divided into two groups, a calculation group and a memory group, is used for this. During the calculation or processing of each layer is one of the groups the calculation group and the other group the storage group. The input elements of the current shift are saved in the calculation group. This group calculates elements of the output feature map and outputs such elements on the common DRAM data bus. The storage group contains the weights for the next layer to be processed and stores the output items that are output from the calculation group. When all output elements of the current layer have been processed, the calculation group becomes the storage group and vice versa.

This means that instead of storing the output tensor of a layer in the same bank where the input tensor is stored, output elements are transferred from the calculation group to the storage group.

Another benefit of this scheme is savings in memory usage. As previously mentioned, since the third loop can be handled between memory banks, each memory bank requires a copy of the input feature map. This means that four replications of the input tensor are needed when four memory banks do the computation. However, if all memory banks were to calculate simultaneously, eight replications would be required.

In addition to the storage unit itself, the invention also relates to the use of such a unit for processing at least one layer of a neural network, in particular a multi-bit network (in which the weights are not just binary, but can assume more than just two different values ), and a method for processing a neural network, wherein layers of the neural network are calculated using a storage unit.

A computing unit according to the invention has in particular a memory unit according to the invention and is set up, in particular in terms of programming, to carry out a method according to the invention.

The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is advantageous because this causes particularly low costs, especially if an executing control device is also used for other tasks and is therefore available anyway. Suitable data carriers for providing the computer program are, in particular, magnetic, optical and electrical memories, such as hard drives, flash memories, EEPROMs, DVDs, etc. It is also possible to download a program via computer networks (Internet, intranet, etc.).

Further advantages and refinements of the invention result from the description and the attached drawing.

The invention is shown schematically in the drawing using an exemplary embodiment and is described below with reference to the drawing.

character list

• 1 shows schematically a memory unit for explaining the invention.
• 2 shows schematically a memory unit according to the invention in a preferred embodiment.
• 3 shows schematically a memory unit according to the invention in a further preferred embodiment.
• 4 shows schematically a sequence of a method according to the invention in a preferred embodiment.

embodiment(s) of the invention

In 1 a storage unit 100 is shown schematically for explaining the invention. The memory unit 100 can in particular be a so-called DRAM or a DRAM chip. First let's explain the components of the DRAM chip, in 2 then the modifications made within the scope of the invention.

By way of example, the memory unit 100 has a memory bank 102, with a large number of them being provided in practice, which can all be constructed in the same way. The memory bank 102 has four memory cell arrays 104, for example, which in turn have a multiplicity of memory cells 106 arranged in rows or rows and columns and having a storage capacitor 108.

Each of these memory cells 106 or storage capacitors 108 is connected to a word line 110 and a bit line 112 . The memory cells 106 can be selected or controlled by means of a row decoding unit 116 and a column decoding unit 114 in order to read or write to them, for example. It should be mentioned in this regard that each memory cell array 104 is assigned a row decoding unit 116 and a column decoding unit 114, it also being possible, for example, for a column decoding unit 114 to be assigned to a plurality of memory cell arrays 104 at the same time. Likewise, as shown here, two column decoding units 114 can be used for one field 104 be provided if, for example, there is not enough space for a larger (or sufficiently large) column decoding unit on one side of the array (in other words, a column decoding unit can also be divided into two parts, one of which is arranged at the top and the other at the bottom of the array will).

Furthermore, primary sense amplifiers 114 are provided in such a way that each memory cell array 104 is assigned (at least) one of them. In particular, the bit lines at the bottom of the memory cell array 104 are connected to the primary sense amplifiers. Since they have to fit into the narrow grid of the cell field, they are constructed in the simplest form as two CMOS inverters with negative feedback and only four transistors. In addition to their function as an amplifier for the read cell signal, they also have the side effect that their structure corresponds to that of a simple static memory (latch). The primary sense amplifier thus serves at the same time as a memory for a complete memory line. In addition, a secondary sense amplifier 118 is provided for the memory bank 102 by way of example, which in turn is assigned to a plurality of primary sense amplifiers 114 . With regard to the functioning of such a memory unit or a DRAM chip, reference is made to the above statements.

In 2 a memory unit 200 according to the invention is shown schematically in a preferred embodiment. This is a modification of the memory unit 100 according to FIG 1 , wherein the same components are provided with the same reference symbols. In this respect, reference is also made to the description there.

For the memory unit 200 (only) a memory cell array 104 of a memory bank 102 is shown as an example. As also mentioned with regard to memory unit 100, a multiplicity of memory banks can be provided, which in turn each have a plurality of, for example four, memory cell arrays. In addition to memory unit 100, in memory unit 200 for memory cell array 104 (at least) one first processing unit 122, one second processing unit 124 and one adder unit 126 are now provided. As with the column decode unit 114 above 1 explained, two processing units 122, two second processing units 124 and two adder units 126 can also be provided if, for example, there is not enough space for a larger (or sufficiently large) unit on one side of the field (in other words, a unit can also be in divided into two parts, one placed at the top and the other at the bottom of the field).

The functioning or the interaction of these three units or components as well as with the other units or components already mentioned is in 2 shown schematically on the right. As mentioned, input elements A (also referred to as activations) for a layer of a neural network and weights W are initially stored in different rows of the memory cell array. The input elements A are then loaded into the primary sense amplifier 114, the weights W into the first processing unit 122. The first processing unit serves a purpose comparable to that of the primary sense amplifier and can also be structurally constructed in the same way or similar, ie work as a "latch".

From there, the input elements A and the weights W are then loaded into the second processing unit 124, which is used for translation, and multiplied therewith, as indicated by the double-headed arrows pointing to the left. The shift is a substitute for multiplying by 2". For example, the processing unit may receive the input feature map and the weights. The input feature map is then shifted by the weights. Since the weights are just powers of two, it can so that a multiplication can be represented.

These are then added or accumulated (in groups of four products) by means of the adding unit 126 . The result can then also include several bit positions, since the results (words) of the shift operation (for which this also already applies) are added. The two results of the addition(s) shown in the example can then finally be added, for example using the secondary sense amplifier 120 (if, as here, there is no more space for further adders), in order to obtain a sum at the end that is required for further processing . The result of this addition can then be stored again, in particular in another memory bank.

In 3 a memory unit 300 according to the invention is shown schematically in a further preferred embodiment. The memory unit 300 is shown by way of example as part of a computing unit 302 from which, for example, a neural network can be placed in the memory unit 300 for processing.

The memory unit 300 has eight memory banks 102, for example, which in turn each have four memory cell arrays 104, for example. Each memory bank 102 or each memory cell array 104 can, as in relation to the memory unit 200 in 2 explained be trained. For the sake of clarity, the individual components or units are not shown next to the memory cell arrays. Memory banks 102 are now in divided into two groups, a first group 130 and a second group 132.

In 4 is a schematic example of a sequence of a method according to the invention in a preferred embodiment, i.e. the processing of a neural network 40. First, a layer 402 is processed or calculated by output elements R (as a result) based on input elements A and weights W as already mentioned above explained to be determined. Another layer 404 is then calculated in the same way, but with the output elements of the previous layer as input elements and with appropriate weights.

Claims (11)

Memory unit (200) with one or more memory banks (102), each memory bank (102) having one or more memory cell arrays (104) with rows and columns of memory cells (106), each memory cell array (104) being assigned a row decoding unit (116), a column decoding unit (118) and a primary sense amplifier (114), each memory bank (102) being assigned a secondary sense amplifier (120), wherein the memory unit (200) is set up in such a way that memory cells (106) can be selected by means of the row decoding unit (116) and the column decoding unit (118), that in order to read out the memory cells (106), a signal can be amplified by means of the primary sense amplifier (114), in that bits stored in a row of memory cells (106) are loadable into the primary sense amplifier (114), and that a signal obtained from the primary sense amplifier (114) can be amplified by means of the secondary sense amplifier (120), each memory cell array (104) being assigned a first processing unit (122), a second processing unit (124) and an adder unit (126), wherein the memory unit (200) is further set up in such a way that bits stored in a row of memory cells (106) can be loaded into the first processing unit (122), in that bits present in the primary sense amplifier (114) and bits present in the first processing unit (122) are loadable and processable in the second processing unit (124), and that bits processed in the second processing unit (124) can be loaded into the adding unit (126) and processed therein. memory unit (200) after claim 1 , which is set up in such a way that in the adding unit (126) four elements can be processed with one another, in particular can be added. memory unit (200) after claim 1 or 2 , which is set up for processing at least one layer (402) of a neural network (400), and which is set up for this purpose such that weights (W) to be processed and input elements (A) of the layer in different rows of a common memory bank (102) are stored, that the weights (W) are loaded into the first processing unit (122) in response to an execution instruction, that the input elements (A) are loaded in the primary sense amplifier (114) in response to the or another execution instruction, that the weights ( W) and the input elements (A) are processed together by the second processing unit (124), in particular multiplied, and a result is loaded into the adding unit (126), and that in the adding unit (126) the results are processed together as operands, in particular to be added. storage unit (300) after claim 3 Having a plurality of memory banks (102) divided into a first group (130) and a second group (132), wherein the memory unit (300) is further set up so that the input elements (A) in a memory bank of the first group (130) are stored, and that results (R) determined in the adding unit (126) of this memory bank are stored in a memory bank of the second group (132), in particular as input elements for another layer (404) of the neural network. storage unit (300) after claim 4 , which is further arranged to store weights (W) of the other layer in a memory bank of the second group (132). Use of a memory unit (200, 300) according to any one of the preceding claims for processing at least one layer (402) of a neural network (400). use after claim 6 , in which a multi-bit neural network is processed. A method of processing a neural network (400), wherein layers (401, 404) of the neural network (400) are stored using a memory cher unit (200, 300) according to one of the Claims 1 until 6 be calculated. Arithmetic unit (302) with a memory unit (200, 300) according to one of Claims 1 until 6 , which is set up to follow all the procedural steps of a procedure claim 8 to perform. Computer program that a computing unit (302) after claim 9 prompted to follow all procedural steps of a procedure claim 8 to be performed when it is executed on the computing unit (302). Machine-readable storage medium with a computer program stored on it claim 10 .

Images