EP3704638A1

EP3704638A1 - Neural network representation

Info

Publication number: EP3704638A1
Application number: EP17811855.0A
Authority: EP
Inventors: Simon WIEDEMANN; Wojciech SAMEK; Thomas Wiegand; Klaus-Robert MÜLLER
Original assignee: Fraunhofer Gesellschaft zur Foerderung der Angewandten Forschung eV
Current assignee: Fraunhofer Gesellschaft zur Foerderung der Angewandten Forschung eV
Priority date: 2017-10-30
Filing date: 2017-10-30
Publication date: 2020-09-09
Also published as: WO2019086104A1

Abstract

Efficient neural network representations, their derivations and their processing such as their usage in performing a prediction using the neural network represented by such representation are described.

Description

Neural Network Representation

Description

The present application is concerned with an efficient neural network representation, its derivation and its processing such as its usage in performing a prediction using the neural network represented by such representation.

A neural network is composed by a set of neurons that are connected in a particular arrangement. Usually, the connectivity of the neurons can be represented by a set of matrices or tensors. When composing such a network, a common step is to quantize the matrix/tensor values (that is, each element of a matrix/tensor is allowed to only take values from a finite set of real values, instead of the entire R).

In the following, for ease of understanding, we will refer to a matrix representation of such a neural network, although tensor representations may be used as well.

Implementing neural networks on embedded systems (such as mobile phones, watches, autonomous cars, etc. ) is of great interest. However, typical neural networks require the calculation of matrix-vector multiplications of large matrices and vectors, which are highly computationally expensive. In addition, as was stated in [CBD14], multipliers are the most space and power-hungry arithmetic operators of the digital implementation of deep neural networks. This strongly difficults or even prohibits their deployment into embedded systems that are constrained by their limited hardware capacities such as storage and energy. Hence, in recent years, a growing number of work has focused on trying to compress the information contained in them in order to ease their resource requirements [LL16, CB16, LTA15, GAGN 15]. To a large extent this work used the technique of quantization in their compression pipeline, which maps the weight values of each connection into an element of a finite set of real numbers.

Thus, as became clear from the above discussion, neural networks become a tool which is more and more dedicated to be used in the field, i.e. , in handheld or mobile devices, where the computational power is limited and the energy consumption needs to be low owing to limited battery resources. Technically, the advantages resulting from decentralizing such neural networks may be huge. Decentralizing neural networks avoids, for instance, data gathered at decentralized sites to be send to a server for being subject to neural network based prediction with sending the prediction result back to the data origins. Imagine, for instance, speech recognition. Mobile phones offering speech recognition may perform the neural network based prediction themselves without having to send the audio data or intermediate descriptors obtained therefrom, to a server to obtain the recognized text back from the server, thereby saving transmission bandwidth as well as extending the applicability of the speech recognition functionality to occasions where the connectivity is intermediately interrupted.

Accordingly, it is an object of the present invention to provide a neural network representation, a derivation of such neural network representation and the usage of such neural network representation in performing neural network based prediction so that the usage of neural networks becomes more effective.

This object is achieved by the subject-matter of the independent claims of the present application.

Inter alia the present application is based on the finding that the usage of neural networks may be rendered more efficient by elevating neural network computation not only by trying to increase the number of weights of the neural network being set to zero, but also aiming at obtaining a representation of the neural network where the number of actually assumed discrete weight values of the weights of the neural network is controlled, or kept low, or, differently speaking, where the probability distribution of the actually assumed discrete weight values has a reduced entropy. This "aim" may be enforced with respect to a predetermined matrix or tensor of the neural network or a subsection thereof or, more generally, with respect to a portion of a predetermined matrix or tensor of the neural network with the portion comprising the whole predetermined matrix or tensor or a subsection thereof.

The result of enforcing this aim is the fact that many neurons of the neural network share the same discrete weight value. It should be recalled that the weights of matrices/tensors of the neural network representation describe the amplification or strength at which activations of neurons of the neural network are forwarded. One could say they describe the forwarding of the activations from neurons to the other in downstream direction, i.e., prediction direction, but it should be said that this statement shall then nevertheless be applicable to neural network topologies other than layered feed forward — networks such as recursive neural networks, for instance. This statement shall apply for the remaining description. Enforcing the above-outlined aim results in many weights sharing the same discrete weight value a property which is briefly called the weight sharing property, or WS property, in the following. Another insight underlying the present application is the fact that the property of a neural network representation to comprise weights which share the same discrete weight value mutually may be exploited so as to significantly reduce the task of executing the neural network prediction. For instance among the computational tasks to be performed when executing a neural network based prediction multiplications and summations turn out to be the most prominent ones with respect to the weights of the neural network. Among these two tasks, the multiplications are the most cumbersome ones so that reducing the number of multiplications having to be performed results in an increase of execution efficiency when executing the neural network prediction. Weights sharing the same discrete weight value, however, enable to exploit the distributive law, for instance: the way a predetermined matrix or tensor of a neural network representation controls an amplification at which activations of neurons of the neural network are forwarded downstream, involves the computation of a dot product of the predetermined matrix or tensor and an array of activations, the dot product involving sums over component-wise products between weights of the predetermined matrix or tensor and activations of neurons of the array of activations. Weights sharing the same discrete weight value enable the application of the distributive law in that the summation over the dot products may be, as far as the partial sums over the products are concerned which relate to weights which share the same discrete weight value, turned into a product between this shared discrete weight value and the sum over the other factors, namely the corresponding activations. Thus, designing the representation of a neural network in a manner so that the representation comprises one or more first data fields which indicate a set of predetermined discrete weight values actually assumed by at least one of the weights of a predetermined matrix/tensor portion of the predetermined matrix or tensor of the neural network and, for each of the one or more first data fields, a second data field, disjoint to the respective data field, which indicates for each predetermined discrete weight value indicated by the respective first data field, positions of weights in the predetermined matrix/tensor portion assuming the respective predetermined discrete weight value, enables to exploit the distributive law in the just-outlined manner.

Another way of exploiting this kind of representation of the neural network weights is to perform a pre-computation of the occurring component-wise products involved in dot product followed by selectively inserting the pre-computed component-wise products into the individual sums involved in the dot product computation. And an even further possibility of exploiting the just-outlined representation of the neural network weights is the possibility of decomposing the dot product into a weighted sum of binary matrices/tensors each weighted with one of the discrete weight values and having all zeros but the positions corresponding to the positions of the original weight sharing predetermined matrix or tensor where the latter has weights corresponding to the respective discrete weight value.

The above-identified idea of enforcing the weight sharing property may be implemented, from scratch, when training the neural network or as a kind of post processing step by quantizing a pre-!earned state of the weights. In addition to the enforcement of the weight sharing property using optimization which aims at reducing the cardinality of the discrete weight values and/or the entropy a probability distribution (reflecting the frequency of being assumed by the weights) of the set of discrete weight values, optimization may be used in order to reduce a cardinality of a set of discrete activation values of activations for a predetermined set of neurons of the neural network and/or an entropy of a probability distributinon the set of discrete weight values. Reducing the domain of activations, namely the set of discrete activation values onto which activations of neurons are to be quantized during prediction which when executing the neural network, with the set of discrete activation values being conveyed, for instance, within the representation of the neural network, enables, for instance, to increase the efficiency or reduce the complexity of the mathematical algorithms, namely multiplications and additions, to be performed during neural network based prediction.

The latter optimization aim, namely the reduction in the size of the set of discrete activation values, is, in accordance with another aspect of the present application, used without the enforcement of the weight sharing property. That is, in accordance with an aspect of the present application, neural network representation derivation involves the use of an optimization scheme with an objective function which depends on a cardinality of the set of discrete activation values or on frequencies at which discrete activation values of the set of discrete activation values are assumed in a manner so that the cardinality and/or the entropy is reduced.

Advantages of embodiments of the present application are the subject of dependent claims. Preferred embodiments of the present application are described below with respect to figures, among which: - - Fig. 1 shows an example for a predetermined matrix or tensor, here exemplarily a predetermined matrix, with respect to which embodiments of the present application may advantageously be applied;

Fig. 2 shows a schematic diagram illustrating a portion of the neural network described by a matrix according to Fig. 1 ;

Fig. 3 shows a possible representation for the matrix example of Fig. 1 in accordance with an embodiment;

Fig. 4 shows a schematic diagram illustrating a representation for the matrix example of Fig. 1 using pointers for pointing to the positions of the discrete weight values;

Fig. 5 shows a schematic diagram illustrating a representation of the matrix example of Fig. 1 designed for a low data amount for the representation and lending itself from alleviating a dot product computation using the matrix example by matrix decomposition;

Fig. 6 shows a schematic diagram illustrating another example for a representation of the matrix example of Fig. 1 designed for low storage amount and dot product computation alleviation by matrix decomposition;

Fig. 7 shows a schematic diagram illustrating another example for a representation of the matrix example, designed for alleviating the dot product computation by exploiting the distributive law;

Fig. 8 shows a schematic diagram of another example for a representation of the matrix example, modified compared to the one of Fig. 7 with respect to the discrete weight value indication per row;

Fig. 9 shows a schematic diagram illustrating an example for a representation of the matrix example, aiming at reducing the number of operational operations for the recipient of the representation and using the representation for performing prediction, and designed for alleviating the prediction by matrix decomposition; Fig. 10 shows a schematic diagram illustrating another example for a representation of a weight matrix designed for alleviating the prediction using matrix decomposition;

Fig. 1 1 shows a schematic diagram illustrating an example for a representation aiming at alleviating the prediction process by allowing for a precomputation of component-wise products to be inserted as add-ends into the summations of a component-wise product prescribed by the dot product;

Fig. 12 shows a schematic diagram illustrating a representation modified relative to Fig.

7 so as to be more hardware friendly;

Fig. 13 shows a schematic diagram of a further more hardware friendly implementation of a representation allowing to alleviate the prediction process by exploitation of the distributive law;

Fig. 14 shows a schematic diagram of another example for a representation allowing for alleviating the dot product computation by precomputation of component-wise products;

Fig. 15 shows another example for a representation allowing for alleviating the dot product computation by exploiting the distributive law;

Fig. 16 shows a schematic diagram of a representation modified compared to Fig. 15;

Fig. 17 shows an encoder scheme suitable for generating a neural network such that it may be advantageously represented by representations discussed herein, thereby reducing resource requirements to model a neural network, wherein the scheme shows different transformation steps of a given neural network model until a compressed domain representation of it results, wherein procedures are - represented by boxes and take as input a neural network and output same with changed properties, and wherein types of input-output scenarios are numbered and depicted in this Figure.

Fig. 18 shows a schematic block diagram of the encoder scheme of Fig. 17 in an expanded version; Fig. 19 shows a schematic diagram illustrating possible tasks to be performed by a decoder which takes advantage of representations discussed in the present application;

Fig. 20 shows an example for a neural network, i.e. of its general construction and subdivision into layers and layer sets, here exemplarily designed as a neural network designed to receive an image as an input and output probability values or to discriminate images of different contents;

Fig. 21 shows a pseudocode algorithm for performing the prediction using the neural network of Fig. 20;

Fig. 22 shows an example for a pseudocode listing of an algorithm ought to advantageously compute the dot product on the basis of representations assisting in alleviating the computation via exploitation of the distributive law;

Fig. 23 shows a schematic flow diagram illustrating the steps performed by the algorithm of Fig. 22;

Fig. 24 shows an exemplary pseudocode for an algorithm for performing a dot product advantageously by exploiting the distributive law by showing amendments compared to the example of Fig. 22;

Fig. 25 shows a flow diagram illustrating steps performed by the algorithm of Fig. 24;

Fig 26 shows a schematic diagram illustrating the dot product computation for the example of Fig. 1 ;

Fig. 27 illustrates a decomposition of the computation of Fig. 26 in accordance with examples of the present application according to which the computation is alleviated by a precomputation of component-wise products;

Fig. 28 shows a pseudocode of an algorithm ought to perform the dot product computation by alleviating same using the precomputation concept for component-wise products; - ^{. .} _ _ illustrates the concept of alleviating the dot product computation by matrix decomposition; a illustrates special characteristics of a convolutional matrix / tensor; b to 30e show a schematic diagram illustrating representations of a convolutional matrix / tensor example according to different embodiments reusing some concepts of the above representations; a shows and explains the advantages of treating the predetermined discrete weight values separately from an extra discrete weight value such as the most frequently assumed weight value in case of the extra discrete weight value being non-zero and how a recipient of the representation may recover the matrix or tensor represented by a representation according to embodiments of the present application by simple addition of an all one matrix times the extra discrete value when setting the discrete weight values to be represented as offsets relative to the extra discrete value; b shows an example for a representation seeking to alleviate the dot product by matrix decomposition, wherein different formats are used to signal the positions for each of the discrete weight values, respectively; shows a schematic diagram illustrating the interconnectivity of two consecutive layers of a neural network, here exemplarily a full connectivity; shows a schematic block diagram of an apparatus for performing a neural network-based prediction by performing the prediction by exploiting favorable characteristics of representations presented in the present application; shows a schematic diagram illustrating a representation of a neural network and its data fields in accordance with an embodiment; shows a schematic diagram illustrating different concepts in alleviating neural network-based predictions using concepts of the present application; illustrates how to deal with an extra discrete weight value of non-zero in accordance with an embodiment; Fig. 37a, b and c show examples for a general structure of a representation designed for alleviating a neural network-based prediction using any of the three concepts outlined in Fig. 35;

Fig. 38 shows a schematic block diagram of an apparatus for deriving a neural network representation in accordance with an embodiment of the present application; and

Fig. 39 shows a schematic block diagram of an apparatus for deriving a neural network- based representation by training from the scratch or quantizing a pre-trained neural network in terms of weights and/or activations.

As should have become clear from the above introduction and brief description of aspects of the present application, in accordance with one aspect of the present application, embodiments of the present application are concerned with weight sharing, while others are additionally or alternatively concerned with reducing the entropy or cardinality of discrete activation values. The following description starts with an introduction which motivates thoughts underlying these aspects of the present application.

As became clear from the above discussion, quantizing weight values of a neural network inevitably induces the statistical property that subsets off connections share the same weight value. As already outlined above, this property is called weight sharing property or WS property in short during the following description.

1. Motivation

As we will describe later in more detail, we can further ease the resource requirements by choosing an adequate format for representing the information content in the network that leverages this statistical property.

For example, in a typical fully-connected feed forward neural network model, the weights of a layer are represented as single-precision floating-point matrices. Hence, the pre-activation values of the subsequent layer are calculated element-wise as

4 = Zf w^-\

(1 ) Please note that pre-activation values are the result of the sums of products often discussed throughout this document, namely the sums involved in the dot product of a predetermined matrix or tensor, for instance, with an array of activations. The pre- activation values are then, within the neural network, the subject to a non-linear function, for instance, the output of which is then mapped onto the set of activation values which may correspond to the reduced set of discrete activation values with the aspect also mentioned above as being alternatively or additionally performed in addition to the exploitation of the weight sharing property.

Proceeding further with the motivation of the thoughts underlying the present application, it becomes clear that, by performing the matrix-vector multiplication between the activation values of the previous layer (denoted with a'^{- 1} ) and the weight matrices of the current layer (denoted as w^l ) this representational format requires M multiplications and summations per output node, where M E N denotes the input dimension (thus, N x M in total, where IV e fH denotes the number of output nodes).

However, by assuming that the network shares a great part of its values with other connections, we can rewrite the above matrix-vector operation as

where the set ]\_k is the index set containing the indices j of the input nodes that share the same weight value w_k ^l .

In this format we can see, that the number of multiplications that need to be performed depend on the number of different weight vales that point to the node ί in the layer I . That is, the greater the weight sharing property in a neural network is, the lesser the number of multiplications that need to be performed. In addition, if the matrix is sparse we can save an even greater number of operations by neglecting the sum of the input values that share the weight value 0.

In section 3 we will describe a set of new formats for representing weight matrices with shared weights. Each format will be designed such, that they are efficient with respect to a specific optimization objective. But, before we describe the new formats in more detail, we will give a short introduction into formats for storing sparse matrices in the next section for the sake of understanding. A brief note shoud be made as to the explicit mentioning of a matrix-vector prduct. Merely for sake of an easier undnerstanding, this case is further discussed below. In this case, the activations are described as to be an array or vector, a one-dimensional array. However, when doing prediction, one can take a set of inputs und bundle them into a matrix form and perform a forward pass on this matrix. In this case, the activations are matrices of higher dimension and not vectors. The subsequent description is readily applicable to the matrix-matrix product cases and shall not be understood as being limiting to these cases. A similar statement is true with respect to transferring the teaching to tensor notations.

2. Sparse matrices

Sparse matrices are those which contain many zero elements. Since they appear very often in various scientific fields, a great amount of work has been put into deriving new representations (formats) that are more efficient with respect to different optimization objectives. Nevertheless, all formats have in common that they take advantage of the statistical property that there are many zeros present in the matrix.

The most common formats are probably the COO, CS and ELL formats.

In order to facilitate the understanding of the different representations, we will consider the following example

2.1 COO

The coordinate format or COO in short, stores the information of a sparse matrix within three arrays: the weights-, row Index-, collndex-array.

The weights-array contains the values of all non zero elements in the matrix, and rowlndex- and collndex-array their respective row/column positions. Hence, the ab matrix would be represented as follows

W : [4,4,2,4,4,3,3,2,4,4,3] row! ·■ [1,1,2,2,2,3,3,4,4,5,5]

coll : [2,6,2,3,6,1,2,3,4,1,5]

This representation can ease storage requirements for large spare matrices by just storing the non zero elements and their respective positions. In addition, it allows for easy new matrix element insertion, which is advantageous when building new sparse matrices.

Nevertheless, this format is not efficient with regards to fast sparse-matrix-vector multiplications.

2.2 CSR

The compressed sparse row format also represents the sparse matrix by using three arrays. It stores the non-zero values and column indices in row major order and adds a pointer, which indicates where a new row starts. Hence, the previous example would be stored as follows

W : [4,4,2,4,4,3,3,2,4,4,3]

coll [2,6,2,3,6, 1,2,3,4,1,5]

rowPtr : [0,2,5,7,9,11]

It not only reduces the amount of storage required to represent the sparse matrix, but it also allows for fast matrix-vector multiplications.

Nevertheless, this format is inefficient with respect to vector architectures.

2.3 ELL

This format stores the non zero values and their corresponding column indices, but fixes the number of non zeros per row and pads with an additional symbol * empty spaces. Subsequently, it transposes the entries in order to allow for coalesced memory access. Hence, the above example would be represented as

W : [4,2,3,2,4,4,4,3,4,3,* ,4,*,*,*]

coll : [2,2, 1,3,1,6,3,2,4,5,* ,6,*,*,*] This format is efficient with respect to paraiielization, but it becomes inefficient with respect to storage requirement when the max number of non zero elements per row is much greater than the average number of non zeros per row.

2.4 More sparse formats

Other formats leverage additional statistical properties, for example when a matrix is structured in a particular way. Diagonal, block, lower/upper triangular etc. matrices are examples of those. Storage requirements can be greatly reduced if we spare the signal- ization of the position indices by reordering the non-zero values such, that they follow the structure.

In the next section we will show with the example of matrices that have shared weights how different formats can be derived for different optimization objectives and by leveraging different statistical properties present in them.

3. Formats for matrices with shared weights

In this section we will systematically derive a collection of new possible representations of compressed matrices. We will think of many possible statistical properties that a compressed matrix could adopt, and derive different formats that have advantageous properties for different purposes. They will be optimized for the following purposes:

1. Minimal storage

2. Minima! energy

3. Vector architectures

4. Matrix element manipulation

As an intermediate note, it is submitted herewith that the following examples will be revisited later again in order to describe in more detail as to how the formats/representations presented in the following in the present section enable the recipient of the format/representation, such as the apparatus which performs the prediction using a neural network represented by the corresponding representation/format, to more efficiently perform the prediction task.

Resuming the collection of possible representations, we will consider the class of dense matrices with quantized element values. That is, there exist a finite set of weight values (the alphabet) from which the matrix elements can adopt their values. Hence, a genera! form, we can represent this class of matrices either by two arrays:

• W≡ the shared weight values

• wl≡ an N x M integer matrix containing the index pointers to W or by a dictionary, where the keys list the alphabet values and their items their corresponding positions in the matrix.

To illustrate this, let's consider the previous example, that is matrix A depicted in Fig. 1 . Reference sign 10 is spent or assigned to matrix A. Matrix A will be used as a representative example throughout this document as an example for a "predetermined matrix" with respect to which, for instance, the weight sharing property is exploited or, as described in more detail below, enforced. "Enforcement" denotes the fact that the degree of weight sharing property may be increased at the time of training the neural network or quantizing a pre-learned neural network with putting specific emphasis on the increase in the weight sharing property. "Exploiting" the weight sharing property is done in the following two-fold, i.e. , at two occasions, namely by designing the representation/format accordingly, i.e. , forming digital data representing the neural network accordingly, and when performing the prediction on the basis of a neural network having such a weight sharing property. Before proceeding with a description, it should be noted that, although the following description continuously uses matrix A as an example for a predetermined matrix with respect to which the weight sharing property is exploited, embodiments of the present application are not restricted to neural network representations using matrices. Rather, a tensor representation may be used as well. For sake of easy understanding, however, the following description concentrates on matrix representations. Further, it is clear for a man skilled in the art that the number of columns M and the number of rows N, namely six and five, respectively, have been chosen merely for sake of a specific example and in reality the number of weight will be by far greater than that.

Further, before proceeding with the description certain formats/representations ought to exploit the weight sharing property, for sake of an easy understanding of the following description, Fig. 2 depicts as to how the weights of matrix 10 define the neural network. Fig. 2 depicts certain neurons 12 of the neural network, mainly N = 5 neurons. They may belong to a certain layer 1 4 of the neural network, and each of these neurons 12 is connected to, in terms of a prediction direction, set of preceding neurons 16, namely M such neurons 16 wherein these neurons 16 may, for instance, be part of predecessor layer 18 of the neural network. The connections are illustrated in Fig. 2 using lines 20 merely with respect to one representative neuron 12, namely the left hand one. Matrix A controls the amplification or strength at which activations of neurons 16 are forwarded downstream to the respective neuron 12. Each row of matrix A is assigned to a certain neuron 12. In Fig. 1 , for instance, the first row of weights is assigned to reference sign 22. The weights 24 of matrix 10 within this row 22 control the amplification of the activations along connections 20 as depicted in Fig. 2. In other words, each neuron 12 receives a weighted sum of activations, weighted according to the weights 24 of the corresponding row 22 of matrix A, wherein the result of this weighted sum is the aforementioned pre-activation which is also indicated as y in Fig. 2, namely y = A ^* x, wherein this pre-activation is then subject to a nonlinear function which maps the pre-activation nonlinearly onto the output of the respective neuron 12, i.e., its activation.

Let's resume the presentation of examples of representations taking advantage of the weight sharing property. Generally, we could represent matrix A as depicted in Fig. 3, for instance. Fig. 3 illustrates as to how a representation of the neural network could look like with respect to matrix A. It comprises, with respect to matrix A, a first data field 30 indicating the set 32 of discrete weight values assumed by any weight 24 within matrix A, i.e., the complete set of discrete weight values assumed by at least one weight 24 of matrix A. it goes without saying, that set 32 is usually composed of one discrete weight value 34 which may be treated separately because it is zero and/or it appeared most frequently among the weights 24 of matrix A on the one hand and the remaining set 36 of discrete weight values, often called the "set of predetermined discrete weight values^'' in the following description. Besides first data field 30, the format/representation comprises a second data field 40 which indicates for each predetermined discrete weight value of set 36, the positions of weights 24 in matrix A which assume the respective predetermined discrete weight value. According to the example of Fig. 3, this is done by indices 38 each pointing to the respective discrete weight value in set 32 and being associated with the corresponding weight 24 in matrix A by its position in data field 40. To be more precise, the indices 38 in data field 40 are ordered in a manner so as to be uniquely associated with weights 24 of matrix A which circumstance is achieved according to the example of Fig. 3 in that same are ordered in the same two- dimensional manner as the weights 24 in matrix A are, wherein indices 38 corresponding to discrete weight values of set 36 have indices of value other than indices 38 corresponding to weights 24 of value 34, thereby separating indices 38 corresponding to weights having a value within set 36. An alternative way for a representation is depicted in Fig. 4. Here, the second data field 40 comprises, for each discrete weight value of set 32, a set of pointers indicated in square brackets in Fig. 4, namely one for each weight 24 in matrix A assuming the respective discrete weight value and pointing to its position in matrix A. Here, the pointers are addresses addressing the respective position with assigning addresses to weights 24 in matrix A one-dimensionally in a raster scan order from top to bottom with starting at zero for the upper-left weight 24 of matrix A. The pointers are indicated in Fig. 4, generally using reference sign 42.

That is, in Fig. 4 we represented the index positions using the lexographical nomenclature (that is, / = j + M * i, the h position equals the value of the y^'-th column plus the i-th row times the number of columns ).

We will now systematically cover as many statistical properties we can encounter in such class of matrices, and propose formats optimized for the above listed properties. For that, we will start with the least constrained properties and progressively add more statistical constrains.

Before starting with this description, maybe it's worthwhile to again point out a common characteristic of the formats discussed herein. In particular, this characteristic is the fact that the representations discussed herein have an association mechanism which leads from the first data field 30 or, to be more precise, from the discrete weight values indicated therein, towards the position indication by way of the second data field 40. That is, the format or representation allows for the recipient of the representation/format to inspect the first data field 30 in order to obtain information on the discrete weight values within set 32 or 36, and then inspect the second data field 40 in order to obtain, for each discrete weight value indicated by the first data field the information on where in the corresponding matrix 10 weights 24 are positioned which assume the respective discrete weight value. The representations/formats presented herein, thus, avoid any necessity to perform this sort of analysis, namely searching for weights sharing a certain discrete weight value, at the time of performing the neural network prediction. Further, it is submitted before proceeding with the description of certain embodiments for formats and representations, respectively, that the indication of the discrete weight values within set 32 or set 36 may alternatively be split-up onto more than- one first data field 30 with one second data field 40 for each such first data field. For instance, such splitting-up onto several first data fields 30 may be performed on a per row basis, i.e., one first data field per row of matrix 10. Other examples may become clear from the following examples.

3.1 No probability model

Here we consider the type of matrices where

^• There is no known probability model

That is, apart from the fact that the weights share their values, the decoder has no a priory knowledge about the statistical properties that the matrices might have. In such cases, the only thing the decoder can incur are the set of positions of the last weight. That is, given the matrix dimensions (N x M) and the positions of nw - 1 weights (where nw denotes the total number of distinct weights. In the above example we had nw = 4), the decoder can fully predict the positions of the last weight. Hence, for all coming formats we can choose to not send the positions of one specific weight i.e., weight value 34. Obviously, the right choice for the encoder will be to spare the information of the positions of the most frequent weight. Alternatively, zero could be chosen by default.

Considering this, we propose a variety of new formats designed specifically for the different objectives listed above.

3.1.1 Minimal storage

In the following, we will propose new representations of the WS-matrices with the attempt to achieve highest compression ratios without losing any information. That is, we aim to find representations from which we can reconstruct again it's dense version.

Format 1 :

We store the following set of arrays

^• W_max, N, M≡ Most frequent weight value, row/column dimensions

• W≡ Alphabet. The set of distinct weight values (thus stored only once). (Side note: throughout this description, we will refer weight and alphabet values as synonyms).

• /≡ Position indices of each alphabet, ordered in alphabet major order. • wPtr≡ Indicates at which locations in / the next set of sequence belongs to the next alphabet, correspondingly to the W array.

Now, we can think of a variety of possible representations of this form, namely:

• / can store the positions either using the lexographical index or any of the sparse-like formats (we remark, that in the later we spare the storage of the values array, only the index arrays of the respective sparse formats would be required). In the later, / would be decomposed into a set of arrays (typically 2). For example, we could store the coll in row major order, per alphabet. rowPtr would point to the entries in coll, indicating the start of a new row. This would mimic the CSR format, but concatenated in alphabet major order.

^• If / stores the lexographical index, then wPtr may point directly to it's entries. But if / is stored in a sparse-like format, it is only required to point to the locations in the respective array where the change in a weight value can be uniquely identified. In the last example (storing / in csr-like format), wPtr could point either to locations in coll or in rowPtr.

If we store coll and scan the elements in major row order, we know that each time the column index drops, a new row starts. Hence, we only need to send the specification of a new row starting when this case doesn't apply (for example diagonal matrices). A similar logic applies for wPtr. Each time the lexographical number drops, we know that the next sequence of positions belong to a new weight value.

Hence, in order to further reduce the storage requirements, we may:

^• Reduce the size of wPtr by just pointing to the places in / where no obvious weight change is performed.

• Reduce the size of rowPtr by using the same logic. Here, we could even spare saving the wPtr array by keeping track of the number of rows jumped

^• Alternatively, we can pad with * locations in coll where each time a row jump needs to be specified. Nevertheless, the jump symbol * would have the same bit overhead as the indices of coll .

^• In addition, in order to reduce the bits overhead, we may store the difference - index values (this applies to all arrays, with the exception of the alphabet array W). When storing the difference of the coll instead of their actual values, the same logic applies whenever we encounter a negative value. However, since it is likely that the absolute value of the negative values are large (and thus we do not profit from bit-size reduction), we might consider only taking positive values and signalize the start of a new row by padding with 0. In this setting, the trade-off between the reduction of the bit- size overhead and the addition of new bits due to the padding has to be taken into account.

Taking the previous example, we could store A as depicted in Fig. 5, where the representation 50 requires 23 numbers instead of 28 or 30, to represent matrix A (10) which would be the case if we would store it in CSR or dense format respectively.

To be more precise, according the example depicted in Fig. 5, the representation 50 comprises the first data field 30 indicating the predetermined discrete weight values 36. The representation 50 comprises, additionally, the indication 52 of the extra discrete weight value, which is 0 in the present case, i.e., discrete weight value 34. Naturally, representation 50 also comprises an indication 54 of the size of matrix 10, i.e., the number of rows 56 and the number of columns 58 indicated by N and M, respectively. The data field 40 associated with data field 30 indicates the positions of the weights assuming any of the discrete weight values 36 by way of column indices 60 which merely indicate the column index of each search position, wherein the association of the column index values within data set 40 and the columns of matrix 10 are indicated in Fig. 5 at 62. In accordance with the example of Fig. 5, the column indices 60 are ordered in a list of column indices where the column indices are arranged along in the following order: the weight 24 in matrix 10 are traversed in an one-dimensional order, namely here in Fig. 5 exemplary in a raster scan order which leads, or traverses matrix 10, row-wise from top-left to bottom-right of matrix A. In particular, matrix 10 is traversed three times along this order. During the first traversal, the column index of each weight 24 is entered into the list 64 of column indices whenever a weight 24 is encountered during this traversal or scan which corresponds to the first discrete weight value 36 indicated in the list 66 of discrete weight values 36 of data field 30, which is 4 in the present case. Then, in the second scan or traversal, all column indices of those weights 24 are inserted or entered into list 64 at the time of encountering a weight 24 assuming the second discrete weight value in list 66 which is 3 in the present example. This is done in this order along the order of the discrete weight values 36 in list 66. Here, there are three discrete weight values 36 and list 66 so that three scans are performed. Natu- rally, the number of discrete weight values 36 in list 66 may be different from three with a number of scans equaling the number of discrete weight values 36 in list 66.

Each time the column index 60 in list 64 is not monotonically increasing from one column index 60 to the next in list 64, it may be deduced by the recipient of representation 50, that this next column index 60 in list 64 relates to a different row. In this manner, list 64 forms a sequence of column indices 60 which is sub-divided into subsequences or continuous runs 68 of column indices relating to column indices 60 of weights 24 being a) in the same row of column 10 and b) being of the same discrete weight value 36. The subsequences 68 are arranged in list 64, owing to the sequential traversal of the posi- tons of matrix A once per discrete weight value 36, according to their associated row index row with subsequences 68 concerning a certain discrete weight value 36 preceding subsequences 68 concerning a next discrete weight value 36 in list 66. For some pairs of row index row/ and discrete weight value w/ there is no subsequence in lists 64 owing to a lack of a weight 24 in the respective row of the respective discrete weight value 36. In order to locate such missing subsequences as well as subsequences not detectable on the basis of an inspection of a maintenance or decrease in column index from one position in list 64 to the other, according to the example of Fig. 5, the representation 50 additionally comprises a list 70 of differentially coded pointers rowPfr which point to the position of subsequences 68 in list 64 not otherwise detectable or following a skipped subsequence. A recipient of representation 50, thus, would perform the differential decoding task 72 to detect the borders between consecutive subsequences 68 in list 64 and associate each subsequence 68 in list 64 to its respective pair of discrete weight value 36 and row index row/.

Alternatively, representation 50 could be stored, or could like, as depicted in Fig. 6. Here, column index list 64 of the second data field 40 is designed in a manner so that the column indices 60 are arranged in list 64 still according to the number of traversers corresponding to the number of discrete weight values 36, but whenever for a certain row of matrix A, no column index 60 is inserted into list 64 owing to the fact that the corresponding row does not comprise a weight 64 within that row which would comprise the discrete weight value corresponding to the current scan, an escape value 80 is inserted to list 64 instead of a corresponding index 60 in order to indicate this circumstance. By this measure, a recipient of representation 50 is able to associate each subsequence 68 of column indices 60 within list 64 to their respective associated pair of discrete weight value 36 and row of matrix 10, namely by a) checking whether consecu- tive column indices 60 in list 64 are equal or decrease with this check being indicated using reference sign 82, or by detecting b) that a certain row has been skipped in the current scan or traversal for the current discrete weight value 36 by encountering an escape value 80.

Format 2:

In the previous formats we ordered the position indices / in alphabet major order. Now we consider minimal storage requirements when we order them in rows major order i.e. , in the order 90 they are listed in list 66 as depicted in Fig. 5 exemplarily. That is, for each row we need to signalize which weight values appear and where. Hence, we propose the following format which we named compress shared weight row format (CSWR in short)

^• Contains the alphabet values

• Array of column indices, ordered in alphabet major order per row (thus, ordered in alphabet-row-major order).

^• Array of weight indices, that indicates which weights in W are present in the row (also ordered in row major order).

• Points to the locations in coll that corresponds to a specific weight, respective to wl (thus, also concatenated by row).

• Array pointer that indicates at which location in wPtr a new row starts.

Hence, the above matrix A ( 10) would be represented by representation 50 or format 50 in the manner depicted in Fig. 7. Here, the subsequences of column indices 60 in the column index list 64 comprised by the second data field 40 are collected differently. In particular, the column indices in list 64 are ordered in list 64 of Fig. 7 in accordance with a different order, namely according to an order according to which each row of matrix 10 as often as there are discrete weight values in set 36 as indicated by the first data field 30 within proceeding with the next row with the rows being traversed, for instance, from top to bottom. The number of subsequences 68 in Fig. 7 is, accordingly, the same as in the example of Fig. 5, with the order there between within list 64, however, being changed relative to the example of Fig. 5. Along the order of the occurrence, weight indices 90 in a weight index list 92 according to the example of Fig. 7, indicate for each subsequence 68 as to which discrete weight value the column indices 60 of the respective subsequent 68 relate to. That is, a weight index 90 at a certain rank in the order of list 92 indicates the weight which the corresponding subsequence 68 at the same rank within list 64 refers to, by indexing the corresponding weight 36 within the data field 30, namely within list 66. The i^th weight index 90 indicates the weight that the column indices 60 of the i^lh subsequence 68 relate to. Lists 66 and 92 together form an example of having one first data field 30 per row, in that certain portions, indicated using dashed lines in Fig. 7, point to those (sub)sets of discrete weight values in 66 which occur in a corresponding row.

Further, the second data field 40 comprises a sequences of pointers 94 each pointing to a starting position or starting rank of a corresponding subsequence 68 of list 64. That is, sequence 94 comprises pointers 96 pointing to the rank in list 64 where a corresponding subsequence 68 starts. Pointers 96 are ordered along the order of subsequences 68 within list 64, i.e., they monotonically increase. Further, second data field 40 comprises another pointer sequence 98, the pointers of which point to rank positions within pointer sequence 94 from which onwards within list 94 the pointers relate to a next row within matrix 10. That is, even the pointers 96 within list 94 is sub-divided into subsequences 100 each being related to a different row, and being ordered in row direction from top to bottom in matrix 10. The pointers 102 in list 98 point to the starting positions of the subsequences 100 within list 94 in row order.

Accordingly, a recipient of representation 50 is able to locate, for each row of matrix 10, the weights 24 assuming any of the discrete weight values 36. That is, in case of Fig. 7, the second data field 40 is structured into portions 104, namely one for each discrete weight value 36 indicated by the first data field 30, and for each such portion 104, there is a set 106 of pointers present in the second data field 40 which points to this portion 104. In the present case, the set 106 is structured in a manner so that the pointing to portions 104 is done in two stages: each portion 106 comprises a pointer 102 pointing to one or more pointers 96 which, in turn, point to the respective column indices 60 in list 64. The weight indication is done by providing each portion 104 with an index into list 66.

Fig. 8 shows an alternative for the representation 50 depicted in Fig. 7 in that the indirect weight indication by way of weight index list 92 is replaced by a direct indication of the weights in a corresponding list 1 10 of weights 1 12. List 1 10 merely differs from the weight index list 92 of Fig. 7 in that the corresponding weight is entered at the corre- •—spending rank position directly In this manner, this list 1 10 of weights is structured into one data field 30 per row and the second data field 40 comprises one portion 104 for each such first data field 30 along with one or more pointers 106 per portion 104. In this case, the recipient of representation 50 would use, per row, the one or more pointers 106 associated with a respective row also in order to access the corresponding discrete weight values for this row from list 1 10, namely by accessing the corresponding discrete weight values 1 12 at the corresponding rank positions, namely corresponding to the rank positions of the subsequences 68 within portion 104.

Thus, in Figs. 7 and 8 the indication of discrete weight values within matrix A is structured into one data field 30 per row, namely, in the case of Fig. 8, one data field 30 indicating, row individually, the discrete weight values within the respective row directly, and in case of Fig. 7 indicating the discrete weight values within the respective row indirectly by way of indices indicating the respective discrete values out of a reference list 66.

Again, the same techniques as described in format 1 can be applied in order to further reduce the storage requirement. For example, we can spare signalization of obvious changes at the wPtr array, since each time the next number in coll is lower than the previous one we know that the new sequence corresponds to a new alphabet value. In addition, in order to reduce the bit overhead we can store the difference of the index values instead of their absolute values at rowPtr. Hence, the above arrays 94, 98 could be rewritten as

wPtr = [7]

rowPtr = [2,2,1,2,1]

In the special case where the number of distinct weights per row is constant, we can further save storage by sending this constant number and sparing the signalization of the rowPtr array, since by keeping track of the number of weights jumped we can uniquely identify when the next sequence of coll belong to the next row. Nevertheless, signalization of empty rows would still be required.

Note the the above arrays wPtr and rowPtr can be merged wit the coll array, if we add 2 additional symbols to it. Namely, one indicating a change in alphabet value and the other a change in row.

Format 3: (CSWC)

The CSWR format can be trivially modified to account for iterations over column indices instead of row indices. That is, instead of scanning the values row-wise we can scan them column-wise. Hence, the coll array would be changed into a row! array, containing the row indices of the respective weight values per column, and the rowPtr to a coiPtr indicating the change in column. We respectively call this format CSWC (instead of CSWR).

3.1.2 Optimized for minimal energy

In this subsection it is important to consider formats that best trade-off between

• Minimal number of operations for performing a dot product, weighted by their energy consumption

^• Size of the format

For simplicity, we propose different formats where we focused on different strengths with regards to the above mentioned optimality requirements.

Format 1 (CSWR/CSWC):

In its original form (that is, no further storage minimization techniques are applied), CSWR/CSWC can serve as good candidate if we desire to iterate over the rows/columns of the matrices. The format is optimized for minimal number of operations required to perform a dot product per row/column.

Format 2:

Instead of iterating over rows, we can again consider iterating over the alphabet. That is, we could further decompose the sparse matrix into a sum of unit sparse matrices with positions and coefficients corresponding to the elements of the alphabet. This representation also minimizes the number of multiplications required per row.

Hence, in this case we store

W_max, M, N: Most frequent value, matrix dimensions

W: Alphabet

coll : Column indices in column and alphabet major order rowPtr: Row pointer in coll. It pads rowswith no entries using same number

and starts over the counter for each new weight Thus, we would store the representation 50 of A as shown in Fig. 9.

Thus, according to Fig. 9, the representation 50, again, comprises the information 52 and 54 along with a first data field 30 indicating by way of list 66 the predetermined discrete weight values 36, and the second data field 40 comprises a column index list 64 listing, again the column indices 60 in the order as described above with respect to Fig. 5. For an easier understand of the main issues of the embodiments described hereinafter, it shall be understood that the weight values may be given in data filed 30 and list 66, respectively, in a manner indicating the difference to the special discrete weight value 34.

A row pointer of pointer list 70 is also comprised by the second data field 40 in case of Fig. 9, here, however, allowing for an easier access to the subsequences 68 of column indices 60 and list 64. In particular, the pointers in list 70 are construed in a manner so that pairs of consecutive pointers in list 70 point to begin of a corresponding subsequence 68, if present, and the column index immediately succeeding the subsequence. If the pointers of such a pair point to the same position within column index list 64, this denotes that the corresponding subsequence is missing. The pairs of pointers in list 70 are ordered in subsequences, mainly list 70 comprises pointers relating to a first discrete weight value of list 66 in a first subsequence 1 21 within subsequences of pointers concerning the following discrete weight values 36 of list 66 following. The pointer indexing of the column indices 60 within list 64 by way of the pointers 120 within list 70 are reset between the subsequences 120 to 0. In a manner outlined in more detail below, the representation 50 of Fig. 9 qualifies itself as a good candidate to decompose matrix 10 into a matrix decomposition, namely a weighted sum 123 of binary matrices 124 weighted with a factor 1 25, which corresponds to a discrete weight value or an offset of the discrete weight value to the specific weight value 34, and having ones at the corresponding positions, namely at positions where weights 24 in matrix 10 have the corresponding discrete weight value. Fig. 9 depicts the case where this matrix decomposition 123 also comprises an addend concerning the most frequently assumed discrete weight value 34, for example, formed by an all-ones matrix 127 times the value 34.

Sometimes it may be useful to eliminate the padding method for empty rows and add a row! indicating the row number where the next coll belongs to. For example, for low frequent values where we can expect to encounter many empty rows. In addition, another alternative which could be advantageous in specific cases (e.g., when a specific weight is accumulated in the upper part of the matrix) is to add a wPtr array which indicates where in coil (or in rowPtr) the next weight sequence starts.

Format 3:

We could consider changing rowPtr in format 2 by not letting it start over the counter and instead pointing to the parts in coll where a new row starts. In addition, we would have to add wPtr that points to the locations in rowPtr where the next sequence of positions of a new weight starts. This format has the advantage, that we do not need information of the previous selected entries in order to perform a dot product and thus, do it in parallel.

Hence, the previous example of Fig. 9 would be modified as is shown in Fig. 10. Format 4:

Minimizing calculation redundancies per row/col. This type of formats reduce the calculations required to perform a dot product by calculating unique operations (either multiplications or summations) just once and reusing the result in subsequent rows/col. For example, same element multiplication results (thus, a, -values, where w_k denotes the k-tb alphabet value and α,- the value of the t-th element of the input vector) can be reused and don't need to be recalculated. In general, for a WS matrix we only need ∑_ki l_ki number of multiplications, where the function I_ki is the indicator function being 1 whenever the / -th weight is present in the i-th column and 0 else. Once all the w_kX[ multiplication values are calculated, it only remains to add the respective values per row.

One can think of different representations for this format. A possible format could be an array-of-arrays-of-list like format, where the first array 66 comprised by the data field 36 contains the alphabet values, the second array 140 their respective column index positions where they appear at least once, here ordered in n column scans where n is the cardinality of set 36, thereby subdividing list 140 into three subsequences 142 one for each value 36 and the list lists the corresponding values that appear per row. Or, if we represent it by a set of arrays, 2 additional pointed arrays 50 and 160 would be needed that indicate start-end parts of each sequence of the previous entities. Hence, the above example would be stored in the manner depicted in Fig. 1 1.

That is, when represented using the representation 50 of Fig. 1 1 , a recipient of representation 50 may obtain the set of discrete weight values, i.e., 36, from list 66 comprised by the first data field 30, while list 140 comprised by second data field 40 indicates, for each discrete weight value 36 within list 66 by way of a separate subsequence 142, the column indices of those columns where at least one weight 24 within matrix 10 is positioned which assumes the respective discrete weight value 36. These two arrays 66 and 140 are enough information for the recipient of representation 50 to perform the pre-computation of the multiplications: remember the column indices also address the inbound activations. Thus, a recipient pre-computes, for each discrete weight value 36, the products of this weight value and the activation corresponding to the column indices indicated in the subsequence 142 associated with the respective discrete weight value. Accordingly, the number of column indices in list 140 concurrently indicates the number of pre-calculated products. As just-mentioned, list 140 is subdivided into subsequences 142 which are concertinaed in the order with which the discrete weight values 36 are listed in list 66, and pointer list 170 indicates by pointers into list 140 as to where the subsequences 142 start and end. Additionally, a recipient of representation 50 is provided with information as to which pre-calculated product is to be added-up for each output value. As became clear from the above discussion, each row corresponds to an output of the dot product, and is formed by a summation over the component wise products formed along a corresponding row. List 150 is a list of indices pointing to rank positions in list 140, ordered in a manner so that list 150 is sub-divided into subsequences of indices, each subsequence 152 corresponding to a certain row, where the subsequences 150 ordered in row order from top to bottom. Within each subsequence, the indices might be ordered in different manners. Here, in Fig. 1 1 , indices into list 140 pointing to pre-calculated products involving the first discrete weight value 36 in list 66 precede indices referring to positions in list 140 pertaining pre- calculated products concerning a second discrete weight value in list 66 and so forth, but this could alternatively be done differently. In other words, within each subsequence 150, the indices are ordered by size. Lastly, list 160 contains a pointer for each row in order to point the subsequences 152 within list 150. Thus, the recipient of representation 50 is able to locate, for each row, the subsequence 152 belonging to this row, and add-up the pre-calculated products indexed via the indices of that subsequence 152. A possible extension could be to store combinations of partial sums of w_kxi which appear frequently and can be reused. In general, we need one list that contains the information of the columns for which an is required. Subsequently a list that contains a list of indices pointing to the w_kx_t indicating the partial sums that need to be performed just once. And lastly, a list pointing to the elements in the previous lists that indicates which elements of those need to be summed together per row (and corresponding pointer arrays).

3.1.3 Optimized for vector architectures

Here we have to design the formats such that they are optimal with respect the two following properties

^• minimal thread divergence

^• maximal coalesced memory access

Thread divergence is present when all threads within a warp do not (or cannot) execute the same instructions at the same time. Typical causes are if-then-else statements or when threads have to wait for the results of other thread.

Coalesced memory access happens when consecutive threads access consecutive memory addresses. CSR format does not provide this capability, which is the main cause of it's low efficiency when running on vector architectures.

Format 1 :

When designing formats that are optimal for vector architectures, there is a thumb rule we can apply in order to guarantee non thread divergence and memory coalescense. If we design formats whose pointer arrays have constant index difference, we can always construct a format that allows for coalesced memory access and minimal thread divergence. With pointer arrays we refer to those arrays whose indices point to locations in other arrays. For example, in the previous formats the usually called rowPtr and wPtr arrays belong to this type of arrays.

Hence, if we take those formats and fix their index difference then we will have optimized them for vector architectures. This can be done by setting the difference as the max difference that appears in the original format. This induces additional signalization of empty spaces in the corresponding arrays, which we padd with an extra symbol. Thus, in such formats, we signalize the arrays which contain actual values (either weight values or position index values) and the constant difference of the pointer arrays.

As an only exception we have the wl array in the CSWR format, which we maintain in it's original form. In addition, if the entries of a pointer array points to the entries of another pointer array (as it's in the case of the CSWR format where rowPtr points to locations in wPtr), the constants get multiplied and the corresponding empty entries need to be padded.

After fixing the length of the pointer arrays, we get the corresponding format representation by transposing the elements such, that data of parallel threads can access them in a coalesced manner.

As an example, we would transform the CSWR representation of the on going matrix example as depicted in Fig. 12. in other words, the representation 50 shown in Fig. 12 amends the representation depicted in Fig. 7 in that the lists of column indices 64 comprised by the second data field 40 and the list 92 of weight indices comprised by the first data field 30 are ordered differently: they are ordered according to a regular scan of the weight positions in matrix 10 which scan is controlled by two parameters 170, wConst, which indicates the maximum number of discrete weight indices per row of matrix 10, and parameter 172, row- Const, which indicates the maximum number of weights in a certain row which assume one of the maximum number 170 of discrete weight values in each individual row. It should be clear that it is not necessary to indicate really the maximum number of weights. It is only required to be a constant number of weights. One option is to take the maximum number of weights. But the average can also be taken. In the later cases, usually the matrix is decomposed into a structured matrix and an unstructured matrix, where the structured matrix is then optimal parallelisable and the unstructured still captures the non structured parts. This aims to still be able to apply parallelisation methods without having to restrict the matrix to be structured. Both parameters 70 and 172 are comprised by the second data field 40 and they replace the pointers comprised by lists 94 and 98 needed in the embodiments of Fig. 7 and 8. Rather, they control the number of elements in lists 92 and 64 beforehand with escape symbols 174 being inserted into lists 92 and 64 at positions where an index would be left out of these lists according to Fig. 7 and 8. In particular, according to Fig. 12, the indices in list 92, for instance, are ordered according to a raster scan order according to which matrix 10 is scanned as often as indicated by parameter 170, i.e., as often as there are different discrete weight values of set 36 within any individual row of matrix 10. Here, this parameter indicates two and accordingly, matrix 10 is scanned two times. As there are five rows in matrix 10, list 92 as 10 rank positions. The first five indices in list 92 indicate a first discrete weight value in each row of matrix 10, such as in direction from top to bottom. The second five rank positions contain indices within list 92, which indicate any second discrete weight value of set 36 occurring in the respective line, if present, wherein an escape symbol 174 is placed at the respective rank value for the corresponding row if there is no such further discrete weight value. Accordingly, list 92 is structured into subsequences 176, namely as many as indicated by parameter 170 and each being as long as there are rows in matrix 10. In list 64, the column indices 60 are ordered also in row order from top to bottom, wherein the number of scans is equal to the product of parameters 170 and 172: for each of the maximum number 170 of discrete weight values per row, a maximum occasion of the respective discrete weight value of 172 is expected, and accordingly, the rows of matrix 10 are scanned the number of parameter 172 times per the number of discrete weight values per row as indicated by parameter 170. Accordingly, the column indices 60 within list 64 are structured into subsequences 178, each as long as there are rows in matrix 10, and with parameter 170 times parameter 172 in number. The first subsequence 178 in list 64 indicates, along the row direction from top to bottom, the column indices for the first discrete weight value indicated in the first subsequence 176 of weight index list 92, the second subsequence 178 of indices of column indices indicates the column index of the second instantiation of this first discrete weight value, under the same manner, the third and fourth subsequences 178 indicate the column indices of the first and second occasion of the second discrete weight value indicated for each row by the second subsequence 176 of weight indices 90 in list 92. It is clear that the order among the indices in list 92 and 64 could also be defined differently. For example, instead of structuring these lists into subsequences 176 and 178 of a length corresponding to the number of rows in matrix 10, the structuring could be done in manner so that subsequences result which are as long as the number of discrete weight values, for instance, i.e., as long as indicated by parameter 170. The scan of the weight indices 92 could, accordingly, be such that the first run of indices 90 forms a subsequence 176 indicating the weight index for the discrete weight values within the first row, followed by the subsequences relating to the subsequent rows. Similarly, column index list 64 could be structured into subsequences 178 of column indices 60 of a length corresponding to parameter 170, each subsequence 78 collecting the indices 60 corresponding to one certain row of matrix 10 and a certain discrete weight value as indicated for that row. Even alternatively, column index list 64 could be structured into subsequences 178 of column indices 60 of a length corresponding to parameter 172, so that each subsequence would collect the indices of weight within a certain row corresponding to the n^th weight within the respective row assuming each of the maximum number 170 of discrete weight values. The next Fig. will present an example.

Format 2:

We propose another format which will induce structure in the position matrices per dalphabet value. It contains the alphabet values, an integer array of equal length indicating the max number of appearance per row, a coll indicating their corresponding col indices values and the matrix size (maybe we can neglect col size). Here we pad with an extra symbol (e.g. *) in coll the spaces with no entries.

One possible variation of this format would be to store just one max appearance value in order to reduce storage requirements and enhance structure in the distribution. But this variation is probably prompt to be inefficient when the distribution of number of appearance per row strongly differs per alphabet. Another possible extension of this format is to consider storing an additional row I array indicating the row position of the elements. This can be advantageous for cases where many row/col are empty.

Here we can consider two possible ways how to concatenate the coll values. One would be to order them in alphabet major order. This would correspond to storing the matrix as a decompostion of a sum of index matrices times their respective alphabet value, with the index matrices stored as in the ELL sparse format. Another possibility would be to concatenate in row major order.

The example matrix would be stored as depicted in Fig. 13.

Fig. 13, thus, shows a modification of the example of Fig. 8: here, no weight index list is spent. Rather, list 66 is used directly. The column index list 64 is structured in a manner so that, as it has been the case in Fig. 12, the column indices relating to a certain discrete weight value and a certain row may be accessed directly by computing their rank according to a predetermined formula. To this end, the representation 50 of Fig. 13 comprises a list 180 of parameters, one for each discrete weight value within list 66, and each indicating for the discrete weight value within list 66 having the same rank, the number of column indices 60 contained a list 64 for the respective discrete weight value. Obviously, any number in list 180 is greater than or equal to 1 . Within list 64, the column indices 60 are ordered in subsequences 182 collecting column indices of weights of a certain row having an n^th occurrence for all those discrete weight values for which the parameters in list 180 indicate that such n^th occurrence could possibly exist, i.e. , where n is smaller than or equal to the parameter given for the respective discrete weight value. Accordingly, the first subsequence 182 within list 64 contains column indices 60 relating to the first row and to the first occurrence of each discrete weight value, while the second subsequence 182 collects column indices referring to weights within the first row and indicated the second occurrence of the discrete weight value merely for those discrete weight value for which the parameters in list 180 exceed 1 . In this manner, the subsequent rows are scanned with respective subsequences 182 being contained in list 64.

An alternative approach for ordering the column indices 60 within list 64 is also shown in Fig. 13. For sake of an easier comparison, both lists are depicted in Fig. 13, one on top of the other although it is clear that representation 50 may merely comprise one of these lists 64. In the alternative, called coll', the column indices 60 are ordered in subsequences of a length corresponding to the number of rows in matrix 10, i.e., the column indices are ordered along a row-wise scan with the number of scans corresponding to the sum of parameters listed in list 180. The first subsequence collects the column indices for the rows of matrix 10 indicating the first occurrence of the first discrete weight value of list 66, the second subsequence, the column indices of weights corresponding to the second occurrence of the first discrete weight value, while then preceding with subsequences for the second discrete weight value and third discrete weight value, respectively.

Here coll was ordered in alphabet major order and transposed (such that different threads can access memory contingently), and coll' in row major order and transposed.

Format 3:

For this last format we maintain the dense matrix structure but minimize the number of multiplications required (when the alphabet size is small enough). This format follows the same idea as format 4 of the previous subsubsection. We first list all unique multiplications required for the dot product. Then, we assign this multiplications to their corresponding entry in the dense matrix format. In this way, multiplication as well as summation operations per row are highly parallelizable. This format can be considered the extreme case of format 4, since we maximize the structure in the matrix (even neglecting sparsification for this purpose). The relaxed version would be to build a matrix with the number of columns equal to the maximum frequency of appearance of unique multiplications per row. Analogously, in such case we would pad with * zero entries.

The resulting example is depicted in Fig. 14. Like the example of Fig. 1 1 , the second data field comprises a column index list 140 indicating for each discrete weight value indicated by list 66 a column index of all those columns of matrix 10 within which at least one weight is contained which is of the respective discrete weight value. Merely for sake of exemplifying possible modifications, Fig. 14 also shows the possibility that list 66 also includes the special discrete weight value 34. Consequently, list 140 of Fig. 14 is sub-divided into runs or column indices, i.e., subsequences 190, namely one per discrete weight value of the weight list 66, with the order among the subsequences corresponding to the order the discrete weight values are listed in list 66. The pointer list 170 indicates the location of the subsequences 190 within list 140. However, instead of the pointer lists and index lists 150 and 160, respectively, the second data field 40 comprises an array 192 of rank indices indicated for each positions of matrix 10 to a corresponding rank position of list 140. In other words, matrix 192 is of the size of matrix 0 to be represented but comprises, instead of the respective weight value, a rank index pointing to a corresponding column index in list 140 as already explained with respect to Fig. 1 1 , each of the column indices in list 140 corresponds, however, by way of its value indicating a certain column out of matrix 10 and by way of its position or membership of any of the subsequences 190, to one of the pre-calculated products, namely the activation corresponding to the respective column index times the discrete weight value corresponding to the subsequence 190, this column index in list 140 is a member of. Accordingly, matrix 192 is an index of pointers pointing to pre-calculated products.

We could further reduce redundancies if we consider storing partial sums symmetrically.

3.1.4 Optimized for matrix element manipulation

In this type of formats we optimize for ^• Easy element write operations

Format 1 :

We consider an array-of-structure-of-array-like (AoSoA) format which consists of an array containing the weight values and their respective row and col positions, stored in 2 further arrays respectively. The idea behind this format stems from decomposing the sparse matrix into a sum of sparse index matrices weighted by the respective weight value and storing each index matrix in COO format.

This idea can be extended to any other efficient dynamic sparse format (e.g. Iist-of-lists, dictionary format, etc) for storing the indices of each matrix.

Format 2:

For faster element read operations, we can also consider a AoSoA format which iterates over rows/columns first and stores in the respective entries the weight values present in that row/column, along with their respective col positions. However this format type would be equivalent to CSWR/CSWC.

3.2 Low entropy

Here we consider the type of matrices where

^• The weight values have low entropy

That is, the probability distribution P(w) is such, that the entropy H w) is low. P(w) expresses the frequency of appearance of the weight values in the network.

Format 1 :

Apart from the use of the CSWR (which is clear to not be efficient under this setting), all previous mentioned formats seem to be also effective for this type of matrices. In addition, we can consider decomposing the matrix into a sum of index matrices. In this way, we expect that the index matrices of each subsequent alphabet will most probably contain higher number of empty rows and and col. Hence, it might be advantageous to store the index matrices values in a different format that exploits this statistical property. For example, low frequent values might be stored in COO format, indicating the row position of the non zero elements. However, signalization of the change in format type would be required (either signalization per alphabet or signalizing the point in the alphabet array that indicates that all subsequent index matrices will correspond to the next format).

Format 2 (CER):

We can modify the CSWR format and take advantage of the given weight distribution by sorting the W array (the array containing the alphabet values) into high to low probable values. We do so because we assume that the first entries in W will appear more likely than the rest values. Subsequently, we define the entries of wPtr as to be the locations in coll that correspond to the ordered weight values. This implies that we can neglect the hole wl array, possibly reducing the amount of storage required to represent the matrix. Hence, the format will represent a specific matrix given the following 4 arrays:

• Contains the weight values. They are ordered from high to low frequency of appearance. That is, the first entry stores the weight value that appears most frequently and the last value the least frequent one.

• List of column indices, ordered by the appearance of a specific weights and concatenated by row.

^• Points to the locations in coll that corresponding to the ordered list of weight values W (thus, also concatenated by row). That is, the first entry corresponds to the index locations of the first weight value, the second entry to the second weight value, and so on. For each new row, the series starts again. If between two weight values there are non appearing weights, we pad their index location using the same number as the previous weight.

• List pointer that indicates at which location in wCtr a new row starts.

We call this format the compressed entropy row (CER) format. Hence, the above example matrix A would be represented in the CER format as in Fig. 15.

In accordance with Fig. 15, the representation 50 is similar to the ones discussed above with respect to Figs. 7 and 8, but there is no weight index list 92 as it has been in Fig. 7, and there is no corresponding list immediately indicating the discrete weight values per row as it has been in Fig. 8 by way of weight list 1 10, but this functionality is adopted by the pointer list which subdivides the sequence of column indices in list 64 into subsequences 68 of which each is associated with a separate pair of row and discrete weight value. To be more precise, the column indices in the case of Figs. 7 and 8 were ordered along a sequence which traversed each row of matrix 10 as often as there are discrete weight values in set 36, with inserting the column index of a weight having the respective discrete weight value at the time of being traversed. By this measure, the subsequences 68 were ordered in a manner so that subsequences concerning the first row precedes subsequences of a second row and so forth with, additionally, subsequences 68 relating to a discrete weight value 36 listed earlier preceding subsequences concerning the same row but a discrete weight value following in the order of the listing of the discrete weight values in the first data field. The pointer list 94 in Figs. 7 and 8 simply indicated where these subsequences are with list 1 10 or 92, respectively, indicating as to which discrete weight value the respective subsequences 68 related to. This pointer list is now, according to Fig. 15, extended to result into a pointer list 200 of pointers pointing into column index list 64: immediately following rank indices in list 200 cut-out a certain subsequence 68 out of column index list 64, but if this immediately following pair of rank indices results into a difference of zero, this simply denotes that a certain subsequence is missing, namely a subsequence for a certain discrete weight value. The pointer list 98 is still present and used according to the example of Fig. 15 and cut-out of list 200 those sequences 202 of pointers relating to the cut-out of subsequences of column indices relating to a certain row of matrix 10. By this measure, a recipient of representation 50 of Fig. 15 is able to deduce, for each row, as to how many non-empty subsequences are for the respective row in the column index list 64, and as to which discrete weight values these non-empty subsequences 68 relate to.

Analogously to CSWR, we can trivially modify the CER format into iterating in column major order instead of row major order. Thus, the coll array would be replaced into a rowl array, containing the row indices, and rowPtr would be changed into a colPtr. We call the corresponding format the compressed entropy column format (CEC). Format 3 (Sparse representations):

Finally, we can also consider extending all known sparse matrix formats into reducing the bits overhead of storing the weight values by entropy coding them.

3.3 Structured matrices

In this type of matrices we can identify a subset of indices where all elements of the matrix lie. Examples of such matrices are diagonal matrices, block matrices, lower and upper triangular matrices, etc. In usual sparse formats, storage savings are leveraged by defining a function φ: Ν [0, a]→ / c / which maps a subset of the natural numbers to a subset of the set of indices of the matrix. If the map φ is known by the decoder, the values can be reordered with respect to φ and consequently we spare the storage requirements of saving the positions of the elements. Nevertheless, in many cases the function φ depends on unknown parameters (e.g. first index position where structure starts, length of blocks, diagonal offset, etc.), which have to be signalized too. Naturally, φ is designed such, that it minimizes the previous mentioned optimization goals (low storage requirements, low energy consumption, optimal for VA and optimal for matrix element manipulation).

We can leverage in the same way this property if we can define a map φ_κ for each of the alphabet elements k. In this way, only the unknown parameters of each map 0_k need to be signalized instead of the actual index positions. For example, if a specific alphabet value can be found in some diagonal, only the signalization of it's offset would be required in order to uniquely identify it's position.

However, unless a known structure is present for each of the elements of the alphabet, we cannot ease the storage requirements as straight forward ly. Even if we can identify a subspace of indices where all the alphabet values lie, we wouldn't know their relation to the elements of the alphabet. Thus, we still would have to send the information of their positions. Nevertheless, we can still gain some storage savings by sparing the storage requirement of the positions of the most frequent alphabet value and subsequently reducing the bit-size overhead of the index positions, by storing the [0, a] values instead of their actual values. 3.4 Other statistical properties 3.4.1

• Given is a model of the conditional probability distribution over the alphabet values given some substructure in the matrix P{w_k \i[).

An important subset of this type of matrices are those were we can identify different set of alphabets at different regions of the matrix. An example would be a matrix, which has a different set of alphabets per row.

In some cases, the decoder knows a set of substructures φ_ι where the new set of alphabets are organized. This type of matrices can be viewed as a generalization of the 5.3 set of matrices that have more than one sets of alphabets. For such matrices, sig- nalization of the parameters of the structures, start-end array signalizing the start and end of the sequence of a structure, their corresponding alphabets and max frequent values is required. For example, in the case where a matrix has different alphabets per row, we would extend the CER or CSWR formats into storing the different alphabets in row major order. In addition, an array (and not a single value) of max frequent values (thus size of nr) and an additional pointer array (also of size nr) which points to parts in the alphabets array indicating the start of the sequence of the new alphabets. In this particular case signalization of the φ, parameters is not required, since it is implicitly known by the decoder that we scan the values in row major order.

We exemplify this with the previous example. That is, the representation of Fig. 16 is similar to that of Fig. 15, but adds an explicit indication of the discrete weight values per row by using the list of discrete weight values 1 10 as used in Fig. 8 along with a corresponding pointer list 210 which subdivides this weight list 1 10 into individual portions, namely one first data field 30 per row. A recipient of the representation 50 of Fig. 16, thus, is able to immediately access the discrete weight values present in each row by use of the pointer list 210 pointing to the respective portions 30 and list 0, and is able to access the associated subsequences 68 of column indices for these discrete weight values present in a certain row by use of the pointers in lists 98 and 200.

Here, wPtr points to the locations in W where the new set of alphabets start (per row).

3.4.2

• Uniform quantization This particular case is characterized by the property that the alphabet values w_k can be represented by an integer k E U such, that w_k = kAs + w₀ (where As, w₀ e R). That is, the weight values were uniformly quantized.

In all previous formats we can gain storage savings by replacing the alphabet values with their corresponding index value k. Note, that the values As and w₀ need to be additionally stored.

In the particular case of the CSW /CSWC format, we can spare the signalization/storing of the VV-array, since with the iv/-array (and As, w₀) we can reconstruct their actual real value.

4. Special layers

So far we considered leveraging the shared weight property for layer in a neural network whose connectivity can be represented by such a matrix. This is in general the case for fully connected layers. Nevertheless, state of the art neural network architectures include special types of layers that perform particular types of operations during inference. In the following we will discuss how we can apply similar ideas as previously mentioned to this special type of layers.

4.1 Convolutions

Here we consider the case where the filters in a convolution layer share weight values.

Since storage requirements are usually not of problem in practice, here we focus on formats that are efficient with respect to energy consumption and/or vector architectures.

For a particular input patch (that is, a subregion in the input space where the convolution operation will be performed), it seems redundant having to calculate each time the same multiplications across filters. Operational time might be reduced if we calculate unique multiplications just once and sum the results across filters respectively. Hence, a representation alike Format 4 in section 3.1 .2 for a convoiutional layer might be advantageous with respect to speed and energy requirements. Note that again, storage requirements can be reduced if we eliminate the most frequent weight value by subtracting it with the other values and obtain as such, sparse filters.

Since it is highly probable that most of the weights appear at least once at all points in the input patch, it might be advantageous from a parallelization point of view to always multiply ail pixels in the input patch with all weights as a first step. Subsequently, a list pointing towards the calculated elements per feature map is needed in order to know which of those elements have to be summed up. Naturally, we can extend this format to include partial sums that appear frequently at different feature maps.

It is important to note that the above case will probably not obtain gains if the number of alphabet elements is greater than the number of filters. Instead, the original version of the format might be more efficient (were we explicitly point to the locations where a multiplication is performed).

In the same way we can leverage redundant calculations in the transform domain. If we calculate convolutions using transformations (e.g. FFT or Fast algorithms), we can constrain the transform coefficients to take only discrete values, having a WS matrix in the transform domain. The difference is that here we only need to calculate the element-wise multiplications just once and subsequently order them into their respective feature maps (thus, we spare the summation operation).

As a extreme case, we may think of a format where we minimize the number of multiplications required on the entire image and across all filters (but at the expense of increasing storage requirement during the convolution operation). Without loss of generality, we assume the worst case scenario namely, the distribution of the WS filters are such that all pixels in the input image are multiplied at least once with all the elements of the discrete set of weights after a convolution operation is performed on the entire input image. Under this scenario, the number of unique multiplications required are si x nw where si denotes the input image size and nw the number of elements in the discrete set of weights. Hence, if our goal is to minimize the number of multiplications, then we may outer product the input image with the discrete set of weights (thus, / (g) W where / denotes the input image and W the set of discrete weights, both in vector form) and obtain a matrix with unique and minimum number of products needed for the convolution operation. Subsequently, we may sum the elements of the resulting matrix correspondingly to the element of the output feature map. For performing this type of convolution, it suffices to store the index of the weight at the corresponding filter position or, if size is not a matter of concern, to store pointers per feature element indicating which elements of the multiplication matrix we need to sum together.

5. Techniques to transform matrices into desired formats

Here we describe techniques that may help convert the matrices into the different formats, thus increasing their efficiency.

5.1 Matrix decomposition

As we already have discussed earlier, we can increase the efficiency of the matrix representation if we decompose it into a summation of matrices, each showing different statistical properties from which we can leverage on. By having a set of known formats in the decoder side (which we will refer as primitive formats), we can write each matrix as a linear combination of them. Although, signalization of the coefficients and of respective pointer arrays is required. It might be sufficient to restrict the coefficient values to be binary, since each weight value can be redefined into being the multiplication of the coefficient value times the respective weight value.

Based on the same idea, we may decompose the matrix into a multiplication of several matrices with different statistical properties. In fields like numerical analysis, decompositions like LU, QR, etc. improve optimisation goals.

Also low rank matrix decompositions may be included.

5.2 Transformations

This might be considered as a special case of multiplicative matrix decompositions, where some matrices have fixed values across different neural networks.

On the one hand, transformations might allow to exploit the different primitive statistical properties in the transform domain. For example, if in the frequency domain many coefficients share the same value (thus, they are low in entropy), then applying the DCT to the matrix will allow us to exploit this statistical property.

This particular technique might be useful for convolutional layers. 5.3 Permutations

We could also permute the matrices to reach desire statistical properties. This, again, might be seen as a special case of the multiplicative matrix decomposition technique. For example, the JAD sparse format leverages this idea by permuting the sparse matrix such, that it is optimal for use in vector architectures. Similarly, we could permute rows/col such, that we enhance the effectiveness of different primitive formats. For example, we could enhance that particular weights appear with high probability along a primitive structure (e.g. in series per row/col, diagonals, blocks, etc.).

Although, this technique comes as the cost of signalization of the permutations (which are of row/col-size order). Nevertheless, since FC layers are permutation invariant (in the sense that the prediction of the NN is invariant to permutations of the nodes of the hidden layers), we may spare or minimize the information required to send the permutation by defining a particular type of matrix for the decoder. For example, we can always permute rows and col in decreasing/increasing density order. We can fix this property at the decoder side and as such, reduce storage requirements.

6. Towards a general encoding-decoding scheme for neural networks

When running neural networks (either for inference or learning), most of the resources used are for handling matrix (or tensor) operations. Therefore, in the previous sections we focused on deriving different formats for representing matrices with shared weights that are more efficient with respect to a desired optimization objective (e.g., lower storage requirements, faster inference, etc.). However, the optimized representations were constrained to the connections of a specific layer. Hence, in this section we will propose a compressed domain representation for the entire neural network. We will do so by introducing an encoding-decoding scheme, where we explain all possible scenarios of how to convert the neural network into it's compressed representation (encoder), and subsequently interpret it (decoder). We note, that this only applies to the type of neural networks whose connectivity can be expressed by a set of matrices (however, extensions to higher order tensors may be possible).

The encoding-decoding scheme is depicted in figures 17, 18 and 19. They describe the different scenarios or flow of processes that may be involved in the encoding-decoding pipeline. We will provide a detailed explanation of their meaning in the coming subsections.

6.1 Encoder

The entire encoder scheme is shown in figure 17 (and a decomposed version of it in figure 18). We can see, that it is composed into two majors steps, a lossy 300 and lossless compression step 310.

The lossy process 300 is characterized by the fact that it induces a modification in the prediction accuracy of the network (it can either improve or worsen). Hence, it takes any type of neural network as input 312 (either trained by pre-training 313 or not trained and either continuous or discrete) and outputs a discrete neural network 314 with desired trades-off between prediction accuracy and statistical properties.

By continuous we mean a neural network where the weights are modeled through continuous latent variables (either deterministic real values or a set of parameters from a posterior probability distribution) and thus, re uire high precision numerical representations. This consequently implies, that the probability of two or more latent variables sharing the same parameter values tends asymptotically to 0 during the training procedure.

In contrast, by discrete we mean a neural network whose latent variables that parametrise the weight values (either ail or just a subset of weights) are elements from a finite (and relatively small) discrete set of real values. Hence, the probability of two or more latent variables sharing the same values during training is high.

In the lossless step 310 we transform the resulted (from the lossy step) quantized neural network 314 into a format that is efficient for certain desired objectives, without affecting the accuracy of the network. We call this final representation the compressed domain representation 316 which may be in any of the above discussed ones which were denoted with reference sign 50 or derived therefrom by additionally applying entropy coding thereto.

6.1.1 Lossy process The change in the accuracy is due to the quantization step 318, which is mainly characterized by the scalar quantization of the weight values of the neural network. That is, in the process we perform a mapping of the (continuous) weight values into a finite set of real numbers. This quantization of the weights can be performed partially (only a subset of the weights are quantized, e.g., sparsity) or totally (all weights are scalar quantized) and the activation values may additionally be quantized too. Furthermore, the mapping may be chosen such, that certain statistical properties in the matrices are enhanced (such as low entropy or particular structures).

The flow maps in figures 17 and 18 show possible processes involved during the quantization step. They mainly distinguish themselves from the type of input and output they produce, but not necessarily from the method. Some methods may be applicable for different types of inputs and produce respectively different types of outputs, and some other methods may be specialized for just one type of input-output maps. We also remark that the training data is always involved during the lossy encoding process.

In the following we will enumerate all possible input-output scenarios, which correspond to the numbers shown encircled in figures 17 and 18.

1. Input: Network topology (nonspecialized).

Output: Trained neural network.

This process corresponds to the usual training practice of neural networks. Namely, it takes in a neural network which is not specialized for a given task and outputs a new neural network with the same topology but has the desired prediction accuracy.

2. Input: Trained neural network.

Output: Partially quantized neural network.

In the final stage a subset of the networks connectivity is quantized. The method may search for the optimal subset and quantizer that reaches the desired trade-off between prediction accuracy and enhancement of a desired statistical property. Hence the size of the subset may vary from case to case as well as the quantizer design, and the complement set (the non quantized weights) may change their values to compensate for the quantization process. An example method would be maximizing sparsity in the weight matrices. 3. Input: Network topology.

Output: Partially quantized neural network.

The output properties correspond to those of process 2. However, in this scenario the method has to not only learn the optimal partial quantizer, but also the correct predictions for the task.

4. Input: Partially quantized neural network.

Output: Totally quantized neural network.

All connections of the neural network are quantized. In this particular scenario, only the previously non quantized connections are taken into account. The quantizer is chosen such, that the desired trade-off between prediction accuracy and quantization strength is reached.

5. Input: Trained neural network.

Output: Totally quantized neural network.

The output properties are equal to process 4. However, in this scenario the method attempts to find the optimal quantizer for all connections simultaneously. Naturally, it is also optimized for the desired trade-off between prediction accuracy and quantization strength.

6. Input: Network topology.

Output: Totally quantized neural network.

The output properties are equal to process 5 but it also simultaneously learns the correct predictions for the task.

7. Input: Continuous activation values.

Output: Quantized activation values.

This process is depicted only in figure 17 (through the small white square symbol). This process quantizes the activation values such, that the desired trade-off between - prediction accuracy and quantization strength is reached. It may be implemented either between the previous mentioned steps (thus, either before or after applying them) or simultaneously. 8. Input: Quantized/discrete neural network or continuous (trained/not trained) neural network and/or continuous or discrete activation values.

Output: Specially quantized weight and activation values.

In figures 17 and 18 this process is shown only as a process that is applied after the network was quantized. However, it can be applied in any of the previous mentioned scenarios as we!!. It refers to the processes where special types of quantizer designs are applied. For example, if a uniform quantizer whose parameters Δ and w_Q (corresponding to the constant difference between values and the k = 0 value) are estimated, given the previous scenarios. The motivation of the use of special quantizers will become clear in the lossless step, but we mention that this relates to changing the numerical format in which the weight and activation values are going to be represented.

Some of the the methods that try to find the optimal trade-off between prediction accuracy and enhancement of the statistical properties may be

^• Regularization: Adding a regularize tion term to the usual cost function which enforces the elements of the weight matrices to take a desired probability distribution. For example, weight sharing can be enhanced by adding a term that mimics (or actually is) the entropy of the probability distribution of the frequency of appearance of the weight values. Similarly, specific structures can be enhanced by adding corresponding terms. However, this requires continued learning of the weight values. This type of methods can be applied in all the above mentioned scenarios.

^• K-means (weights domain): We can cluster the weight values using the Lloyd algorithm. Modifications that take the enhancement of statistical properties into account, or the influence of the quantization to the prediction accuracy, may be considered (for example by the weighted k-means method or by changing the optimization objective). When performing weighted K-Means, the weights may be determined using some ranking method. They may include Sensitivity: gradients and/or diagonals of hessian Fisher information content Relevance measure Variance of some posterior (e.g. Gaussian posterior)

This extra information may be calculated either during or after the training procedure. If only the K-mean algorithm is applied, then this type of methods can be applied only in scenarios 2, 4 and 5 (hence, can be applied only for trained networks). However, if the K-means method is also considered during training, then it also belongs to the 3 and 6 types of procedures.

• Quantization learning: Given a trained neural network, this type of methods learn (sub)optimal quantizers by analyzing the distortion in the prediction accuracy induced by the quantization step. Here, additional information provided by the unquantized NN can be used in order to achieve better results. Additional information may include Preactivation values Gradients

This type of methods may be applied in the 2,4,5 case scenarios.

All the above mentioned methods may also take special quantizations and quantization of activation values into account.

6.1.2 Lossless process

Once the weight values in each layer are quantized (and maybe activation values too), we may further lossless compress them. However, now we aim to apply encoding methods such that the resulting format is more efficient for a particular objective, and that we can reconstruct the matrices and activations with complete fidelity (thus, without affecting the accuracy of the network).

Formatting matrices in the previous sections 3 and 4 we showed a variety of formats for representing the content of the matrices. Their structure/design were dependent on the statistical properties that the matrices presented and the desired objective one aimed to optimize them for. In order to properly decode each format, we propose to store (in the decoder side) a library/set of known formats and assign them with a unique number. In this way, only signalization of the respective number will be required, and the decoder will know how to decode the given format. The set formats we propose are

1. Usual dense matrix format

2. All formats presented in section 3 and 4.

3. All known sparse formats. In the last one (sparse formats), we add a flag in order to specify if the non zero weight values also have to be stored. If the flag says no, only the position indices will be signalized (cases where this applies is for example when a matrix with shared weights is decomposed into a linear combination of sparse matrices with coefficients corresponding to the alphabet values. In such cases, only the sparse index position of each matrix have to be signalized, along with it's corresponding alphabet value).

In addition, we propose to format each matrix into two instances, a header and a body. The header may contain meta information such as

• the format type (single number)

• the corresponding array instances which point to the parts in the body where the respective data can be found

• the array-dependent numerical representation (and their size)

• the arrays- and total matrix-sizes

The body may contain the actual matrix information encoded in the corresponding format.

Formatting activation values

If during the lossy encoding process the quantization of the activation values was also included, we propose to convert the non linear activation functions into a lookup-table representation. That is, we store tables that directly indicate how to map ranges of preactivation values into their corresponding activation value. In this way we speed up the calculation of the activation, since we only need to calculate a lookup instead of calculating the activation of each preactivation value and then quantize it applying the learned quantizer.

On the other hand, if the activation values were not quantized during the lossy encoding process, we then signalize the type of function that is being used. Analogously to the signalization of the matrix format, here we propose to store a library of known activation functions (along with their derivatives) and assign them with a unique number.

Formatting layers where special quantizers were applied If during the lossy encoding process special types of quantizers were used (such as uniform quantizers), changing the numerical representation of the resulted discrete values may increase the efficiency of the network (such as reducing the energy or storage requirements). Hence, we then may correspondingly change the numerical representation of the weight/alphabet values as well as activation values (if applied). If necessary, we further signalize additional parameters that might be required in order to uniquely reconstruct the numbers.

For example, in the case where weight as well as activation values were uniformly quantized, only their fixed point representation may be stored along with their constant difference Διν and A a and their zero values w₀ and a₀. In this way we not only reduce the bit-size required to represent the discrete values, we also can apply more efficient multiplication and summation operations. The stored parameters may help to reconstruct their actual value when so needed.

We remark, that this change in numerical representation need to be signalized in the two previous formatting processes since it affects their representation in memory. Also, the activation functions need to be able to calculate the corresponding activation values, given the preactivations in the new numerical representation. Same goes for the table-like design, which maps ranges of preactivations from the new numerical representation into their corresponding activation values (in either same or other numerical representation).

Formatting the entire neural network

We also separate the information of the neural network into two instances, the computational graph (CG) and the latent variables (LV). The computational graph contains information regarding the operations needed to perform inference (thus, the calculations needed to manipulate the input data such that we get the predictions as output) and it may also contain a graph for learning (that is, the calculations needed for updating the weight values). The LV also contains the header information of the matrices, but their content or body will be stored in the latent variable instance in their respective formats. Hence, whenever a matrix operation is needed, the CG will point to the parts in the LV instance that need to be loaded. It also gives information about the format under which the matrix content is stored and hence, the decoder will be able to know how to correctly perform the instructed matrix operation. For example, if a sparse matrix-vector multiplication is instructed, then the CG will load weight matrix in sparse format from the LV instance and the dot product will be performed using this loaded matrix representation.

This type of format has the advantage that we can reduce redundancies in the neural network's information content. For example if several matrices share the same weights or alphabet, it would only be required to store their values just once in LV. The CG would then point to the part in LV where this weights/alphabets are stored.

In figures 17 and 18 we describe the formatting process in a flow diagram.

1. Input: High precision values of the specially quantized weights and/or activation.

Output: New numerical representation (usually lower precision) of this values + the respective quantizer parameters.

In this process we change the numerical representation of weight and activation values that were as such quantized.

2. input: Quantized weight matrices.

Output: Matrices stored in compressed format.

Here we convert the weight matrices into the desired compressed format. The choice depends on the objective that is desired to be optimized. Moreover, the new numerical representation of the values is taken into account in the conversion (when applied).

3. Input: Quantized activation values.

Output: Lookup tables.

In this step we convert the quantized activation functions into lookup tables. It also takes the conversion of new numerical representations into account.

4. Input: Compressed matrix formats + activation lookup tables + extra-info. Output: Compressed domain representation of neural network.

In the last step, we build the computational graph as well as the latent variable instance by considering the converted weight matrices and activation functions and new numerical representations. Moreover, the extra-information input may contain information regarding redundancies across layers (such as multiple layers share the same weights or alphabet values) that will be taken into account when building the compressed domain representation of the neural network.

We want to remark, that the compressed domain representation also supports non compressed representations. That is, the decoder knows how to interpret dense matrices, activation functions and high precision numerical representations. The steps in the lossless diagram may be applied only when possible or so desired.

6.2 Decoder

The decoder scheme is shown in figure 18. It has a relative simple structure. It's an apparatus that is able to perform 5 types of operations 350-358, inference, reconstruction, learning, converter and modification; given a neural network in it's compressed representation 316. Any apparatus may be useful which is able to perform merely one or a proper subset of the 5 operation types 350 to 358.

6.2.1 Reconstruction

The decoder should be able to reconstruct 350 the neural network into it's uncompressed format. That is, he should know methods of converting compressed matrices back into dense format, replace tables with their corresponding activation functions (consequently dequantizing the activation values) and reconvert the low precision numerical representations back into high precision formats.

6.2.2 Inference

The decoder should also be able to perform inference 352 in compressed domain (thus, without having to reconstruct the network). Hence, it should be able to correctly perform all matrix operations required for doing inference, with any given matrix format and numerical representation of it^'s values. This implies that the decoder has to store a corresponding set of methods that allow him to do so.

6.2.3 Learning

If possible, learning in compressed domain 354 should also be an option. That is, the decoder should be able to update the matrix distribution given information for the update rule. In addition, it should be able to (if required) propagate error information between layers, create a new matrix per layer in any format type and incorporate this new matrix in the computational graph for inference.

The information considered for the update rule may be:

• Input data values

• Partial derivative values. Here, the value of the partial derivatives between the neurons of a layer and the weight of the same layer are at least required. However, partial derivative values from upper layers may also be considered.

• The error of the networks outputs. Either the difference between the data values and the output (in supervised learning) or the partial derivative of the cost to the neural networks output values are sufficient (in general).

As for the error propagation between layers, the necessary information required for applying this method is the error measure corresponding to the neurons of the considered layer and the matrix information content of the same layer (hence, the header and body).

Finally, given the same information that is necessary for performing the update rule, a matrix might be created in any desired format.

6.2.4 Converter

The converter stores a set of methods that allows to convert 356 the matrices from one format to another without loss of information.

6.2.5 Modification

Finally, given the desired changes (maybe from an outside source), the decoder should also provide a set of methods that allow him to modify 358 the neural network by any means. That is, it should be able to replace parts of a matrix with desired values (matrix stays in the same format), replace an entire matrix by another one (here the type of format may change), replace non linear activation values (respectively, replace tables) and replace type of numerical representation. In addition, it should also allow to modify the computational graph by extending it (for example by adding new operations in a layer or by adding an entirely new layer) or contracting it (eliminating operations). As an additional property, the decoder may be modular in the sense that we can construct a decoder that is able to perform just a subset of the above described tasks (a subdecoder if we may call it so). This allows to reduce the complexity (and thus the resource requirements) of running the decoder, since only a subset of methods and functions need to be stored. For example, if the user only wants to run inference on compressed neural networks, then the installed decoder may only store the methods required to run inference, neglecting all other methods.

We note, that we can reduce resource requirements even further if we consider this in the encoder side. That is, if the encoder knows that he is communicating with a subdecoder, the bit requirement for signalization of the methods in the computational graph can be reduced, since only a subset of methods exist on the decoder side. However, we reemphasize that most of the resource requirements come from preforming matrix operations (concretely, storage as well as manipulation of the latent variables) and thus, we may neglect the bit-overhead coming from the computational graph in almost all real cases.

6.3 Further notes

We were able to derive a term which enhances statistical properties such as weight sharing or structures on the weight matrices. By adding this term into the cost function, we are able to train the neural network directly for the trade-off between prediction accuracy and enhancement of statistical properties. Moreover, we can simultaneously train a uniform quantizer for the activation values.

Hence, we can apply this methods in order to obtain a neural network with uniform quantized weights and activations, plus desired trade-off between size of the network and prediction accuracy (this would follow the scenarios 5, 6, 7, 8).

Once we obtained the discrete network, we proceed with step 9 and change the numerical representations into their fixed-point variants. In addition, we apply step 10 and 1 1 and convert the weight matrices as well as activation functions into their compressed domain formats. Thus, the activation functions are converted into lookup tables and the weight matrices into, e.g. , GER formats. Finally, we create a new computational graph which takes the compressed domain representations into account, for example specifying fixed-point multiplication methods and algorithms that perform the correct dot product. In addition, in the latent variables instance we store unique weight values and eliminate as such, further redundancies.

Now, the decoder is able to perform inference in compressed domain in a very efficient way. However we note, that by applying the above procedure we optimize the efficiency with respect to the inference procedure. We may change the statistical properties of the matrix, the quantization or numerical as well as format choice depending on the desired objective. For example, we may have chosen another format for representing the matrices that is more efficient for learning (that is, for weights update) but at the expense of losing efficiency for the inference process.

7. Introduction on decoder issues

In the following sections, we will focus on the task performed at the decoder side in order to render more clear as to how such an apparatus, i.e., a decoder as it has been denoted in the previous section 6, which is a "recipient of the representation 50" as it has been denoted above, too, may take advantage of the advantageous aspects of the format or representation at hand. The following section will show in more detail what has already been indicated above briefly, namely that the above-discussed representation have different characteristics and are adapted to different types of rendering easier the typical decoder tasks.

We recall, that the decoder was described as to be an apparatus that is able to perform different tasks given a neural network in its compressed domain representation. Such tasks may include: inference, i.e. prediction, learning, conversion, modifications and modularity (in this case we omitted the reconstruction task, since this can be viewed as a special case of conversion). in the following we will explain how the decoder may execute some of the mentioned tasks in a step by step example.

8. Inference/prediction

Suppose we have encoded the neural network shown in Figure 20. The neural network of Fig. 20 is exempiarily used in order to show as to how a neural network may look like. According to Fig. 20, the neural network is dedicated for receiving as an input 400 an image, i.e., an array of pixel signals. Here, it is exemplary assumed that the image size is 32 x 32 pixels, but this is naturally merely an example. The first hidden layer 402 of the neural network is actually a set of hidden layers, here exemplary six such hidden layers. One of these six hidden layers of the first set 402 is exemplary indicated using reference sign 402a. In the example of Fig. 20, all of these first convolutional layers 402a are of equal size, but this is not mandatory. They all have 28 x 28 neurons and the number of six convolutional layers 402a is, naturally, also merely an example. All of the neurons of one such layer 402a is connected to a certain region or pixel sub area of image 400 in a manner so that the pre-activations of the neurons of these layers correspond to a convolution of the input image 400 with a certain filter tap with the weights of the filter tap corresponding to the weights of the neurons and the shape and form of the filter tap corresponding the aforementioned pixel subar- ea. The next layer of the neural network, namely 404, is again a set of neuron layers, each of which is associated with a corresponding one of layers 402a and connected therewith so as to effectively perform a subsampling. Representatively, layer 404a which is associated with layer 402a is indicated in Fig. 20. Layers 404a of set 404 may have different size. In the example of Fig. 20, these six layers 404a are of size 14 x 14 neurons. The next layer 406 of the neural network is again a set of hidden layers and they may be best understood as feature maps. They may be connected to the previous layers 404a of set 404 in a manner maintaining the spatial correspondence between neurons of layers 406a of layer set 406 on the one hand and the neurons of layers 404a of set 404 with which same are connected, but each of layer 406a may have a connection to each of the layers 404a of set 404. In the present example, layers 406a are of equal size, but this is merely an example. They are of size 10 x 10 neuros and 16 in number. The transition from layer set 406 to the next layer set 408 again corresponds to a subsampling ending-up into one layer 408a for each of layers 406 of the previous layer set 406. The next layer 410 of neurons of the neural network has here exemplary 120 neurons each of which is fully connected to the previous layer set 408, i.e., to each neuron of each layer 408a of layer set 408. The number, namely 120, is again merely representative. A next layer 412 is also a fully connected layer of here exemplary 84 neurons and the output neurons 414 are here exemplary 10 in number when connected to the previous layer 412 by Gaussian connections. The neural network 398 of Fig. 20 is, for instance a neural network dedicated for finding in the input image 400 certain image content or objects such as, for instance, certain car types, certain animals or the like. That is, the output neurons of output layer 414 would correspond to probability values indicating as to how likely it is that a certain image object is in the input image 400. It is again emphasized, that Fig. 20 has merely been chosen for representation purposes and explanation, and that the present application is at no manner restricted to such kind of neural network. Thus, we assume that the neural network 398 of Fig. 20 has been encoded into a representation discussed above. Thus, the values of the weight matrices (convolutions + fully-connected layers) as well as the activations are quantized and stored in their respective compressed domain formats (matrices in any of the proposed formats and nonlinear functions in a look-up table-like format). In addition, we assume that the quantization was uniform, such that all values are encoded in a fixed- point representation (here we note that the neural network may have a dynamic fixed- point representation. That is, different layers (or even different connections within a layer) may belong to different set of numbers).

The computational graph for performing inference under such a compressed domain representation can be depicted in the Algorithm 1 shown in Fig. 21 .

Notice, that now the convolution operation as well as the dot product depend on the format type used to represent the weights (3rd entry (formType), and corresponds to an integer that identifies the format that has been used) and the numerical representation (4th entry (numType). Here we convey parameters specific to the numerical representation used for the weights as well as activation values, such as numerical range or precision). In addition, the nonlinear functions are represented through tables, which assign preactivation values to corresponding activation values (e.g. [-0.1 ,0.1 ] --> 0 or [0.2,0.77]→ 0.35).

By designing the right convQ and dot() functions for the different presented formats, we can considerably reduce the resource requirements for calculating this operations. To illustrate this, we will now discuss some concrete examples.

8.1 Dot product in the following we will give dot product pseudocode examples for some of the formats presented above. We focus on the formats optimized for high efficiency, but provide single examples for the case where they were optimized for vector architectures. Later, we also describe in detail the case for convolutional layers.

We note, that we can also derive inference algorithms for formats that were optimized for minimal storage. However, their design was not intended for performing efficient inference and therefore, we do not provide examples for this type of formats. Nevertheless, in the later section we will discuss some use cases where this formats play an important role.

8.1.1 CSWR

A (possible) dot product pseudocode for the CSWR format illustrated in Fig. 7 is described in algorithm 2 shown in Fig. 22. At a first glance it may seem complicated, but the idea is actually relatively easy to understand For each element in the output vector 442 (thus for each row in the matrix 10), we first sum the elements of the input vector a, 440, that correspond to a specific weight and then perform the product with that weight. We do this iteratively for all weights that are present in that row.

That is, Fig. 22 shows as to how the recipient of the representation 50 of Fig. 7 acts. Firstly, the recipient derives at 450 the information indicated in Fig. 7 from the representation 50. For example, it reads at line 2 the information on the special discrete weight value 34 from the representation 50, at line 3, list 66 concerning the predetermined discrete weight values comprised by the first data field 30, in line 4, the column index list 64 comprised by the second data field, in the fifth line, the weight index list 92 also comprised by the first data field, and in line 6 pointer list 94 and in line 7 pointer list 96. The formation of the partial sums for each row is then performed at 452. For each row, the recipient uses 454 consecutive pointers in pointer list 98 to cut-out those portions of pointer list 94 and weight index list 92 which belong to the corresponding row. The ω_ρΐΓ is set to at 456 to scan subsequence 100 of pointer list 94 and portions 30 of weight index list 92. Thus, for each ω_ρ!Γ, the corresponding next loop 458 forms a partial sum at 460 over the activation values of the input vector 440 corresponding to the column indices contained in the subsequence 68 of the column index list 64 indexed by the current pointer ω_ρ!Γ at 462 and the corresponding weight referred to by the weight index at rank a>ptr and the partial sum is formed at 464 with the partial sums being added-up at 466. Fig. 22 illustrates further the case that the special discrete weight value 34 is not known a priori to be zero, but for instance merely the most frequently assumed weight value. In that case, the preliminary formed 466 sum of products over the partial sums with a corresponding predetermined discrete weight value is then, for each row of matrix 10 or each output value of vector 442, summed-up with the sum of input values of vector 440, formed at 468, multiplied at 470 with the special discrete weight value with the summation per output value being performed at 472 within for-next-loop 474. Hence, rowPtr points to the elements in wl that belong to a particular row and iv Pin^¬ points to the respective column positions in coll .

In the computational tree depicted in Figure 23 we exemplify the computations performed for calculating the second element of the output vector, thus ouput_x = 2 ^■ 2 + 4^■ 3 + 4 · 5 = 2 · 2 + 4 · (3 + 5) = W₂ ^■ a_x + W₀ ^■ (a₂ + a₅)

Algorithm 2 in Fig. 22 calculates the dot product using the last form of the above equation.

Again, as shown in Fig. 23, starting point is, for a certain row, the pointer list 98 which helps cutting-out the corresponding pointers of pointer list 94 at 480. Two pointers are cut-out for row two, thereby indicating that two partial sums have to be performed. These two pointers point to the subsequences of column indices 68 of column index list 64 shown at 482 and 484. The column indices are used for indexing the corresponding input activations and summing-up same individually, i.e., for each subsequence 68, at 486 and 488, respectively. The pointers cut-out from pointer list 94 are also used in order to then locate the associated discrete weight value with which each partial sum is then multiplied at 490 and 492, respectively. The product sums, thus formed, are then to subject to summation at 494. The extra treatment in case of the special discrete weight value not being zero has been explained above with respect to Fig. 22.

Extensions:

A possible extension to this format is to consider different weight values per row. Thus, W = [Μ _{1 (} . . . , Μ_η] would contain all weight values, concatenated by the different weight sets. In such a case, it may be advantageous to have two additional arrays. One pointer-array that points to the parts in wl where a new set of alphabets is being referred to, and an index-array which indicates the initial positions in W where the corresponding new set is being stored. Consequently, this two new arrays would have the same dimension as the number of rows.

We can further improve efficiency in the particular cases where a high number of rows share the same set of alphabets. Here, it may be advantageous to apply a row- permutation transform where all rows are order by the same sets. In this way, the dimensions of the two additional arrays wouldn't necessarily be equal to the number of rows (they would be smaller equal). However, we would have to additionally signalize the permutations performed (therefore, this slight extension is only advantageous iff the combined dimensions of the two additional arrays are smaller than the total number of rows).

Finally, a trivial variation is to store the information column-wise instead of row-wise, and correspondingly exchanging the sums and assignments in the pseudocode. We would analogously call this type of format CSWC.

8.1.2 CER

As a reminder, the idea behind the CER format design as shown in Fig. 15 was to assume that some weight values will be present in a row with higher probability than others. In particular, the statistics of the matrix is such, that only a few number of distinct weights appear per row. Therefore, we take advantage of the prior knowledge that higher probable values will most likely be the ones that appear at each row. Hence, we organize the signaiization such, that we first signalize the information of the higher probable values. Thus, the matrix in the previous example would be represented in the following form

W_max = 0

W = [4,3,2]

coll = [ 1,5,2,5, 1,0, 1 ,3,2,0,4]

wPtr = [0,2,4,4,5,5,7,8,8,9, 10,1 1 ]

rowPtr = [0,1,4,6,9,11]

We can derive an almost identical (with respect to algorithm 2 of Fig. 22) dot product algorithm for the CER format. We only need to change one line of the previous pseudocode (see algorithm 2 of Fig. 22 ). The result is shown in Fig. 24.

The corresponding computational tree for the example is depicted in Figure 25.

Thus, here again, the starting point for a certain row computation, i.e. , for a certain output value corresponding to this row, is pointer 98 which leads to the pointers or subsequence 202 of pointers within pointer 200. They cut-out corresponding subsequences 68 of column index list 64, wherein the subsequences 68 are ordered in a manner so that consecutive ones or runs of subsequences 68 belong to one row. Such runs of subsequences are denoted in Fig. 15 using reference sign 203. The number of subsequences 68 in a run of subsequence 203 indicates the number of discrete weight values out of set 36 being present in the respective row. The pointers cut-out the subsequences out of column index list 64 so as to yield a subsequence per predetermined discrete weight value at 490, 492 and 494, respectively. Some are, however, empty, i.e., some subsequences 68 do not contain any column index, such as the one corresponding to the second discrete weight value in the example of Fig. 25. For each predetermined discrete weight value and each subsequence, respectively, the partial sum is formed at 496, 498 and 500, respectively, wherein the partial sum formation 498 for empty subsequences 68 is, naturally, trivial as there is no add-in. The corresponding predetermined discrete weight value is assigned according to the order these predetermined discrete weight values are indicated in list 66, namely by multiplying each partial sum with that predetermined discrete weight value which correspond in rank in weight list 66 to the rank of the corresponding subsequence 68 of column indices which were used to address the add-ins of the partial sums, namely the corresponding input values of the input vector, i.e., the activations. The products where the partial sums per predetermined discrete weight value are indicated are located in Fig. 25 at 502, 504 and 506, respectively. The partial sum products are then summed-up at 508.

Extensions:

The same extensions that were suggested for the CSWR format can be also applied in the CER format (also the transposed type, the CEC format is trivially given).

8.1.3 Format 4: Minimizing redundancies in calculations per column/row

This format is the 4th format introduced in section 3.3.2 shown in Fig. 1 1 . The idea behind this format was to first calculate all unique operations that appear in the sum-of- products operation of a dot product and subsequently, add the corresponding terms at each element of the output vector. For instance, in the above example where we calculate the dot product as depicted in Fig. 26 we would perform the multiplication 4 · 5 twice. Hence, we find redundancies in the multiplication operations. These redundancies increase for larger matrices, where the probability of finding a weight stored in the same column several times increases.

In the following we will present an algorithm designed such, that we spare the calculation of redundant multiplications for performing a dot product. Following the previous example, the matrix M would take the form as shown in Fig. 1 1 .

Hence, arrays coll and wPtr signalize the unique multiplications that need to be calculated, and arrays rowList and rowPtr signalize where these results have to be added in their respective rows. That is, when performing a dot product we would first calculate all multiplications and subsequently add these terms into their corresponding element entries in the output vector. Thus,

1 . mul = [W₀ ^■ a₀, W₀ ^■ a_x, W₀ ^■ a₂, W₀ ^■ a₃, W₀ ^■ a_s, W₁ ^■ a_Q, W_x ^■ _α, W_t · a₄, W₂ ^■ a W₂ ^■ a₂] (noteice that the indices of a follow the indices of coll and that we iterate over the elements of W (because of wPtr))

2. output Q = m / [ l ] + mul \A] (from [0,2] c rowPtr and [ 1,4] c rowList)

3. outpu _! = muL[2] + mul[ \ + mul [8]

4. ;

Maybe the following matrix dot product representation in Fig. 27 helps in the understanding this format paradigm.

The W, coll and wPtr would store the content of the mul_M matrix in a format alike the CSR (but smarter in the sense, that we eliminate redundancies by taking advantage of the prior knowledge regarding the structure and statistics of the mul_M matrix. Namely, it only contains 1 element per row and consecutive rows entail the same weight. Hence, we only need to signalize the unique weights that appear in the matrix, the W array, and at which rows a change in weight value happens, the wPtr array). And the rowList and rowPtr arrays would store the index matrix sum_M in CSR format.

This view of the paradigm will become advantageous for the understanding of the motivation behind it's variations with respect to optimal vector architectures.

Algorithm 4 describes a possible dot product pseudocode and is shown in Fig. 28.

As shown in Fig. 28, a recipient of the representation 50 of Fig. 1 1 , would firstly derive all of the information indicated at the left hand side of Fig. 1 1 from the representation 50 at 510, i.e. , the special discrete weight value 34, the list 66 of predetermined discrete weight values,^' the columhlndex ^'list 1^'40, the p^'ointer^' list 1 70^", the row index list 150 and the row pointer list 160. The number of products to be pre-calculated is determined at 512. It corresponds to the length of the column index list 140. The pre-calculations are then performed subsequence-wise, i.e., in units of subsequences 142 of column indices in list 140. The pointer list 1 70 is used to access the subsequences 142 for each of the predetermined discrete weight values. That is, the predetermined discrete weight values are cycled through in the for-next-loop 514 and each of the column indices of the current subsequence 142 of column indices is used to address the corresponding activation, namely the one address by the same column index, and multiplied by the predetermined discrete weight value corresponding to the subsequence 142 in the for-next- loop 516. The recipient then uses the pointer list 160 to access, for each row, the corresponding subsequence 152 of rank values of list 150. In particular, the previous precalculation of the products is intermittently stored in a vector 520 having one entry for each entry of column index list 140, namely at the corresponding position. That is, the product at the n^th position of this intermittent vector 520 corresponds to the product formed by multiplying the activation indexed by the corresponding column index times the predetermined discrete weight value corresponding to the subsequence 142 which the corresponding column index is part of. Accordingly, for each row which a certain pass of the for-next-loop 518 relates to, a for-next-loop 520 cycles through all rank indices contained in the subsequence 152 indexed by the pointer list 160 for the respective row, and sums-up the corresponding pre-calculated products at 522. These sums then yield the output for each row as indicated at 524. Fig. 28 does not show explicitly as to how to deal with the special discrete weight value 34 in case of same not being zero, but in principle, the same may be done as described in Fig. 22 in lines 25-33.

Extensions:

This formats can be trivially extended such, that redundant partial sums are also taken into consideration (thus, when two or more weights are stored in the same column configuration several times across rows). This can be accomplished by adding an additional array that indicates which elements of mul have to be added and subsequently concatenating the results into the same vector (and consequently row List would also signalize this elements).

8.1.4 Format 2: Sum of weighted sparse index matrices

This format corresponds to format 2 (and 3) in section 3.3.2 presented with respect to Figs. 9 and 10. Here, we decompose the matrix into a sum of sparse index matrices, respectively multiplied by their corresponding weights. Thus, during inference, we would perform several sparse dot products, but at the same time we would minimize the number of multiplications needed for the inference operation.

There is a variety of ways how one can represent such decomposition but overall the main idea holds namely, we iterate over the weights, and signalize their corresponding positions at each iteration. This idea also applies when we perform inference. Thus, we first perform the dot product of the input vector with the position matrix 124 of a particular weight 125, then we multiply the resulting vector with the particular weight 125 and finally, sum over all outputs. See Fig. 9, for instance.

Hence, the design of the format depends only on the particular way used for representing the positions of the weights.

In the following example we stored the positions in CSR format, as illustrated in Fig. 9. Thus, for each weight, we store a rowPtr and a coll indicating the positions of that weight.

Notice, that in this case we don't need a wPtr since by knowing the dimensions of the matrix we can infer it.

The corresponding dot product would be performed in the manner depicted in Fig. 29. As can be seen, owing to the matrix decomposition 123, the result 540 of the dot product compotation, i.e. the dot product, namely the pre-activation of the neurons in questions, may be obtained by dividing the computational task onto several partial tasks, namely the task of applying the individual addends, 124 times 125 and, if present, 34 times 127, of the weighted sum 123 individually onto the inbound activations 542. The individual tasks are easy to compute and involve merely one non-trivial product per row.

Extensions:

A worth mentioning extension is to add a sparseType array (same length as W) which indicates which sparse format is being used for storing the positions of a particular weight. This can be advantageous in cases where different weights have different statistics with regards to their position distribution (e.g. a weight value may appear only in a low number of rows and therefore, a COO-like format for storing the positions of this weights may be more advantageous. Or maybe the distribution of some weights is structured, and formats like skyline or block-sparse formats may be a good choice).

In such extension we would need a wPtr array which indicates the size of each sparse matrix format. In this way, we would be able to identify when the next sequence of bits belong to the next set of positions of the subsequent weight.

8.2 Formats optimized for vector architectures

As mentioned above, the focus in this type of format is to ensure that different threads have same program lengths and that they access memory in a coalesced manner. This can be accomplished by fixing the difference in the pointer-arrays (in the previous formats usually denoted with a Ptr suffix) and transposing the elements in the index arrays (usually denoted with a / suffix). Additional padding of resulting empty entries need to be taken into account.

We illustrate this method by using the above CSWR example depicted in Fig. 7.

The max difference in wPtr and rowPtr are 2 and 2 respectively. Consequently, each row need to be assigned with 2 distinct weights and each of the weight need to be assigned with 2 columns. Hence, we would change the above representation into

W_max = 0

W = [4,3,2]

wl = [0,* | 0,2 | 1,* 10,210,1]

coll = [ 1 ,5,*,* ( 2,5,1,* | 0,1,*,* | 3,* ,2,* | 0,* ,4,*]

wConst = 2

rowConst = 2

Which corresponds to representations 50 of Fig. 12 with column index vector 64 being transposed to be subdivided into sequences 178 corresponding to gain x. If we would assign one thread per row (thus having 5 threads), then the j symbol separates the data that would be loaded by the different threads (notice, that we have exactly 5 subsets at each matrix). Discrete weight value and row, each indicating two occurrences of that value in that row. Each subset collecting the consecutive subsequences 1 78 belong to one row. We see, that the threads load the same amount of data and that they would perform the same number of operations, thus resulting in no thread divergence. How- ever, in this form they don't access memory in a coalesced manner. For that purpose, we transpose the coll and wl arrays (with respect to the | symbol) and change their representations into wl = [0,0,1,0,01 * ,2,* ,2, 1 ]

coll = [ 1,2,(),3,0 | 5,5,1 , «,* | * ,1,* ,2,4 | *,*,*,*,*] which results in the example of Fig. 12.

Now, each | separate the data that the five threads would coalescedly load (notice, that now each subset contains 5 elements).

Analogously, we can change the CER format of Fig. 15.

Format 2 presented in section 3.1 .3 with regards to Fig. 13 combines this paradigm with the above format 8.1.4 paradigm. We decompose the matrix into a weighted sum, where the sparse positions would be stored in ELL sparse format which is known to be optimal for vector architectures. In addition, we would have to send an array which indicates the max number of appearance of each weight per row (in the above denoted as

^lmax)-

Format 3 presented above with regard to Fig. 14 combines this paradigm with the above format 8.1 .3 paradigm. Since the thread divergence resulting from naively paralleling the dot product mul_M ^■ a (that is, we divide the number of rows in mul_M by the number of threads and assign this many rows to each thread) is minimal, we focused on efficiently formatting the sum_M . However we note, that we can trivially design an efficient representation of mul_M by fixing the difference in wPtr as discussed above.

If the number of indices per row is constant, then sum_M can be efficiently represented using the ELL sparse format. In the above embodiments we suggested to include the W_{m X} in the format representation because when doing so, we guarantee that the sum_M matrix has this constant index distribution condition. However, we see that by adequately padding the mul_M matrix with further 0 rows we can always guarantee that the sum_M matrix satisfies the condition. Hence, here we extend the format and propose this padding technique for doing so.

8.3 Convolutions Let's first recall how a 2D-convolution operation on a 2D-input is performed. Consider following generic, but simple example

We want to perform a convolution between the input / and the kernels (or filters) K_x and K₂ , starting from the upper left corner and finishing in the lower right. In this example we take a stride size 1 (that is, by how many elements to the right and down we move the kernels). The corresponding operations can be written in the following form:

output

(3) where I_p denotes the matrix that contains the patches of the input matrix (ordered by row), and K denotes the matrix containing the kerne! elements.

Now consider the case where K has shared weights. For example, let's imagine that K entails only two distinct elements, k₁ and k₂ , distributed in the following way

Notice, that if we now perform the matrix-vector multiplication (3), we would calculate the multiplication of the entire first and last column of l_p twice (the motivation is the same as introduced in the previous formats 8.1 .3 and 8.1 .4. The equivalence can be easily seen by transposing the dot product operation). Hence, we find redundancies in the number of multiplications needed for the convolution. These redundancies may increase considerably if we increase the number of kernels (specifically, the probability of getting high number of multiplicative redundancies is proportional to the ratio between tikernels/ w (number of kernels and number unique weights)).

8.3.1 Format 1

Ideally, we would perform the multiplications needed just once. Therefore, we propose a format whose design idea is similar to format 8.1 .3. Namely, we first signalize the unique multiplications that need to be performed on the entire image and subsequently, where these results need to be added on the corresponding elements on the output matrix. For instance, we could represent the above convolution layer as follows

W_max, nK, p, s, M, N, M Ni, nC = / , 2,0,1,2,2,3,3, 1

W = \ k₂ - / ]

coll = [0,1 ,2,3,4,5,6,7]

wPtr = [0,8]

kl = [0,2,0,1 ]

nW = [2,2]

The structure of the format is similar to the format 8.1 .3, however here we make slight modifications that take advantage of the natural structure of convolutional operations. At first, the convolution operation of a layer depends on the predefined number of kernels nK, how much we zero pad the input by p (zero padding means to increase the input dimensions by padding with zeros the contour of the input matrix and consequently apply convolutions on the expanded input), the stride size s, the height M and width N of the kernels, coll stores the indices of the elements of the input matrix which need to be multiplied by the weights contained in W . wPtr points to the positions in coll that indi^¬ cate a change of weight. Hence, with this two arrays we can calculate all unique multi^¬ plications that need to be performed for the entire convolution. We will call the resulting matrix mul.

Now, we need to know which elements in mul need to be added together and in which element of the output matrix they need to be stored. By the structured nature of the convolution operations, one can define a function getMulindx(w_indx, K_w, p, s, out_indx) which, given the positions K_w of a specific weight in the K matrix, the padding and stride numbers p and s, and the index of a specific element in the output vector out_indx, it returns the indices in mul that need to be added into that output element. Hence, we only need to store the positions K_w of the different weights. Therefore, the kl array stores this positions, concatenated by kernels and subsequently concatenated by weights. Since we know the total number of kernels riK and the total number of weights len(W), we only need to know the number of appearance of each weight per kernel nW in order to uniquely identify the positions in kl that correspond to a specific kernel and to a specific weight.

In this way, we are able to efficiently calculate convolutions by minimizing the number of multiplications and efficiently storing the information of the layer (note, that in the above discussion we excluded the trivial operations required to incorporate the influence of W_max in the final result).

The same logic applies when the input has different channels (e.g. 3 different color channels for an input image). Recall, that in such cases one performs independent convolution operations at each channel, with channel specific kernel matrices K^c, and sums the result of the convolutions across channels. However, we can easily interpret this operation as one big convolution by concatenating the different channels into one extended (in dimensions) input matrix, as well as concatenating the kernel matrices of the different channels and forming an extended kernel matrix.

Extensions:

A number of variations/extensions can be thought for this type of format.

On the one hand, if storage is sufficiently available, we could think of directly storing the indices resulting from getMulindx(w_indx, K_w, p, s, out_indx) instead of storing the kernel positions K_w of each weight. In this way, we spare the calculations needed for getting the indices while doing inference.

On the other hand, if storage requirement is of a great issue, this modification would greatly decrease in efficiency. Even the above presented method necessitates of great amount of storage if the input matrix is big (which is mostly the case in deep learning applications), since it requires to store a matrix of same the input dimension times the number of weights. However, we can reduce the storage requirements by segmenting the input matrix and applying the convolutions on the created submatrices. In the extreme segmentation^' case we would segment the input matrix into all of its patches, and thus recover the l_p matrix. We can also consider cases where different set of weights are applied at different input channels. In such a case one may consider to store each input channel independently and calculate the convolutions accordingly (and of course at the end sum the results across channels). An array signalizing start-end of the information of each independent channel would be required.

8.3.2 Format 2

In addition to the previous format, we can also efficiently represent convolutional layers by storing the kernel matrix K with the same format idea as 8.1 .4. That is, the idea is to decompose the kernel matrix K into a sum of weighted position matrices, where each weight would correspond to a weight value in W. Thus, in such a representation, we would perform the convolution operation on the entire input matrix for each weight, and subsequently sum the results (actually, we would perform the sum iteratively. That is, at each convolution we would call the result of the previous convolution, add the current computed convolution and then store back the result of the summation).

Hence, here it is only required to store the kernel matrix K, in addition to the numbers nK, p, s, M and N.

For the convolution operation we would again use the getMulindx(_w_indx, K_w, p, s, out_indx), but in a different time than in the previous format. Here, we would first get the positions of a specific weight K_w. Then, for all output elements call the function getMulindx and sum the respective input entries. Subsequently, multiply the resulted output matrix with the corresponding weight and store the result. And repeat this for each weight (of course, while summing the outputs after each iteration).

Extensions:

The same suggested extensions in 8.1 .4 and in the previous format also apply here.

8.3.3 Additional Notes on Convolution

In the following, we generalize and clarify the ideas regarding how to represent convolutional layers efficiently by taking advantage of the weight sharing property across differ- ent kernels.

in particular, so far we represented the convolution operation between

matrix-matrix dot product that took the following form output

where I_p denotes the matrix that contains the patches of the input matrix (ordered by row), and K denotes the matrix containing the kernel elements.

We then mentioned that if the kernel matrix has shared weights we can take advantage of this property by representi it as

and perform an efficient matrix-matrix dot product between l_p ^■ K, where we also take the natural structure of the I_p matrix (which can be fully predicted if the kernel size, the stride, patch and channel number is known) into account.

However, for the sake of clarity, we will now describe this ideas again by considering the equivalent matrix-vector dot product form of the convolutionai operation. We think that it is relatively easier to see the analogous to the format paradigm presented before by considering this representation.

The respective matrix-vector dot product representation of the convolution operation is output' =

One can easily verify that output' is the flattened version of output. Thus, the above dot product between matrix K' and I describes the convolution of the inbound activations, which form an array as indicated above, with the two kernels both being of equal size two as indicated above, with a corresponding output' for the two convolutions being a vector having a convolution result entered one on top of the other. In other words, the components of vector output' also shows a spatial relationship between its components which is illustrated in Fig. 30a. The following statements become immediately clear when looking at Fig. 30a: determining the set 36 of discrete weight values of matrix K' which, in fact, is the one to be compared with the above-discussed predetermined matrix or tensor 0, i.e., determining how predetermined discrete weight values assumed by at least one of the weights of matrix K\ is equivalent to, or yields the same result, as determining the set of predetermined discrete weight values by inspecting the weights of the kernels K only. And another circumstance is worth noting: as the kernels are of equal size, they share a common set of kernel positions. Each kernel position is uniquely associated with a set of one or more positions in matrix K'. Along column index direction, the arrangement of this one or more positions in matrix K' is constant irrespective of the kernel. That is, weights a common kernel position, but stemming from different kernels, are mapped onto sets of one or more positions in matrix K' which are aligned in column, i.e., they are arranged in the same columns of matrix K', but offset so as to be positioned in mutually disjoint sets of rows of matrix K' just as the components of output vector output' associated with the convolution with the different kernels are positioned one on top of the other. As a consequence of the latter issue, it is clear that, for instance, as soon as a certain predetermined discrete weight value is assumed by any weight of the various kernels at a certain kernel position, that those activations associated with columns of matrix K' with which this kernel position is associated, are to be multiplied in order to form a component-wise product involved in the row-wise sums of the dot product between K' and I. Fig. 30a indicates the just-outlined relationship between kernel positions on the one hand and corresponding weight positions in matrix K' on the other hand representatively for one of the four kernel positions. Let us first mention how we can considerably reduce the amount of information needed to store the matrix K' . Notice that the dimensions of the matrix K' e u.^^nK'nP^^{x (}~^{Nl 'Ml} where nK denotes the number of kernels, nP the number of patches 550 in the input image where we apply the convolution, i.e. the number of locations 550 where each kernel is positioned according to the convolution to form a multiplication 551 with the activations of I at that location 550, N, and M, the width and height of the input image or array 553 which the activations in I form. In this particular example, nK = 2, nP = 4, N, = 3, M, = 3, .

As one can see, the structure of K' repeats after each nP number of rows. It is easy to verify, that by knowing the values and positions of a particular kernel the respective positions on an entire nP x (N, ^■ ,)-block in K' can be fully predicted. For example

Hence, it is only required to signalize the kernel matrix K. Additional padding, different stride and channel number only changes the positions and structure of the resulting block, but the block will still be fully predictable if they are known (in the later subsection we will show a concrete example).

Now let us consider again the case where K' has shared weights. Let's take the above example involving k₁ and k₂. The corresponding expanded matrix takes then the form

As one can clearly see, the matrix-vector multiplication K'^■ I is highly redundant with respect to the number of multiplications needed to be performed. In fact, we could now apply any of the paradigms described above in order to efficiently perform the dot product. However, by naively applying those paradigms the matrix-dot product wouldn't be fully efficient since we wouldn^'t take advantage of the natural structure of the matrix K' . Instead, our goal should be to apply all described paradigms to the matrix K, i.e. the matrix having all kernels inscribed, since there exists a direct mapping between the positions and values between the matrices K and K', i.e. K→ K' . Or in other words, by knowing K we know exactly which operations we need to perform on the input in order to get the output.

Firstly, we can again spare to store the most frequent value by subtracting it on the other values (as mentioned previously). Hence,

To stress again how the corresponding mapping looks, we write down its expanded form

8.3.4 Applying the different paradigms

8.3.4.1 CWSR (CER)

As mentioned above, this paradigm takes advantage of the redundancies in multiplications that appear per row (or column). Thus, we would store the K matrix using this representation, where the corresponding coll array would scan over the rows (maybe rowl would then be a better name) of the matrix K (or we can think of also transposing the matrix K^T and store the transposed version in the presented way). However, in this case we need to apply the corresponding operations on the input image while scanning through the indices in K. Remember that we can always define a function getK' _idx{Kidx, layer Par am) which, given a particular index position Kidx in K, it returns the corresponding set of position in K' . The term layerParam considers additional layer parameters from which the function depends on such as, padding number p, stride number s, number of input channels nC, kernel matrix dimension and input dimension. Therefore, while scanning through the indices we apply the corresponding sums of the input data, on their corresponding output, and subsequently multiply all output results with the corresponding weight.

As an example we store the previous example using this format as follows

W_max, nK, p, s, M, N, M„ N,, nC = k_it 2,0, 1,2,2,3,3,1

W = [k₂ - k_l

wl = [0,0]

row! (or coll) = [0,2,0, 1 ]

wPtr = [0,2,4]

colPtr (or rowPtr) = [0,1,2 ]

Fig. 30b illustrates the construction of the representation 50 for this example. The concordance should become readily clear: the representation 50 of Fig. 30b indicates or comprises an indication 52 of, the special discrete weight value 34, i.e., k1. The input activations or number of input activations corresponded, in the above-outlined examples, to the number of columns, i.e., M, but here the number of input activations is conveyed via two parameters 555, namely Ml and Nl, which indicate the size of the array 553 formed by the input activations. Further, a pair of parameters 556, mainly M and N, indicates the size of the kernels, with a number the kernels indicated at 557. Stride length is indicated by parameter 558, i.e., s, and padding of the array 553 along the outer circumference thereof is indicated by parameter 559, i.e., p. Finally, a parameter 560 may optionally be present and indicate a number of channels, i.e., a number of input activations per position of array 553 so that, in effect, multiple such input arrays 553 are acted on concurrently. In accordance with an alternative embodiment, and this statement is true for the subsequent embodiments, too, parameters 560, 559 and 558 where at least one or more thereof, are left off.

Beyond this, representation 50 indicates by way of list 66 comprised by first data field 30 the set of predetermined discrete weight values. Also comprised by the representation 50, is the list 92 of weight indices containing one set of weight indices into a list 66 per kernel which corresponds to one set of indices per column of matrix K which is composed of all kernels involved. Remember the explanation of Fig. 7 and 8: the idea of this representation type was to indicate for each row summation to be done for the dot product computation as to which predetermined discrete weight values are to be taken into account in the respective summation, namely in order to cluster the summation into partial summations and apply the distributive law. Translated into the convolu- tional case of Fig. 30b, this translates into the indication of occurring or assumed predetermined discrete weight values on a per kernel level, as the output of the convolution is composed of components oij each of which is formed, as can be seen from Fig. 30a, a summation over component-wise products between certain input activations and the weights of one of the kernels only. Accordingly, the representation of Fig. 30b contains one first data field 30 per kernel which indication is done in Fig. 30b by way of list 66 and another list 92 being partitioned into one partition per kernel and indexing the respective subset occurring in the respective kernel out of list 66 for which there exists an alternative as discussed above with respect to Fig. 8 relative to Fig. 7.

What is then done in accordance with a representation 50 of Fig. 30b is to indicate for each first data field 30, i.e., for each kernel, as to where the predetermined discrete weight values indicated by list 92 for the respective kernel are within the respective kernel. To this end, the second data field 40 comprises a list of indices 64' structured into subsequences 68', namely subsequences of indices indicating for each pair of predetermined discrete weight value and kernel, as to which kernel positions within the respective kernel has weights equal to the respective predetermined discrete weight value. The usage of the apostrophes shall render the difference clear to Fig. 7: the indices do not refer to the predetermined matrix or tensor 10, which would be K' in our example, or, to be more precise, the matrix resulting from the decomposition of matrix K' into one add-in concerning the special discrete weight value 34 and another concerning each of the predetermined discrete weight values. Rather, the indices in list 64' discriminates the various kernel positons 552 within each kernel with this circumstance being illustrated in Fig. 30b at 561 . Additionally, the second data field 40 comprises the list 94 in order to locate the subsequences 68' of indices in list 64', in subsequences 100 or pointers pointing to subsequences 68' belonging to the same kernel but to different predetermined discrete weight values, the pointers in list 94 being ordered in the same order as the subsequences 68' in list 64' are, namely, according to some order among the kernels with, for each kernel, traversing the associated predetermined discrete weight values for this kernel as indicated by the respective portion within list 92. Finally, the second data field 40 comprises list 98 of pointers pointing within list 94 to the corresponding subsequences 100 so as to enable a recipient of representation 50 to locate the corresponding subsequence 100 of one or more pointers within list 94. That is, a recipient of representation 50 of Fig. 30b would do the following: it would traverse the kernels according to some order among the kernels and locate for the current kernel the corresponding pointer in list 98 so as to locate the corresponding subsequence 100 of pointers within list 94. On the basis of this, the recipient could locate the corresponding subsequences 68' of indices in list 64' each subsequence 68' indicating, for an associated predetermined discrete weight value indicated by a weight index in list 92 having a corresponding rank in list 92 in terms of the ordering among the subsequences 68' in list 64', the kernel positions for the current kernel where weights of the kernel assume the respective discrete weight value. Each index in list 64' and accordingly, each index within each of the subsequences 68' may then easily be translated or mapped into a set of one or more column indices coll indexing corresponding columns of matrix 10 and this, in turn, addresses, as already discussed above, those input activations which are to be component-wise multiplied with the corresponding predetermined discrete weight value which component-wise multiplication, however, may be alleviated by forming partial sums of those input activations corresponding to one discrete weight value and then multiplied with the corresponding discrete weight value.

However, in the case of convolutional layers we predict that this format won't be as efficient as the other formats since we usually have a low row dimension for K (because it corresponds to the kernel size). This is the reason why we didn't mention it above.

8.3.4.2 paradigm 8.1.3

This paradigm would suggest that we look to minimize the number of redundant multiplication by identifying the unique multiplications, perform them and subsequently sum their results correspondingly to the outputs element.

We already discussed above in section 8.1 .3, how we could efficiently store the matrix K' in such a way. However, as we already mentioned, this wouldn't maximize the efficiency when it comes to convolutional layers. Instead, here we would apply again the logic to the matrix K. Thus, according to the example depicted in we first signalize the row indices where each unique weight appear at least once in K. In the above example this would correspond to the following two arrays

- - W_max, nK, p, s, M, N, M,, N„ nC = /q , 2,0,1,2,2,3,3,1

W = [k₂ - k,}

rowl = [0,1,2] wPtr = [0,3]

With this information we know which elements of the input image need to be multiplied (and stored) by which weight element. Remember, that we can do this because we can always define a function getK' _idx(Kidx, layerParam) which returns all the K' indices associated to a particular index Kidx in K and consequently, we can know which elements of the input image appear in the multiplication.

For example, we could trivially select the column index to be 0, and get the set of indices in K' that correspond to the indices rowl. Thus,

layerParam = (nK, p, s, M, N, M,, N,, nC) = (2,0, 1 ,2,2,3,3, 1 ) rowl[0] = 0→ getK' _idx((0,0), layerParam) = [(0,0), (1,1 ), (2,3), (3,4)1 rowI[l] = 1 → getK' _ldx((l,0), layerParam) = [(0,1), ( 1 ,2), (2,4), (3,5)] rowl [2] = 2→ getK' _idx((2,0), layerParam) = [(0,3), (1 ,4), (2,6), (3,7) j Now we can see that the weight {k₂ - k_t) appears only in the indices [0,1,2,3,4,5,6,7] of the input vector. Or in other words, we can check which columns in K' are empty with respect to that weight (notice that the last column in (??) is empty).

Remarks: Notice that above we wrote the expanded rowl array instead of its compressed version presented here. This could be a reasonable alternative if storage is not of a big issue, since we spare the calculations needed to perform for getting the indices in K' . Moreover, since usually the kernel size is small, it is reasonable to assume that most of the time, in practice, each weight will appear at least once in all of the row indices of K (or in other words, almost no empty columns in h" for all weights). Hence, we propose to signalize the row indices where a weight does not appear. This implies changing rowl = [0,1,2] to rowl = [3] (or in it^'s expanded version, rowl = [8]).

Then, after we multiplied the input elements to the corresponding weights and stored them into a mul vector, we have to signalize which of the entries in mul have to be added up at each element of the output. Here we have again two possibilities. First, we can follow the paradigm naively and store a suml (in the above description denoted as row List. I would suggest to change it to suml since it follows more the index and pointer array logic, whereas here suml points to locations in the resulted mul array) which points to locations in mul that need to be added together, scanned with respect to the outputs element order. An additional array rowPtr signalizes where the sequence of arrays in suml corresponds to another output. Hence, in our example mul would be an 8 dimensional array with the unique multiplications as entries, and

sum! = [0,3,1 ,4,3,6,4,7,0,1,1,2,3,4,4,5] wPtr = [0,2,4,6,8,10,12,14,16]

which corresponds to the matrix-vector multi lication

mul

The resulting representation 50 is depicted in Fig. 30c. The concordance to the representation 50 of Fig. 1 1 is again highlighted by using the same reference signs as in Fig. 1 1 as far as possible and by indicating by the usage of apostrophes that the meaning of certain indices changes. Further, reference is made to the description of Fig. 30b with respect to parameters already discussed there. Thus, the representation 50 of Fig. 30c comprises the list 66 comprised by the first data field 30, here commonly, i.e., for the whole matrix K' the predetermined discrete weight values which are in there. The second data field 40 comprises the index list 140', but the indices contained therein and sub-divided into subsequences 142' do not relate to the column index coll indexing certain columns in matrix 10, but rather, they form indices discriminating or indexing certain kernel positions in order to, this time, indicate for each kernel position 552 whether at least one of the kernels has a weight at the corresponding kernel position which assumes the discrete weight value to which the subsequence 142^' belongs. The remaining part of the second data field is again similar to what has been described above with respect to Fig. 1 1 : a pointer list 170 of the second data field points to the various subsequences 142' in list 140' in the order of the predetermined discrete weight values indicated in list 66 and an index list 1 50' is also present and structured into subsequences of indices 152' in order to have each subsequence of indices 152' index those pre-computed component-wise products to be summed up for the output component of the convolution output, i.e., per row of matrix K', according to the dot product prescription. To be more precise, a recipient of representation 50 of Fig. 30c is able to map 565 the list of kernel position indices indicated by each subsequence 142' onto a set or list 566 of column indices coll indexing the columns of matrix K' where a corresponding predetermined discrete weight value associated with a respective subsequence 142^' resides. The indices in list 1 50^' index into this set or list 566 and they are clustered into the subsequences 1 52' in order to do this row-wise for matrix K' and, ac- cordingly, for each output component of the convolution output separately, respectively. The pointer list 160 also comprised by the second data field 40 enables to locate the various subsequences 152' so that the recipient of the representation 50 of Fig. 30c is able to precompute, for each predetermined discrete weight value indicated by list 66, the component-wise products of this predetermined discrete weight value with the activations associated with the columns indicated in list 566 onto which the corresponding kernel position indices indicated by the associated subsequences 142' in list 142' are mapped 565 and to then access, for each row of matrix K', i.e. , for each output component of the convolutional output, the corresponding subsequence 152' in order to sum- up those precomputed component-wise products associated with the indices in the subsequence 152'. A minor note shall be made with respect to the fact that Fig. 30c concerned an easy example where merely one predetermined discrete weight value was indicated in list 66. If there are more than one, then the lists 566 of column indices determined on the basis of each subsequence 142' of kernel position indices for each predetermined discrete weight value would have to be concatenated so that the indices in list 150' and the subsequences 152', respectively, would be able to distinguish component-wise products associated with different predetermined discrete weight values.

However, as one can clearly see, the sum matrix is also highly structured, which implies that storing it as described above is inefficient. Nevertheless, it's structure can also be fully predicted if one knows the positions of the weights in the kernel matrix K plus the layerParam. Hence, the second option and the more efficient one, is to store the matrix K in either CSWR or CSR format. Notice that in the above description we stored it using the CSR format (with index difference in nW , which should be called colPtr or rowPtr). Also, we changed the name from surnl to kl since now the indices point to rows in in the kernel matrix K (row! could also be a possible name).

Hence, the convolutional layer would be represented as

W_max, ηΚ, p, s, M, N, M„ N_h nC = k_{l t} 2,0, 1 ,2,2,3,3, 1

W -- \k₂ - k_x ]

rowl = 10,1,2 ]

wPtr = [0,3]

kl - - [0,2,0/11

rowPtr = [0,2,4]

That is, compared to Fig. 30c, the latter representation 50 just-discussed and depicted in Fig. 30d has lists 150' and 60 replaced by lists which correspond to lists 64' and 94 having been discussed above with respect to Fig. 30b. The aim is the following: a recipient receiving representation 50 of Fig. 30d is able to perform the precomputations as discussed with respect to Fig. 30c on the basis of lists 66 and 140'. Each precomputed component-wise product is, however, addressable uniquely by knowing the predetermined discrete weight value associated or involved in the respective component-wise product, and by knowing the column index coll of matrix K'. As mentioned before, however, each of the indices within subsequences 68' may uniquely be mapped onto a set of one or more column indices coll. Remember, each subsequence 68' is associated with a certain row of matrix K', i.e., with a certain convolution output component, and a corresponding predetermined discrete weight value of list 66. Thus, for the indices contained in each subsequence 68' it is known as to which predetermined discrete weight value the respective indices belong to. Accordingly, the recipient 50 is able to form, for each convolution output component, i.e., for each of row K', the corresponding sum of component-wise products by collecting those precomputed products associated with a pair of the corresponding predetermined discrete weight value on the one hand and the one or more columns onto which each index in the subsequence 68' for this predetermined discrete weight value is mapped for the corresponding row of matrix K'.

8.3.4.3 paradigm 8.1.4

Finally, we will describe a format for efficiently storing convolution layers that follows paradigm 8.1 .4.

This is actually pretty straight forward. We decompose the matrix K into a linear combination of index matrices with coefficients equal to the respective weight value. We then store each individual matrix using any type of sparse format and concatenate the respective arrays together. When performing the convolution, we would perform the convolution of each weight independently and then sum the output vectors together. We would make use of the getK' _idx function by applying them on each element of the index matrix, getting as such the input indices that need to be summed together at each output element, and subsequently multiplying the results from the sum with the respective weight.

In our example, we would decompose the matrix as in (7) and store the second matrix as

W_max, nK, p, s, M, N, M,, JV„ nC = k_{l t} 2,0,1,2,2,3,3,1

W = \ k₂ - k _l ]

rowl = [0,2,0,1]

colPtr = [0,2,4] wPtr = [0,4]

We then scan over the row indices in rowl, with getK'_idx we then call their respective K' indices, sum the input elements that have same row index in A" (for each row in K') and then multiply the output with (k₂ -

A resulting representation 50 corresponding to the just-outlined possibility is depicted in Fig. 30e and the essence thereof is that the first data field 30 indicate that the predetermined discrete weight values occurring in matrix K' by way of list 66 and that simply, then, for each predetermine discrete weight value, for each kernel, those kernel positions are indexed in a corresponding subsequence 68' within index list 64' which assume the respective predetermined discrete weight value so that the recipient of the representation may perform a matrix decomposition of matrix K' into a binary matrix 570 concerning, and weighted with, the special discrete weight value 34, and one binary matrix 572 for each predetermined discrete weight value, weighted with the offset 174 of this predetermined discrete weight value to the special discrete weight value 34, which indicates by ones the positions within matrix K', where weights corresponding to the respective predetermined discrete weight value are positioned, wherein the latter binary matrices 572 are computed on the basis of the second data field 40, namely its portions for each predetermined discrete weight value and the above-discussed mapping using parameters layerParam. Owing to this decomposition, it is possible to distribute the computational task of computing the dot product emulating the convolution onto several entities, each taking care of one add-in of the decomposition 576 being formed by summation over the add-ins just-discussed and each involving the binary matrices 572 and 570, respectively. As to binary matrix 570, it should be noted that as matrix, two, indicates by ones the positions of weights within matrix K' assuming the special discrete weight value 34 and is uniquely determined by the parameters layerParam.

8.3.5 Taking different stride, padding and channel numbers into account

Different stride, padding and channel has the only effect of modifying the structure in K'. However, this structure is still fully predictable if we know their values. For example, let's consider the same example as before

but now we say that s = 3 and p = 1. This means that we pad the contour of the input image with zeroes and that the filters move three steps to the at each convolution iteration. The corresponding matrix-vector representation would then be

output

As one can see, padding has the effect of adding extra zeros in between the kernel values (here exactly 2, since in both sides of the image 1 zero has been added) and the stride number changes the shift of the values at each next row (here they are shifted by 3 entries).

The convolution on an input with different channels (by channels different input values but that belong to the input itself. For example, if we have an image as an input, it is usually fed into the network with 3 channels, corresponding to the RGB colors) are calculated such, that the convolution operation is calculated at each channel independently and with their own set of kernels. Then their outputs are added together. Hence, this operation translates into concatenating the different channel kernel matrices K_c together by row, creating one bigger matrix K. Or in the expanded version, we flatten each image of the input channels and concatenate them into one big vector. Correspondingly, we then concatenate each expanded kernel matrix K'_c by column. We can then apply the same logic as described above for efficiently storing the concatenated matrix K and correspondingly applying the getK'_idx function, which now also depends on the number of input channels c.

Let's for example consider an additional input with its own set of kernels

11 ^ 21

K'x = 31 ^ 41

12 22

32 fc' 42

Then, the corresponding concatenated convolution operation would be

output =

8.4 On the different numerical representations of the weights

Recall, that in the inference algorithm 2 the dot products as well as the convolutions depend on the numerical representation used for the weight values. Here, we restrict the discussion to the case where weight values as well as activations are represented in fixed-point format, however different weight as well as activation values may belong to a different range of numbers. In the literature this representation is called dynamic- fixed-point representation. Thus, the decoder has to be able to perform elementary operations (multiplications and summation) using dynamic-fixed-point numbers.

In a naive implementation, this would imply having to signalize to which set of numbers each weight belongs to. However, since the probability of weights that belong to different number ranges having the same value is minimal, we can decompose the weight matrix into sub matrices, each containing weight values of only the same range. Consequently, we would have to only signalize the change in set of weights and the respective fixed-point parameters.

In the extensions section of the above format we already discussed how to handle cases when different sets of weights are present. The same discussions of how to accordingly modify the formats also apply here. In addition, as explained in section 3.6.2, we would store the index values of the weights (along but separately, with the fixed-point parameters) in the W arrays instead of their actual value.

8.5 On exploiting efficiency when 0 is not the most frequent value

Throughout our previous discussions we assumed that the value 0 was the most frequent value in the matrix and suggested format representations that exploited this fact. However, it may be the case where 0 is not the most frequent value in the matrix. In such cases, we can still efficiently exploit minimal storage and computation capacities if we decompose the matrix in the way depicted in Fig. 31 a

The second matrix can now be formatted using any of the above mentioned formats. Notice that the cost of performing the dot product of the first matrix is almost negligible. We only need to perform the sum of the elements of the vector and subsequently multiply it by the most frequent weight value once, and then add the resulting coefficient to each output element.

In general it applies that we need to store the most frequent element w_max, the array W stores then the discrete weight elements w_k— w_max (consequently it cannot contain the 0 element) and that in each dot product algorithm we need to add the additional coefficient a = iv_{m x}∑i cLi (a being the input vector) at each element of the output vector.

9. Converter

Apart from inference, another important set of functions that the decoder should to be able to perform is conversion between different formats. In particular, conversion between formats that are optimized for different objectives.

9.1 Formats designed for efficient communication

If we think of the use case where neural networks are deployed into machines that are part of a distributed or central system setting, an important aspect is the ability to efficiently communicate their network configuration to either the central server or among the other distributed machines. Assuming that all the deployed machines use the same neural network topology (that is, they all share the same computational graph) and that they have compressed network representations then, in general, we would only need to send the set of distinct weights and their respective positions in the matrix and the activation tables (we assume that the information regarding the parameters of the numerical representation of each value is encoded in the matrix format, as explained in section 2.4). However, now the objective is to design representations of the matrices (for now, we neglect deriving efficient representations for the activation tables since their bit overhead is negligible comparing to the matrix overhead) that have no (or minimal) reconstruction error, rather than no prediction error.

In the above description we focused on deriving lossless compressed formats. We derived two types of formats, format 1 and format 2, which respectively follow the same paradigm as format 8.1 .4 and CSWR.

In the first one, we signalize the matrix such, that we iterate over the weight values and respectively send their positions. Again, their respective positions can be stored in any sparse format matrix (and therefore the extension with the sparseType array also applies here) but, we additionally apply the explained tricks (taking index differences, signalizing just not obvious row or weight changes, etc. ) in order to reduce the storage (or signalization) requirements. The tricks can be basically categorized as follows: we take the difference in the pointer-type and index-type arrays in order to reduce the bit size overhead and we compress the pointer-type arrays by just signalizing situations where a change in weight or row cannot be trivially identified.

The same idea applies when compressing the CSWR and CER format.

Notes on the compression of the pointer-arrays:

In format 1 , rowPtr needs to only signalize change in rows when the first position in the next row has higher coll value than the last position in the previous row, or for empty rows. For instance, if we again consider the above matrix example

then we see, that for the weight value w = 2, wPtr would need to signalize the empty (with respect to vv = 2) rows 0,2 and 4 and also the row 3 because the position of that weight has a higher co//-vaiue than the one in row 1 . wPtr need to only signalize change in weights for cases where all positions of the next weight lie in higher row values than the positions of the previous weight. Hence, the above matrix example would take the form

W_maxi N, M: 0,5,6

W [4,3,2]

coll: [ 1 ,5,2,5,3,0|0,1,4| 1,2]

rowPtr: [4) 2,0,2,0, 11, 1,0, 1 ]

We inserted the j-symbol in order to show where a change in weights happens. Here wPtr has no entries since we encounter no case where all the positions of a subsequent weight lie in higher row values than the previous one (this would be the case if, for instance, all w = 4 would lie in the first and second row, then the w = 3 in the third and fourth, and finally the w = 2 in the last row).

In contrast, in the CSWR and CER format we cannot compress the rowPtr in the same way since the coll doesn't follow the same order. However, wPtr need to only signalize locations where the positions of the consecutive weight have higher coll values than the positions of the last weight. Hence, the above example can be rewritten as

W = [4,3,2 ]

coll = [1,5,2,5,1,0,1,3,2,0,4]

wl = [0,0,2,1,0,2,0,1]

rowPtr = [0,2,2,1,2,1] where we took the difference of the indices for the wl and rowPtr arrays in order to reduce the bit overhead.

Notes on the compression of COO-arrays:

If we consider storing the above matrix using the extended form, we could represent it as follows W_max, N, M: 0,5,6

W: [4,3,2]

coll : [ 1,5,2,5,3,0|0,1,4| 1,21

rowPtr: | 4,0| 2,2,4,1 | 1 , 1,3,1J

sparseType: [csr, coo, coo] where we separated with | different formats in which the positions are represented. We note, that during implementation the sparseType array contains only integers.

This would correspond to compressing the matrix using the representation depicted in Fig. 31 b.

In the above COO representation we used the variation where, instead of signalizing the row index for each element, we signalize the row index once and the number of elements that a rows entail. For instance, the weight 3 would store it's positions using the coll - [0,1,4] and rowPtr = [2,2,4, 1] arrays.

9.2 Formats efficient for training

The training of a neural network is usually performed in two steps: a forward and a backward pass. On the one hand, during the forward pass we perform inference on a set of data samples and calculate as such the prediction error. On the other hand, in the backward pass we backpropagate the output error to the weights of the network and subsequently update the weights accordingly.

Since the forward and backward propagation usually involves performing various matrix-vector multiplications, they can be efficiently performed using the above matrix representations (once quantized). However, training also involves constructing an error matrix of the same dimensions as each weight matrix, entailing the error value of each weight. This new matrix may also be quantized and hence, the decoder should be able to construct such error matrix directly into its compressed domain representation. For such a task we would propose the format paradigm 8.1 .4, where for each discrete error value we store their positions using the COO-sparse format. This representation of the error matrix allows for efficient new element insertion operations.

After the error matrix has been build, the decoder should support operations that allow to add the weight and error matrices and reconvert the resulting matrix into an efficient format for inference.

9.3 Example interplay between formats

Let's consider the particular use case where one desires to deploy a trained neural network to several embedded systems, continue the training locally, and then send back (to the central server) the resulting trained networks. An example of such scenario could be, for instance, a company trained a network for the task of object detection on a big collected data set, desires to test it's prediction accuracy on a user case scenario, if the network performs poorly then train it locally on the user data, and finally get the resulting networks back in order to update the central network by (for instance) taking the average weight values of all received networks.

In this scenario, we would thus follow the following steps:

1 . Quantize the trained neural network on the central server such, that a desired

compression-accuracy trade-off is reached on the central data set.

2. Convert the resulted quantized network into a representation that is minimal in

storage requirements.

3. Send the network in its compressed representation to ail embedded systems.

4. The embedded systems convert the network from a minimal storage representation into one, that is most efficient for inference.

5. The embedded systems perform inference and/or training in the compressed domain representation.

6. Once training is finished, the embedded systems convert the resulting network into a minimal storage representation.

7. They send their networks back to the central server using that last representation.

8. The central server reconstructs the received networks back into their expanded

forms and performs desired calculations on them.

10. Modularity and modifications

We can imagine that the complexity of the decoder increases considerably for each task it may be used for and for each format type it is able to interpret. Hence, the modularity property allows to select a subset of functions and formats one desires for it's particular use case, constructing as such a subdecoder with lower complexity. i For instance, if one desires to only perform inference on an embedded system in an efficient way, it may be reasonable to just select a few particular formats along with their inference functions and deploy this subdecoder into the embedded systems.

Another useful property may be the ability to change the network computational graph and data content in it's compressed domain representation. For instance, it may be the case that one desires to increase the number of layers of a number of deployed neural networks, or change their quantisation tables, or even change one convolutional layer for a fully connected one.

11. Summary

A neural network in it's compressed domain representation is composed by a set of matrices stored in a compressed format, a set of non linearities stored in a table-like format, and a set of instructions that define how the input data is to be processed. The decoder is an apparatus that is able to interpret this set of values and instructions and perform the tasks of inference, conversion, learning and modification on it.

We derived a set of formats for representing the weight matrices of the network that are efficient for performing inference. They can, in general, be categorized into 3 distinct paradigms:

1 . The information of the matrix in it's compressed representation is structured such, that the column (or row) positions of distinct weights are signalized sequentially, the respective weight value right after, and finally the change in row (or column). We presented the CSWR and CER format under this paradigm.

2. First non redundant operations (such as unique multiplications or partial sums) are signalized and subsequently, their respective appearance in each output row. The 8.1.3 format) belongs to this paradigm.

3. We decompose the quantized matrix into a weighted sum of sparse index

matrices and perform the matrix operations separately per weight. The format 8.1 .4 belonged to this paradigm.

For each of this paradigms we derived representations that are efficient with respect to a particular optimization objective. The above mentioned formats were specifically designed for performing inference using the minimal amount of operations. However, we suggested variations in order to adapt them to different optimization objectives. The suggestions included:

Vector architectures: Fixing the difference in the pointer-arrays and transposing the elements in the index arrays. This allowed to derive representations that were efficient for vector architectures.

Convolutions: Treating the kernel matrix as a matrix with shared weights. This allowed to derive efficient representations by following the above 2) and 3) paradigms.

Minima! storage: Reducing the bit overhead of representing the indices in the index-arrays (by taking their differences) and compressing the pointer-arrays such, that they only signalize non trivial changes. This allowed to derive lossless compressed representations of the weight matrices. We only discussed cases for paradigms 1 ) and 3), because paradigm 2) is not efficient for minimal storage purposes.

Furthermore, in section 8.4 we discussed how to tackle differences in fixed-point numerical representations of the weights and activation values.

12. Encoder functionality

Before we provide some generalizing embodiments of the present application subsequently, we try to provide the reader in the following with insights with regards to the functionally of the encoder apparatus.

We recall, that the encoder was described as to be an apparatus that is able to transform a given neural network into a desirable compressed domain representation, while taking the accuracy of the model into account.

In the following we will provide some examples of how the encoder may accomplish such task.

13 Examples of lossy compression methods

Lossy compression was characterized by the fact, that the process affects the networks prediction accuracy. Therefore, this type of methods attempt to minimize a rate- distortion like objective, where the trade-off is measured between the prediction accuracy and the quantization strength. 13.1 Ranking the weights and activations (weighted k-means)

Let's first consider the case where we have a trained neural network and we desire to quantize it's weight and activation values using uniform quantizers. Thus, weights as well as activation values will be quantized by the following function x_q = q (x) = Δ · floor (J + 0.5)

(9)

Let's further assume that we decided we want to quantize the weights of each layer with it's unique quantizer, and equivalently for the activation values of each layer. Thus, for all layers / we aim to find quantization step sizes Δ'_νν, Δ' (for the weights and activation values respectively) that minimize the trade-off between loss in prediction accuracy and reduced neural network size. Hence, we aim to find argmin d(Y , ?) + fiR (W_q')

argmjn d (Y, 9) + βϋ(α_η ^ι )

(10) where we denoted with d (Y, Ϋ) the prediction error between the true output Y (Y could either be the outputs from the trained neural network or the original labels from the data) and the output from the quantized neural network Ϋ = output(W_q, a_q) (W_q and a_q are the set of quantized weights and activations), and R {W_q ), R {a_q ^l ) the rate (or size) of the quantized weights and activations of layer /.

From now on we will omit the upper index /, but the discussions apply equivalently to all layers.

Because the above optimisitation objective (10) is hard to solve (may be NP-hard), we relax the problem and instead try to find the quantization step sizes that argmin ∑..,„ J |w - q (w) \ \ ² + βΗ (W_q) argmin ∑„ μ_η \ \ α - q(a) \ \ ² + βΗ (α_η)

(1 1 ) where H (W), H (a) denotes the entropy of the weight and activation matrices respectively. The weights μ„, μ_α reflect the influence of the particular weight or activation to the prediction error, when subjected to perturbations in their values. Thus, smaller weights indicate smaller influence and can therefore be stronger quantized.

Once we know the weight values μ^,, μ^ (1 1 ) can be efficiently solved using an entropy- constrained weighted Lloyd algorithm (well known algorithm in the literature). Hence, the question lies more on finding the weights μ. Since finding the exact /<'s may also be an NP-hard problem, we will now list some approximating candidates along with the ones that have already been proposed in the literature:

1 . Taking the diagonals of the hessian matrix Hess(d(Y, Ϋ )) with respect to W and a (proposed in the literature).

2. Taking the coefficient of the Fisher information matrix (proposed in the literature).

3. Taking the inverse of the variance of the posterior (method

derived in the literature, but not directly proposed for the above optimisation problem (3). More comments on this below).

4. Taking average LRP (layer wise relevance propagation) values

(not proposed in the literature).

5. Taking the inverse of the variance of the gradients (not proposed in the literature).

The idea of point 3 comes from a novel method for training neural networks under a bayesian framework. Each weight and activation is injected with a Gaussian noise during training, whose variance is trained along with the actual values. After training, the variances of each weight and activation can be viewed as an approximative measure of the amount of influence of that weight/activation to the prediction error (the higher the variance of a particular weight is, the higher the noise and thus, the less is it^'s influence to the outcome of the network).

Point 4 comes from the idea of taking advantage of the novel algorithm, which allows to determine which neurons and weights are most relevant for a particular outcome of the network.

Finally, point 5 was is inspired by physics and motivated by it's simplicity. Weights and neurons may be viewed as the positions of more particles that are subjected to a stochastic force field (e.g. , small particles in a thermal bath top) and thus, the gradient noise is inversely proportional to their masses. Consequently, more massive particles are more influential with respect to the total energy of the system (in other words, with respect to the prediction error of the network).

13.1.1 Example

Without loss of generality, let's consider a one layer fully connected neural network as in figure 32.

Let's further assume that the neural network was trained with respect to the following input data matrix

I 1.639 -0.328 0.046 0.527 -0.785

0.882 -0.468 - 1.093 0.094 - 1.033

-1.143 -0.274 - 0.244 -0.190 0.510

-0.303 - 1.363 -0.973 0.963 - 1.2 1 1

0.199 - 1.610 0.197 1.604 1.264 and that the resulting weight matrix is

0.2 -0.2 - 1.2 1.8 0.1

1.9 - 0.9 -0.8 -0.1 0.0

0.2 2.2 - 1.1 0.1 -0.2.

Assuming linear activation functions, the corresponding activation values after performing a forward pass are given by the following matrix 0.996 -2.258 -1.211 2.211 -2.617N

a = W - X = 3.265 0.153 1 .367 0.973 -0.85

V3.477 -0.77 - 2.25 0.458 -3.24 /

Now, we aim to find the discretized versions of W and a such that the problem 1 1 is minimized. A possible way to do this is by assuming that each element in the matrices are samples from an unknown random variable and apply the entropy constrained Lloyd algorithm on them.

To recap, the entropy constrained Lloyd algorithm is an iterative algorithm that finds (sub)optimal (with respect to problem 1 1 ) partionionings and cluster centers (in other words, the values of the discretee set), given samples from a random variable. It iterates over the following three steps:

1. Given the cluster centers and their respective codeword lengths (a measure of the bit-size of the cluster, relaxed into the continuous domain. Concretely, given the probability mass value of a cluster center, say prn_k , the corresponding codeword length is \og₂pm_k), the algorithm finds the optimal cluster assignment with respect to the cost function q w) = argmii½ j j - . This can be solved analytically without the need of calculating gradients.

2. Given the cluster assignments, the algorithm finds the optimal cluster center points with respect to w = min∑_w | |w_fc - (where w_k denotes a cluster center). This can also be solved analytically.

3. Recalculates the probability mass distribution.

The entropy constrain is being implicitly considered in the first step, by applying the codeword length constrain into the cluster assignment. Thus, this algorithm minimizes the cost function 1 1 .

Remarks: Note, that until now we only considered the MSE as a distance error measure in our discussion. However, the same logic also applies for other distance measures (hence, we can rewrite the above optimisation problems by replacing the ||... \\ term by a more general distance term d(. , . )). Also, we modify the distortion term by multiplying it by the influence coefficients of each weight element such, that the algorithm also takes (approximately) the prediction error of the neural network into account.

Another important remark is that we can distinguish between two cases under which the activation values will be quantized. The first scenario would be to directly apply the Lloyd algorithm to the activation values a. The second and more recommendable scenario, would be to first quantize the weight values, then perform the forward pass and subsequently quantize the resulting activation values, for example, let^'s assume that we applied the Lloyd algorithm onto the weight values and found the following optimal cluster assignment for each element of the matrix

The corresponding activation values would then be /0.839 - 1.089 -0.729 1.155 -1.722X

' = q(W) x l 1.639 -0.328 0.047 0.527 -0.786

V2.909 -0.663 - 1.948 0.379 -2.577/

As one can see, a'≠ a. In the case of multiple layers, one could then learn optimal cluster assignments and centers iteratively, starting from the input layer and finalizing in the last layer.

13.1.2 Gradient based optimization methods

The Lloyd algorithm is an efficient algorithm that allows to minimize the optimization objective 1 1 . However, this optimization objective may be considered too coarse. However, solving the exact optimization 10 would require to at least replace the relaxed distortion term in steps 1 and 2 with the prediction error of the network. This translates to,

1 . finding cluster assignments such that the prediction error and the average codeword length is minimized

2. finding cluster center positions such, that the prediction error of the network is minimized

3. recalculate probability mass distribution

Without adequate gradient based optimization methods this may be unfeasible. To our knowledge, step 1 may not be solvable. However, step 2 can by applying conventional neural network training methods (gradient descent with respect to each weight element and then update cluster centers taking the average of the gradients that were assigned to them). Therefore, the Lloyd algorithm may be extended into considering the prediction error of the network in step 2.

13.2 Method for finding the optimal discrete neural network

In the previous section we motivated an optimization problem for finding (sub)optimal quantizers when provided with a trained neural network (equation ( 1 1 )) and proposed the entropy-constrained weighted Lloyd algorithm for solving that problem. However, the quality of the solution of ( 1 1 ) is constrained to the trained network configuration, which may be suboptimal with respect to the actual compression objective (10). Hence, we will now propose a method that attempts to find the closest solution to (10), provided just with the networks topology. We assume again that we decide to use one quantizer per weight and activation per layer (that is, one A_w and Δ_α per layer). In addition, we also relax the problem and attempt to instead minimize the entropy-constrained neural network cost function arg min d(Y, Ϋ = output(W_q, a_qJ) + βΗ( _η, a_q

(7)

Inspired by the Lloyd algorithm, we also separate the above optimization problem into three subproblems

1 . Given cluster centers (or in other words, values of the discrete set) and their respective codeword lengths, find optimal weight distribution such, that after quantisation the prediction error and the average codeword length (or entropy) is minimized

2. given cluster assignments, find optimal cluster centers such that the prediction error is minimized

3. update probability mass distributions

13.2.1 Minimizing cluster assignments

Here the cluster assignment function is already fixed. That means we already know the decision thresholds and the cluster center positions. Hence, the objective here is to find the positions of the weight values such, that after quantising, the prediction error is minimized.

One way to optimize this is to apply an additional constrain into the Lagrangian that enforces the weight values to be close to their assigned cluster centers. Consequently, we would apply gradient descent methods to optimize the prediction error along with the regularization constrain. We would also increase iteratively the Lagrangian multiplier such, that after convergence, the weight values coincide with their respective cluster center. Note that during training, the assigned cluster center per weight element changes since the weight positions change according to the prediction error. If we fix decission thresholds that take the codeword length of the cluster centers into account, we implicitly minimize the entropy term with the above method (since cluster centers with less codeword lengths have wider decission thersholds and therefore, it is more likely that a weight element lands into it's decission boundary). However we can further enforce the minimization of the entropy explicitly by taking it into account as a further constrain in the Lagrangian. We may do this by assuming a gaussian distribution over the weight elements and minimize the cross-entropy between the given probability distribution and the actual probability distribution. Hence, we add the cross-entropy term

HiWij. qiWij)) = -~∑_k G(w_u - w_k)\og₂pm_k ¾^■ =∑_k G(w_u - w_k) for each weight element w_i7 into the Lagrangian (along with it's lagrangian multiplier), and minimize for it with respect to the elements w_Lj (note that Η(\ν^, qiw^ id differentiable ith respect to u¾).

Note, that any other normalized distribution model may be used.

13.2.2 Minimizing cluster centers

This part is equivalent to the section 13.1.2. Here, we already have assigned the cluster centers to each element and now aim to find the cluster centers that minimize the prediction error. We can do this by applying conventional training methods.

13.2.3 Considering activation functions

Here we also aim to learn optimal distributions of the activations such, that after quantization, the prediction error and their entropy is minimized. We may do this in a similar way as previously described, namely, by spearating the problem into subproblems and solving them separately (either iteratively or simultaneously).

In the cluster assignment step, we aim to train weight distributions such, that the activation values are as close as possible to the activation cluster centers the prediction error is minimized. We may also do this by adding the quantization constrain into the Lagrangian. In addition, we may also minimize the entropy (or cardinality) of a probability distribution the activations by aiming to minimize the cross-entropy between the given mass distribution and the actual mass distribution. Second, we calculate the cluster centers that minimize the prediction error by collecting their respective gradients and updating them using a gradient descent step.

And lastly we recalculate the probability mass distribution of the activations with respect to the forwarded sample batch.

So basically the optimization algorithm for the activation values is equivalent to that of the weights, with the difference that we calculate it's statistical properties with respect to a forwarded sample batch (thus, the activation values and consequently probability mass distribution change per minibatch, but the cluster centers and decission thresholds stay fix if not updated). Moreover, the gradients are still applied with respect to the weight elements. Remember that here we ask to change the weight values such, that the optimal activation values distribution will be sampled (the Lagrangian of the activation values is a differentiable function with respect to the weights).

13.2.4 General remarks

It is not yet entirely clear to us how to adequately combine the optimization of the weights and activation functions. The naive approach would be to optimize both simultaneously. However, one may be able to think of different algorithms that iterate over first weight quantization learning and then activations or vice-versa, with different advantages/disadvantages with regards to optimization efficiency and convergence. Further experiments need to be performed in order to answer this question.

In addition, in both discussions regarding the cluster assignment optimization steps (step 1 ) we considered calculating the prediction error with respect to the continuous values instead of the actual cluster values. We suggested adding the regularizer and increasing the lagrangian multiplier such that the relaxed prediction error eventually matches the prediction error of the quantized network by the end of training. However, one can train the cluster assignments by calculating explicitly the prediction of the quantized error and backward passing its error during training. In this way adding an additional regularizer wouldn't be necessary. So far in our experiments this method appeared to be more noisy and less efficient than the discussed above. However we do not discard the possibility of finding alternative and efficient algorithms that take the actual prediction error into account.

Finally, we also found out that in practice we can optimize the 3 subproblems simultaneously without considerably affecting the resulting convergence performance of the algorithm. This means, that we forward and backward pass for an input batch once, and use the same gradients for updating the weight/activation distributions as well as the cluster centers. Hence, one may consider to do for efficiency purposes.

14. Lossless compression

After we attained a discrete neural network, the encoder will proceed into converting the entire neural network into an optimal compressed domain representation.

For now, we assume that the discrete network was already trained for a particular optimization objective, and that the statistics of the weight matrices are close to the desired ones (for instance, if the purpose was to optimize for parallelisation, then the discrete weight distributions are close to optimal for that purpose).

Firstly, the encoder may change the numerical representation of the weight as well as activation values (when applied). For instance, if the weight values of a layer were uniformly quantized,

(W_q)_u = k_uA_w, k_l} 6 N then their actual values may be replaced by the indices (thus we use a fixed-point representation for the weight values). The quantization step size A_w may be stored as a property of the layer (and hence just needs to be signalized once). Analogously may the activation values be represented.

Subsequently, the encoder may lossless convert the weight matrices into one of the formats presented above (also presented above, where the encoder may select the one which is more optimal. Note, that the compressed matrix representation depends on the previous step, namely, if the weight values were previously converted into another numerical representation.

Then, knowing the quantizer for the activation values and thus it's decision thresholds, we can build tables which map preactivation values into corresponding activation values. Notice, that the information entailed in the table also depends on the first numerical conversion step. If the numbers were converted into fixed-point representations, then the table maps fixed-point numbers into fixed-point numbers.

Finally, once each component of the neural network has been adequately converted into it's compressed domain representation, the encoder may run a (maybe complex) compiler that analyzes the entire resulting network structure and adapts/optimizes it further, by for instance eliminating redundancies across layers and/or taking specific hardware configurations into consideration. For example, if one may have used same quantizers across different layers, then it^'s discrete values may be stored only once.

15. Generalizing embodiments

The above description revealed embodiments which enable to effectively use neural networks mainly effectively in the sense that the neural network handling or neural network related operations such as the performance of a prediction based on such a neural network, may be performed more efficiently in terms of, for instance, computational power, power consumption, time consumption or the like.

The apparatuses interacting with this sort of representation were already mentioned above with respect to Figs. 17-19. Depending on whether such an apparatus has been designed to output such a representation or whether the representation formed the input, same were denoted as encoder and decoder, respectively. One of the major tasks performed at decoder side pertained the task of performing a prediction based on the represented neural network. Fig. 33 shows an apparatus 600 for performing a prediction using a neural network. The apparatus 600 may be implemented in software in order to run, for instance, on a mobile device such as a mobile phone, a laptop computer or some other terminal device. However, alternatively, apparatus 600 may be implemented in firmware or hardware.

Apparatus, 600 comprises an interface 602 for receiving a representation 600 of the neural network on the basis of which apparatus 600 is to perform the prediction. The representation 604 comprises one or more matrices or tensors of weights which control an amplification at which activations of neurons of the neural network are forwarded downstream. An example of such a network has been, for example, presented in Fig. 20. The following description and the description brought forward above, concentrated on the representation with respect to one of these matrices or tensors, namely matric 10, but it is obvious that the description brought forward above may be extended to also apply to more than one of the matrices describing the neural network. At an output in- terface 606, apparatus 600 outputs the prediction result 608 such as the activations of the output neurons when you think of the example of the neural network 398 of Fig. 20.

Internally, apparatus 600 comprises a discrete weight value deriver 610a and a position deriver 610b. The former derives from the representation 604 at least the set 36 of discrete weight values actually assumed by at least one of weights of a predetermined matrix/tensor portion of the predetermined matrix of tensor 10 and the latter derives for each predetermined discrete weight value of the set of predetermined discrete weight values an indication of positions of weights assuming the respective predetermined discrete weight value within the predetermined matrix/tensor portion. The "predetermine matrix/tensor portion" may cover the whole matrix, or merely a subsection thereof, such as a subarray thereof which covers certain rows and/or columns of the predetermined matrix or tensor 10.

Further, apparatus 600 comprises the actual prediction computer 612 which actually performs the prediction, and this prediction computer comprises a dot product computer 614 which uses the set of predetermined discrete weight values and the indication of the positions as output by derivers 610a and 610b so as to compute the dot product of the predetermined matrix or tensor 10 with an array or activations of neurons of the neural network. For example, apparatus 600 may receive the inbound representation 604 from a digital storage medium, via digital communication such as wireless of cable based communication, or from a lossless data decompressor which retrieves representation 604 by, for instance, entropy or arithmetic decompression from a transmitted representation. In case of representation 604 stored on a digital storage medium, same may for instance be a programmable read-only memory, a non-transitory solid state memory, or a transitory solid state memory or any volatile memory. That is, the apparatus 600 may be provided with a representation 604 once and then, from then onwards, used the representation 604 for predictions to be done on inputs 616 which apparatus 600 may receive, for instance, at a prediction input interface 618 so as to be subject to the prediction on the basis of the neural network within prediction computer 612. Device 600 may, alternatively, intermittently receive updates of representations 604 so as to perform future predictions based on an updated representation 604. The representation 604 may be any of the above-discussed representations 50 or may be any other with deriver 610a and 610b deriving the corresponding information from the inbound representation 604- by -respective format conversion. Any format conversion, however, is preferably lossless, i.e., the derived information at the output of derivers 610a and 610b equals the corresponding portion of representation 604. Apparatus 600 may, for instance, be configured to read representation 604 from digital data which forms representations 604 so that derivers 610a and 610b simply obtain the corresponding derived information which represents any of the representations 50 discussed above, simply by digital data reading. However, the derivation may additionally comprise differential decoding and/or entropy decoding or some other similar task.

As a side note, it is recalled that computer 614 may apply the quantization of preactiva- tion values as resulting from any dot product computation onto a discrete set of activation values which may optionally be indicated by the representation 50, too. This procedure may involve a linear mapping of the preactivation value with then quantizing the mapped value using, for instance, a constant quantization step size, wherein alternatives may apply as well.

As became clear from the above discussion, the prediction computer 612 is configured to perform the prediction directly on the basis of representation 50, i.e., without deriving therefrom any other representations such as a simple component wise representation indicating each weight of matrix 10 individually in matrix arrangement, for instance. Rather, representation 50 comprises one or more data fields 30 concerning the discrete weight values and, for each first data field 30, a second data field 40 indicating the positions of weights assuming the respective discrete weight value. As already discussed above, representation 50 may additionally comprise optionally further information such as the indication 52 of a special discrete weight value such as the one most frequently assumed. Another information or data field 620 comprised by representation 50 may indicate the format underlying the representation. It is to be noted that this format indication 620 might have been maintained or conserved in case of any transitioning from inbound representation 604 to representation 50. That is, even in case of format conversions having been performed since the first generation of representation 50, the recipient of representation 50, namely the apparatus 600 of Fig. 33, is able to see as to which format is most suitable for the represented neural network, in other words, the generator or deriver of representation 50 an example for an apparatus of which is described later, might have chosen format 620 with respect to some optimization thoughts so that is advantageous if this information is maintained in representation 50. Accordingly, representation 50 may, as illustrated in Fig. 34, additionally comprise information on alternative formats 622 which might be useful in certain situations differing from ones for which format 602 at which- representation 50 is currently represented. For example, some formats discussed in the above sections aim at lower representation amount, while others are advantageous merely in case of parallel processing capability being available at the prediction computers 612. Naturally, these alternative format indications 622 may be left off by deriver 610a and 610b when forwarding representation 50 to prediction computer 612. The information items just-discussed with respect to Fig. 34, are together indicated using reference sign 624 and relate to one predetermined matrix or tensor 10 or a predetermined portion thereof and accordingly, representation 50 may comprise one or more further instantiations 626 of this form for other matrices/tensors or portions thereof. And beyond this, representation 50 may comprise one or more data fields 628 indicating a set of discrete activation values onto which activations of neurons are to be quantized during prediction, wherein the scope, i.e., the portion of the neural network, for which each of these data fields 628 apply, may be complete matrices, portions thereof or any other sections of the neural network.

The prediction computer 612 may apply data fields 628 when performing the prediction by nonlinearly mapping pre-activated neurons, i.e., pre-activations computed using the weights represented in portions 624 and 626 onto the final activations.

And even further, representation 50 may comprise coordinating data 630 which indicates the relationship between the portions and matrix/tensors described by data portions 624 and 626, respectively. In other words, the coordinating data 630 may provide a kind of survey or overview of the neural network described by representation 50.

As became clear from the above discussion, the dot product computer 614 may take advantage of representation 50 and its construction into one or more first data fields and, disjoint therefrom, i.e. additionally or separately therefrom, a second data field per first data field, 40, in one or more of the following manners. "One or more" shall denote the fact that the dot product computer may be capable of any of these three concepts discussed above in sections 8.1 in order to, for instance, apply different ones of these three possibility for different portions of a neural network of a different neural network based predictions. Alternatively, however, a dot product computer may only be able to perform one of these procedures.

In particular, the dot product 650 may, as illustrated in Fig. 35 involve sums over component-wise products between weights of the predetermined matrix or tensor 10 and activations 652 or neurons such as the sum over corresponding components of input vector 442 -and a corresponding row of- matrix 10 so as to compute the pre-activation of a certain successor neuron of the network. In order to compute the pre-activation as indicated at 654, one of the three above-discussed options may be used which are illus- trated in Fig. 35: according to the first option 656, the sums 654 is decomposed into partial sums. That is, computer 614 identifies, on the basis of representation 50, component wise products, i.e., add-ins, within sum 654, which have the same discrete weight value as a factor in 658, computes the partial sum over these groups of component wise products, multiplies each partial sum with its corresponding discrete weight value in 662 and performs a sum 664 over these weighted partial sums if more than one group of component wise products exists. The number of multiplications has, thus, been reduced. The identification in accordance with option 656, thus, operates row- specific. This is different in the case of the second option 666. Here, an identification 668 takes place, namely the identifications of weights of the predetermined matrix or tensor 10 which are of equal predetermined discrete weight value and are subject to a component wise multiplication with a same one of the activations 652 of array 442, wherein this component-wise product is computed only once in step 670 and used as an add-in in more than one of the summation 654 as indicated in 672. The identification 668 may, as illustrated in the above examples, identify all occurring component-wise products, i.e., all combinations of discrete weight value and activation value occurring in any of the component-wise products of the sums 654. They may be pre-calculated 670 and then addressed according to information conveyed by the second data field so as to be used as the individual add-ins in summation 654.

In accordance with option 676, the dot product 650 is decomposed 678 into a weighted sum over dot products of modified versions of the predetermined matrix or tensor 10 and the array 442 of activations, wherein each of the modified versions is a binary valued matrix, the ones of which are located, or which is populated, according to a representation 50, namely in a manner so that the ones are positioned at positions where matrix 10 has a corresponding discrete weight value, and each such binary valued matrix is weighted using the respective predetermined discrete weight value or a difference of the respective predetermined discrete weight value and an offset value, namely the special discrete weight value 36. As described, the computation overhead may thus be distributed onto several computation cores. Each index matrix 124 may have ones at positions corresponding to the positions of weights having of the respective discrete weight. The dot product can then be performed such that first the activation values are summed accordingly at each output row and then, the weight value is multiplied to this values. This is done at each core, separately for its addend in the weighted sum 123. Or firstly the weight value 125 is multiplied with each activation value and then the sum over ones in each row of index matrix 124 is performed and multiplied with the corresponding premultiplied activation times weigth 125. This doesn't change the representa- tion, but it changes the order of execution of product representing each discrete weight related addend in weighted sum 123. Using one or the other may have advantages/disadvantages in particular cases.

As has also been denoted above, in all options 656, 666 and 676, the dot product computer may use the predetermined discrete weight values 36 in a representation where same indicate an offset relative to the most often assumed discrete weight value 34. In that case, the pre-activation sums 654 indicated in Fig. 35 change as indicated in Fig. 36 and already explained above, namely to the extent that a further add-in occurs and the summation 656 for each output or pre-activation j, namely a sum over all inbound activations times the most often ly assumed discrete weight value 34.

This manifests itself in a further add-in used in options 656 and 666 for each row, and a further add-in and the weighted sum matrix/tensor decomposition of 678, namely a further add-in formed by all-one matrix multiplied by the most oftenly assumed discrete weight value.

In associating the steps indicated in Fig. 35 with the corresponding portions of the above description, it is noted that the identification of the partial sum add-ins in step 658 have been performed in Fig. 22 on the basis of the corresponding entries, i.e. , the ones corresponding to the current row, of rowPtr and the summation itself, 660, had been performed in the for-next-loop 460. The identification of equal component products, 668, in turn, has been performed by deriving from the representation for each weight having one of the predetermined discrete weight values, an index to the corresponding product, namely using row List. These identified to the pre-calculated component-wise products, pre-calculated only once, so that multiple instantiations of such products were used at add-ins of multiple summations 654. And the decomposition has oftenly been illustrated such as with respect to Fig. 29, and with respect to Fig. 31 b, for instance, it became clear that different formats may be used in order to convey within the second data field information on the various binary matrices describing where the individual predetermined discrete weight values lie, i.e. , where the binary matrices have their ones. Depending on the option among 656, 666 and 676 for which representation 50 has been designed, representation 50 has different characteristics.

- — Fig. 37a, for instance, shows that representation 50, if aiming at exploitation or option 656 has at least with respect to the second data field a structuring into portions each of which associated with a different one of the rows for which a summation according to which 654 is to be performed. For instance, the left hand side illustrates the case where the representation 50 comprises one first data field per row, i.e., each first data field 30 indicates the pre-determined discrete weight values 26 occurring in the respective row, wherein a second data field 40 is present for each such first data field, thereby indicating the positions of those predetermined discrete weight values indicated by the respective first data field for this row. According to the right hand option, one first data field commonly denotes the predetermined discrete weight values assumed by any weight within matrix 10, i.e., within any row, but the second data field 40 is sub-divided into portions 690, each associated with a separate row. The format according to Fig. 13, for instance, was an example for the right hand side structure, where the examples of Figs. 7 and 8 were examples for the left hand alternative.

As described in the previous examples, it be that, in case of the left hand portion, the first data fields 30 are of equal size and concatenated so as to be individually accessible by computing an address for accessing the respective first data fieid by use of a row index, for instance, at the indication of a number of assumed discrete weight values occurring in each row, with the same applying to second data fields, i.e., same may be concatenated and of equal sized so that each may be addressed by computing the address of the respective second data field 40 on the basis the row address and the number of a maximum number of discrete weight values occurring in each row, respectively. The latter circumstance would likewise apply to the portions 690 of the one second data field for the one first data field.

As discussed above wrt. Fig. 30b, here exemplarily for the right hand option of Fig 37a, both options may also be used to describe a convolutional matrix / tensor. The second data field 40 or portion 690, would, however, indicate for each row, i.e. for each kernel or summation for an convolution pre-activation output value, the columns in the indexing domain of the kernel positions which translates into column indices of the actual matrix / tensor 10, K', by mapping 565.

Representations 50 aiming at being exploited according to options 666 have a different structure as explained in Fig. 37b. Here, the first data field indicates the occurring or assumed predetermined discrete weight values within the matrix of tensor 10 globally and the second data field indicates, by column index which is, as explained above, equivalent to an address for the inbound activations, within a first portion 692 for each predetermined discrete weight value, such as within an own sub-portion 694 for each predetermined discrete weight value, those columns or activations which are to be mul- tiplied with a corresponding predetermined discrete weight value in advance, so as to result into a corresponding precomputation. That is, each of these column indices in this portion 692 results into a pre-computed product and the second data field then comprises additionally a further portion 696 which indicates as to which row summation 654 may be computed by adding-up which precomputed products. In Fig. 14, for instance, list 140 plus pointer list 170 assumed the role of portion 692, wherein some index array 192 assumed the role of portions 696.

As discussed above wrt. Fig. 30c and 30d, this type of representation when used to describe a convolutionai matrix / tensor, would have the portion 692 of the second data field 40 indicating the columns/activations to be subject to pre-computation with any of the discrete weight values in the indexing domain of the kernel positions which translates into column indices of the actual matrix / tensor 10, K', by mapping 565, and portion 696 of the second data filed would for each row, i.e. for each kernel or summation for an convolution pre-activation output value, collect the pre-computed products by indexing the pre-computed products in some default indexing domain allowing index computation of the precomputed product on the basis of an index indexing the respective predetermined discrete weight value, an index indexing the kernel, the row of K' or the convolution pre-activation output value, and an index indexing the kernel position, or simply by indicating the distribution of the predetermined discrete weight values onto the kernel positions for the kernels with leaving-up the task of indexing the precomputed products per convolution pre-activation output value to the recipient of the representation 50, such as apparatus 600.

Fig. 37c shows that a representation 50 designed for being used according to option 676, simply comprises one first data field 30 indicating the predetermined discrete samples values assumed by at least one of the weights within the matrix/tensor portion in question, while the second data field 40 is sub-divided into portions 698, one portion 698 for each predetermined discrete weight value indicated by the first data field 30 which indicates the positions of weights comprising the respective predetermined discrete weight value so that, on the basis of each portion 698, any of the binary matrices 124 may be construed and weighted according to the predetermined discrete weight values so as to obtain the matrix decomposition and compute the result thereof. As indicated above with respect to Fig. 31 b, different formats may be used in order to indicate the positions in the various portions 698. To be more precise, different portions 698 corresponding to, or relating to, different predetermined discrete weight values may indicate the positions of weight assuming any of these predetermined discrete weight values may be contained in the second data field 40 in different formats so as to allow for a taking into account of different statistical characteristics of the frequency of occurrence of the individual predetermined discrete weight values. For instance, the second data field 40 could comprise for each portion 698 an additional indication as to what weight specific format has been used for the individual portion 698.

As discussed above wrt. Fig. 30e, this type of representation when used to describe a convolutionai matrix / tensor, would have the portion 698 of the second data field 40 indicating for its predetermined discrete weight value, the kernel positions within each kernel where weights assume the corresponding predetermined weight value so that the recipient of the representation 50, such as apparatus 600, is able to construct the decomposition and the binary matrices 572, respectively, using the mapping of kernel positions 552 to position 549 in the matrix 10.

Before proceeding with the apparatus which generator representation in accordance with the present application, it should again be denoted that, preferably, the derivation done by derivers 610a and 610b merely involves a decoding, i.e., a plain text reading from digital data, or decoding procedures such as predictive decoding and/or entropy decoding. No analysis, however, is preferably done within deriver 610a and 610b which are rather tasks performed by the apparatus described now with respect to Fig. 38.

Fig. 38 shows an apparatus 700 for deriving a representation of a neural network. The apparatus 800 may be implemented in software in order to run, for instance, on a mobile device such as a mobile phone, a laptop computer or some other terminal device. However, alternatively, apparatus 700 may be implemented in firmware or hardware. The apparatus 700 receives at an input interface 702 there of information on the neural network, i.e., its weight, the neuron interconnectivity and so forth. By way of analysis, a determiner 704a then derives the one or more first data fields 30 and a position deriver 704b derives, for each of the one or more first data fields 30, the second data field 40. That is, at the output of determiner 704a and 704b, the representation 50 results in the apparatus 700 is configured to output representation 50 either as it is or in a form related to the two by lossless data conversion, at its data output 706.

Thus, the determiner 704a and 704b of the apparatus of Fig. 38 preform an evaluation or analysis of the neural network at hand rather than a simple decoding from an already present representation. The apparatus 700 of Fig. 38 may receive at its input 702 an initial representation of the neural network by lossless decoding. The reception may involve, additionally, decompression and/or predictive decoding.

At its output, apparatus 700 may output the representation 50 as a data file, for storage on a digital storage medium and/or to an apparatus for performing prediction using the neural network. The initial representation 708 might stem from an apparatus further described with respect to the following Figs. , namely one which optimizes the weights.

Preferably, the initial representation 708 which forms the basis for the determination in determiners 704a and 704b is of certain manner so that representation 50 represents the same neural network, i.e., the transition from the initial representation 708 to representation 50 is lossless.

One consequence of the fact that apparatus 700 of Fig. 38 has determiners 704a and 704b which derive representation 50 by way of analysis of the inbound initial representation is, for instance, the fact that determiner 704a gathers all predetermined discrete weight values of weights within a certain portion of the predetermined matrix/tensor for which a first data field is to be determined. That is, if merely one first data field 30 is to be generated for the aforementioned predetermined tensors/matrix portion, which may be the whole matrix/tensor 10 or a portion thereof as discussed above, then the gathering takes place for this whole predetermined matrix/tensor portion, and if a first data field 30 is to be generated for each row, then the gathering takes place for each row. The gathering may be done by determinator 704a in a manner so as to avoid duplicates. To this end, determinator 704a may check that predetermined discrete weight values of equally valued weights are gathered merely once.

A similar check is performed for the determination of the column indices when generating portion 692 of the second data field with respect to the columns. A further task which is performed by determinator 704a is to determine, for instance, the most oftenly assumed discrete weight value 34 assumed most oftenly by weights of the predetermined matrix/tensor portion for which representation 50 is to be generated. As already denoted above, determinator 704a sets-up the set of predetermined discrete weight values, for instance, in such a manner so that the set 36 includes all discrete weight -values assumed- by- at least one of the weights within- the predetermined matrix/tensor portion, excluding the most oftenly assumed discrete weight value 34. The set of predetermined discrete weight values is then indicated by the first data field within the repre- sentation 50 separately with respect to the most oftenly assumed discrete weight value indicated in the representation 50, too. As already denoted above, instead of searching for a most oftenly assumed discrete weight value, the extra discrete weight value 34 may alternatively be agreed to be zero by default.

In generating the first data field, the determinator 704a may code the predetermined discrete weight values in the first data field as offsets to the extra discrete weight value.

With respect to the further details of determinators 704a and 704b, reference is made to the above discussion as determinators 704a and 704b, generally, perform he determination, respectively, in such a manner that the above-discussed representation results which may then be used or exploited in a manner discussed above with respect to the decoder side.

Fig. 39 shows an apparatus 800 for deriving a representation of a neural network in the sense that apparatus designs or determines the neural network in such a manner that the discrete weight values are newly determined. The apparatus 800 may be implemented in software in order to run, for instance, on a computer or a server some other device. However, alternatively, apparatus 800 may be implemented in firmware or hardware. Apparatus 800 of Fig. 39 may perform its task of deriving the representation of the neural network from the scratch, i.e. by training the neural network from the scratch, on apparatus 800 uses a pre-trained neural network and quantizes its weight accordingly. That is, apparatus 800 may comprise an input for a neural network representation 802 which, as just-outlined, is either pre-trained and, accordingly, also comprises trained weights, or merely roughly describes the neural network by providing, for instance, constraints in terms of, for instance, number of layers, neurons per layer, available neuron interconnections and so forth. For instance, representation 802 at input 804 of apparatus 800 could prescribe the general structure shown in Fig. 19 in terms of the types of layers, the number of layers per layer set and so forth.

- At this point, it shall briefly by mentioned that sometimes herein an "entropy of the discrete set of weight values" has been referred to. However, it should have bcome clear from the description, that this term is to be interpreted to say "the entropy of the probability distribution of the discrete set of weight values". Here, the probability value of a - - particular element of the discrete set measures the probability of it appearing in the neural network. That is, if it's probability is high, then it is highly likely that this value appears frequently throughout the network. Consequently, a particular (discretee) weight configuration of the network can be considered as a particular sample of this probability distribution and thus, the frequency of appearance of a particular value approximates it's probability value for large networks (thus, the frequency of appearance of the discrete set of values is the empirical estimation of their probability distribution). A similar statement is true for the entroy of a set of a discrete activation values.

The apparatus 800 may quantize a pre-learned state of the weights of the predetermined matrix/tensor portion, for instance, iteratively in the manner outlined above using the three steps. Each iteration may involve determining a quantization mapping, q( ), which maps each weight of a pre-learned state, w, of the weights to one of the set of discrete weight values, e.g. the ones indexed using k and denoted w_k , so that intermediate frequencies, pm_k , result at which the discrete weight values, w, have the weights of the predetermined matrix tensor portion mapped thereonto by the quantization mapping. This determination has been described above in step 1 as comprising an optimization of the quantization mapping using a Langrangian cost function, argmin_fc | |w - w_k \ \ + \og₂pm_k , which combines the entropy of a probability distribution of the set of discrete weight values as defined by the intermediate frequencies or any other probability distribution measure and a measure measuring a central tendency of a deviation between the set of discrete weight values to the weights of the pre-learned state. Then, for each of the discrete weight values, the respective discrete weight values are updated so as to obtain a respective updated discrete weight value by optimizing a cost function, min∑_w \ \ w_k - q(w) \ \ , which measures a central tendency of a deviation between the respective updated discrete weight value and the weights of the pre-learned state mapped onto the respective discrete weight value by the quantization function. The central tendency may be L₂ norm, but any other may be used as well. It would also be possible, that the cost function measures an impact of the deviation between the set of discrete weight values to the weights of the pre-learned state onto the prediction using the neural network such as using ∑_w ^ Ι Ι ιν - q(w) \ \². In other words, the L2 norm is merely a particular example and an extension could be the usage of a weighted L2 norm where the weight measures, namely here using μ„, the importance of the weight w with respect to the prediction error. The coefficient μ,_ν, tells how much we can modify a particular element of weight w without affecting the prediction error. The coefficients μ„, may be derived/calculated/estimated by analazying the prediction error of the pre- trained NN configuration. As a general note, it should be mentioned that each step of the Lloyd^~ algorithm entails the actual -prediction^' eTror of thenetwork as distortion term. Because when doing so, we learn the quantiser q(. ) that minimises for it. However this is unfeasible in practice. Therefore we according to the embodiment outlined ehrein the (weighted) central deviation term as an approximation of the prediction error of the network and an active research topic is to find better approximations. Moreover, we also proposed to optimize the discrete values (thus the quantiser centers) under the actual prediction error of the network because this is a feasible optimisation goal. Namely, we can find local solutions using standard gradient based optimisation techniques.

The quantization mapping may be a constant quantize step size function like q(x) = Δ · floor + 0.5 However, this is merely an example and any other quantization function may be used as well. The determining the mapping may, thus, involve varying the location of the quantization function's vertical edges, i.e. varying the step widths of the floor function, to find that optimizes aforementioned Langrangian cost function, and the updating may than involve varying the quantization function's step height's, i.e. the heights of the plateaus of this function. After updating , the next iteration is performed.

When training the neural network from the scratch, apparatus 800 may also change the NN weights such, that their configuration minimises the entropy constrained Lagrangian cost function combining terms one of which measures the prediction error, the other one of which measures the entropy after being quantised. So now the apparatus is allowed to change the weight values during the encoding (optimization) procedure. Thus, the input of apparatus 800 can be any NN configuration (any pre-weight configuration, e.g. , random initialisation of a given network or the configuration of a pre-trained NN is used as a starting pint. Then the encoding is performed such, that it solves two sub- problems either iteratively or simultaneously. Firstly, it tries to find optimal weight configuration such, that after quantisation it minimises the lagrangian as described above such as by relaxing the Lagrangian in the continuous domain, by quantizing the weights at each step and calculating the gradients w.r.t. the current quantised version, etc.). Then, it optimises the discrete weight values under the prediction error. Here also any means are possible (either minimising the prediction error directly by calculating the gradients or by approximating the prediction error by, e.g. , deviation measures. Thus, apparatus 800 may do the training iteratively by performing in each iteration: iterative optimaztion of the neural network with respect to preliminary weights in the predetermined matrix/tensor portion by iteratively applaying a test input onto the neural network usingthe preliminarily weights to obtain a test prediction error and modifying the preliminary weights to reduce a Lagrangian cost function combining a functional term which measures the prediction error and an entropy term which measures an entropy of a probability distribution resulting from mapping the preliminarily weights onto the discrete weight values according to a quantization mapping. The iterative process may be done in quantized domain directly so that the prediction would be with the weights quantized onto the discrete weight values using the quantization function or in non-quanitzed domain. In the latter case, the Langrangian may comprise an additional term, namely a discrete weight value distance term, e.g. L_2 norm, measuring a central tendency of a distance of the preliminarily weights to the discrete weight values to which the preliminarily weights are mapped according to the quantization function. Then, in the current iteration, the discrete weight values are optimized: for each of the discrete weight values, the respective discrete weight value is updated so as to obtain a respective updated discrete weight value by optimizing a cost function, e.g. min∑_vv \ \w_k - c?(w) | ||, which measures a central tendency of a deviation between the respective updated discrete weight value and the preliminary weights mapped onto the respective discrete weight value by the predetermined quantization function.

At the output 806, apparatus 800 outputs a neural network representation or description which, not necessarily, equals representation 50. Rather, representational description 808 may be one which may be subject as input 708 to the apparatus 700 of Fig. 38 so as to be turned, by lossless conversion, into a representation 50. That is, representation 808 describes the neural network in the sense that at least with respect to the predetermined matrix/tensor portion, the weights of the neural network has discrete weight values with a neural network^'s weights being described by a representation which may, Iosslessly, be turned into a representation 50. In order to perform the task of generating representation 808, apparatus 800 uses an optimization scheme with an objective function which depends on a cardinality of a set of discrete weight values actually assumed by at least one of the weights of the predetermined matrix/tensor portion in a manner so that the optimization schemes aim at reducing the cardinality and/or uses an optimization scheme with an objective function which depends on frequencies at which discrete weight values of the set of discrete weight values are actually assumed by the weight of the predetermined matrix/tensor portion in a manner so that the optimization scheme aims at reducing the entropy of a probability distribution of the set of discrete weight values.

It should be noted that apparatus 800 may additionally or alternatively perform the derivation of the representation of the neural network at output 806, i.e. the generation of representation 808, in a manner so that the representation of the neural network, respectively, is optimized with respect to the activation values. To this, apparatus 800 may train the neural network from the scratch or may amend a pre-trained neural net- work in a manner so that the activations are quantized at least with respect to a certain portion of the neural network. To this end, apparatus 800 may use an optimization scheme with an objective functions with depends on a cardinality of the set of discrete activation values of activations for a predetermined set of neurons of the neural network in a manner so that the optimization scheme aims at reducing the cardinality, and/or an optimization scheme with an objective function which depends on frequencies at which discrete activation values of the set of discrete activation values are assumed for activations of neurons of a predetermined set of neurons of the neural network in a manner so that the optimization scheme aims at reducing an entropy of a probability distribution of the discrete activation values, e.g. reducing argmln ∑_a μ_α \ \α - q ( ) \ \² + βΗ(α_(Ι).

A_a

A neural network has, thus, been described above which is composed by a set of neurons that are connected in a particular arrangement. Usually, the connectivity of the neurons can be represented by a set of matrices or tensor. When compressing such a network, a common step is to quantize the matrix/tensor values (that is, each element of a matrix/tensor is allowed to only take values from a finite set of real values, instead of the entire E). The statistical property that subsets of elements may share the same value has been exploited in some of the above embodiments.

Embodiments describe a set of new formats (data structures) for representing this type of matrices. The formats leverage the weight sharing property of the matrices in order to increase their efficiency with respect to certain optimization objectives.

In addition, a general encoder-decoder scheme has been presented where we describe how neural networks may be converted into a compressed domain representation.

The statistical property that subsets of connections share the same weight valuehave been called property weight sharing property (or WS in short).

In short, we described with regard to lossy compression:

1 . Method for specifying a regularizer in the cost function that allows to enhance statistical properties in the matrices and activation values of the layers, such as Weight sharing Low entropy (with respect to frequency of appearance of the alphabet values) Structures Special quantizations (uniform, base-2, etc.). 2. Method for learning (sub)optimal fixed-point representations of the weight as well as activation values of a trained NN.

3. Methods for ranking the weights of the NN.

With regard to lossless compression

1. A set of matrix formats that leverage the statistical properties of the matrices of the NN layers.

Thus, we described an entire encoder-decoder scheme (section 6) for encoding neural networks into a representation that is efficient with respect to desired optimization objectives. Since matrix operations are the most resource hungry operations, one of the main focus in this article was to derive efficient formats for representing the content of the matrices (sections 3, 4 and 5).

We argued, that due to the quantization of the weight values of the neural network during the encoding process, the connectivity (weight) matrix will have the weight sharing property (that is, several elements in the matrix share the same weight value) with high probability. Therefore, in section 3 we explicitly focused on deriving efficient formats for this type of matrices. The optimization objectives under which we focused our format designs were:

• Minimal storage

• Minimal energy

• Vector architectures

• Matrix element manipulation

We derived a set of formats for each of the above optimization objectives, and for considered different statistical properties present in the matrix (but all had the weight sharing property in common). The considered statistical properties were:

• No probability model: We can only leverage the weight sharing property from the matrix (section 3.1 ).

• Low entropy: Here we considered matrices where the weight sharing property is highly present (section 3.2).

• Structured matrices: matrices where the elements of it are distributed according a specific structure (section 3.3).

• Other: we considered matrices with other statistical properties (section 3.4). In section 4 we proposed efficient formats for convolutional layers.

And finally in section 5 we proposed additional techniques that may help enhance the efficiency of the presented formats.

In section 6 we provided a detailed explanation of our proposed encoding-decoding scheme, which was the second focus of this description. We described many different scenarios and methods of how to convert a neural network into an efficient compressed domain representation (section 6.1 ), and subsequently described the necessary methods the decoders has to know in order to perform inference, reconstruction, learning, conversion and modification.

It is worth noting that the schemes presented herein can be considered as generally applicable to a wide field of networks, i.e. are flexible, and that efficiency can be gained if using one of the here proposed matrix formats.

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, one or more of the most important method steps may be executed by such an apparatus.

The inventive neural network representations can form a digital signal which can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperat- ing) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non- transitionary.

A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.

A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein. A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

The apparatus described herein, or any components of the apparatus described herein, may be implemented at least partially in hardware and/or in software.

The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

The methods described herein, or any components of the apparatus described herein, may be performed at least partially by hardware and/or by software.

The above described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein. References

[AEGT16] JJ Allaire, Dirk Eddelbuettel, Nick Golding, and Yuan Tang.

Tensor flow: R Interface to TensorFlow, 2016.

[BBM⁺15] Sebastian Bach, Alexander Binder, Grgoire Montavon, Frederick Klauschen, Klaus-Robert Mller, and Wojciech Samek. On pixel- wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1-46, 07 2015.

Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830, 2016.

[CBD14] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David.

Low precision arithmetic for deep learning. CoRR, abs/1412.7024, 2014.

[CBD15] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David.

Binary connect: Training deep neural networks with binary weights during propagations. CoRR, abs/151 1 .00363, 2015.

[CEL16] Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Towards the limit of network quantization. CoRR, abs/1612.01543, 2016.

[DHLC16] R. Doshi, K. W. Hung, L. Liang, and K. H. Chiu. Deep learning neural networks optimization using hardware cost penalty. In 2016 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1954-1957, May 2016.

[GAGN15] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Prit- ish Narayanan. Deep learning with limited numerical precision. CoRR, abs/1502.02551 , 2015.

[HMD15] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.00149, 2015.

Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. CoRR, abs/1506.02626, 2015.

James King, Thomas Gilray, Robert M. Kirby, and Matthew Might. Dynamic Sparse-Matrix Allocation on GPUs, pages 61-80. Springer International Publishing, Cham, 2016.

James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Has- sabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. CoRR, abs/1612.00796, 2016.

Minje Kim and Paris Smaragdis. Bitwise neural networks. CoRR, abs/1601.06071 , 2016.

Fengfu Li and Bin Liu. Ternary weight networks. CoRR, abs/1605 0471 1 , 2016.

Darryl Dexu Lin, Sachin S. Talathi, and V. Sreekanth Annapured- dy. Fixed point quantization of deep convolutional networks. CoRR, abs/151 1.06393, 2015.

C. Louizos, K. Ullrich, and M. Welling. Bayesian Compression for Deep Learning. ArXiv e-prints, May 2017.

[MKD⁺17] Naveen Mellempudi, Abhisek Kundu, Dipankar Das, Dheevatsa

Mudigere, and Bharat Kaul. Mixed low-precision deep learning inference using dynamic fixed point. CoRR, abs/1701.08978, 2017.

[PB13] Razvan Pascanu and Yoshua Bengio. Natural gradient revisit- ed. CoRR, abs/1301 .3584, 2013.

Wonyong Sung, Sungho Shin, and Kyuyeon Hwang. Resiliency of deep neural networks under quantization. CoRR, abs/15 1 .06488, 2015.

[TBCS16] M. Tu, V. Berisha, Y. Cao, and J. S. Seo. Reducing the model order of deep neural networks using information theory. In 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pages 93- 98, July 2016.

[TBW⁺16] M. Tu, V. Berisha, M. Woolf, J. s. Seo, and Y. Cao. Ranking the parameters of deep neural networks using the fisher information. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2647-2651 , March 2016.

[VSM1 1 ] Vincent Vanhoucke, Andrew Senior, and Mark Z. Mao. Improving the speed of neural networks on cpus. In Deep Learning and Unsupervised Feature Learning Workshop, NIPS 2011, 201 1 .

Claims

1. Apparatus for deriving a representation (808) of a neural network (398), the representation comprising one or more matrices or tensors of weights which control an amplification at which activations of neurons (16) of the neural network are forwarded, wherein the apparatus is configured to determine a predetermined matrix/tensor portion of a predetermined matrix or tensor (10) of weights by training the neural network or quantizing a pre-learned state of the weights of the predetermined matrix or tensor using an optimization scheme with an objective function which depends on a cardinality of a set (32) of discrete weight values actually assumed by at least one of the weights (24) of the predetermined matrix/tensor portion in a manner so that the optimization scheme aims at reducing the cardinality, and/or using an optimization scheme with an objective function which depends on an entropy of a probability distribution of the set (32) of discrete weight values so that the optimization scheme aims at reducing the entropy.

2. Apparatus of claim 1 , wherein the objective function depends also on an estimate of the prediction distortion in case of the training of the network, or a prediction error relative to the pre-learned state caused by the quantizing, with the estimate combined with a term which depends on the cardinality or the frequencies in a weighted sum.

3. Apparatus of claim 1 or 2, wherein the apparatus is configured to quantize the pre- learned state of the weights of the predetermined matrix/tensor portion iteratively by determining a quantization mapping which maps each weight of a pre-learned state of the weights to one of the set of discrete weight values so that an intermediate probability distribution results at which the discrete weight values have the weights of the predetermined matrix/tensor portion mapped thereonto by the quantization mapping, by optimization of the quantization mapping using a Langrangian cost function which combines the entropy of the intermediate probability distribution of the set of discrete weight values and a measure measuring a central tendency of a deviation between the set of discrete weight values to the weights of the pre-learned state or measuring an impact of the deviation between the set of discrete weight values to the weights of the pre-learned state onto the prediction using the neural network, and updating, for each of the discrete weight values, the respective discrete weight values so as to obtain a respective updated discrete weight value by optimizing a cost function which measures a central tendency of a deviation between the respective updated discrete weight value and the weights of the pre-learned state mapped onto the respective discrete weight value or measures an impact of the deviation between the set of discrete weight values to the weights of the pre- learned state onto the prediction using the neural network by the quantization function.

4. Apparatus of claim 1 or 2, wherein the apparatus is configured to train the neural network iteratively by, in each iteration, optimizing preliminary weight values by an iterative optimization iteratively applying a first test input onto the neural network with the weights of the predetermined matrix/tensor portion having preliminarily weights to obtain a test prediction error, and modifying the preliminarily weights so as to reduce a Lagrangian cost function combining a functional term which measures the test prediction error and an entropy term which measures an entropy of a probability distribution resulting from mapping the preliminarily weights onto the set of discrete weight values according to a quantization function, and updating, for each of the discrete weight value, the respective discrete weight value so as to obtain a respective updated discrete weight value by optimizing a cost function which measures a central tendency of a deviation between the respective updated discrete weight value and the preliminary weights mapped onto the respective discrete weight value or measures an impact of the deviation between the set of discrete weight values to the preliminary weights onto the prediction using the neural network by the quantization function.

4a. Apparatus according to claim 4, wherein the iterative optimization modifies the preliminarily weights in a non-quantized domain with the Lagrangian cost function further combining the functional term measuring the prediction error with a discrete weight values distance term measuring a central tendency of a distance of the preliminary weihgts from the discrete weight values and each iteration involves from one iteration to the next an adaptation of the quantization function to the updated discrete weight values in terms of quantization intervals of weights mapped onto the set of discrete weight values.

4b. Apparatus according to claim 4 or 4a, wherein the iterative optimization modifies the preliminarily weight values in a quantized domain represented by the set of discrete weight values.

5. Apparatus of any of claims 1 to 3, wherein the apparatus is configured to determine a set of discrete activation values onto which activations of neurons of the neural network are quantized, by using a further optimization scheme with a further objective function which depends on a cardinality of the set of discrete activation values of activations for a predetermined set of neurons of the neural network in a manner so that the optimization scheme aims at reducing the cardinality, and/or using a further optimization scheme with a further objective function which depends on an entropy of a probability distribution of the set of discrete activation values at which the discrete activation values are actually assumed by activations for neurons of a predetermined set of neurons of the neural network in a manner so that the optimization scheme aims at reducing the entropy.

6. Apparatus of any of claims 1 to 5, wherein the apparatus is configured to train the neural network on the basis of a training set of inputs to the neural network having corresponding target prediction outputs associated therewith.

7. Apparatus for defining of a neural network in terms of a set of discrete activation values onto which activations of neurons of the neural network are quantized, by use of an optimization scheme with an objective function which depends on a cardinality of the set of discrete activation values of activations for a predetermined set of neurons of the neural network in a manner so that the optimization scheme aims at reducing the cardinality, and/or use of an optimization scheme with an objective function which depends on an entropy of a probability distribution of the set of discrete activation values at which the discrete activation values are actually assumed by activations for neurons of a predetermined set of neurons of the neural network in a manner so that the optimization scheme aims at reducing the entropy.

8. Apparatus for deriving a representation (50) of a neural network, the representation comprising one or more matrices or tensors of weights which control an amplification at which activations of neurons (16) of the neural network are forwarded , wherein the apparatus is configured to determine (704a) one or more first data fields (30) indicating a set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of a predetermined matrix/tensor portion of a predetermined matrix or tensor

(10) of the neural network, and, determine (704b), for each of the one or more first data fields (30), a second data field (40), disjoint to the respective first data field, which indicates for each prede- termined discrete weight value indicated by the respective first data field (30), positions of weights in the predetermined matrix/tensor portion assuming the respective predetermined discrete weight value.

9. Apparatus of claim 8, wherein the apparatus is configured to derive the representa- tion of the neural network by lossless conversion from an initial representation (708) of the neural network.

10. Apparatus of claim 8, wherein the lossless conversion from the initial representation of the neural network involves decompression based on predictive decoding and/or entropy decoding.

1 1 . Apparatus of claim 8 or 9, wherein the apparatus comprises an input interface (702) for receiving the initial representation (708) of the neural network and an output interface (706) for outputting the representation (50) as a data file, for storage one a digital storage medium, or to an apparatus for performing a prediction using the representation (50).

12. Apparatus of any of claims 8 to 1 1 , further comprising an apparatus according to any of claims 1 to 6 so as to obtain the initial representation.

13. Apparatus any of claims 8 to 12, wherein the determining (704a) the one or more first data fields (30) comprises, for each of one or more portions of the predetermined matrix/tensor portion, gathering all predetermined discrete weight values of weights within the respective portion, with checking that predetermined discrete weight values of equally valued weights are gathered merely once, so as to obtain a list (66) indicating the predetermined discrete weight values within the respective portion which is free of duplicates.

14. Apparatus of claim 1 3, wherein the one or more portions comprise one or more of the predetermined matrix/tensor portion entirely, and rows of the predetermined matrix/tensor portion, and columns of the predetermined matrix/tensor portion.

1 5. Apparatus of any of claims 8 to 1 , wherein the predetermined matrix/tensor portion comprises the predetermined matrix or tensor entirely, or a subarray thereof.

Apparatus of any of claims 8 to 1 5, configured to determine a most often iy assumed discrete weight value (32) assumed most off ten ly by weights (24) within the predetermined matrix/tensor portion, form the set (36) of predetermined discrete weight values so as to include all discrete weight values assumed by at least one of the weights (24) within the predetermined matrix/tensor portion excluding the most oftenly assumed discrete weight value (32), and include an indication (52) of the most ofteniy assumed discrete weight value (32) into the representation.

17. Apparatus of claim 8 to 15, configured to form the set (36) of predetermined discrete weight values so as to include all discrete weight values within the predetermined matrix/tensor portion excluding zero.

18. Apparatus of any of claims 8 to 17, wherein the predetermined matrix or tensor (10) controls an amplification at which activations of neurons (16) of the neural network are forwarded by computation of a dot product (650) of the predetermined matrix or tensor (10) and an array (442) of activations, the dot product (650) involving first sums (654) over component wise products between weights (24) of the predetermined matrix or ten- sor (10) and activations (652) of neurons (16) of the array (442) of activations, wherein, for each of the first sums (654), the weights related to the component wise products over which the respective sum is formed, are arranged in lines of the predetermined matrix or tensor (10), the lines being one of rows and columns of the predetermined matrix or ten- sor ( 10).

19. Apparatus of any of claims 8 to 18, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons (16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Κ_Ί ; K2) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the apparatus is configured to code into the representation (50), for each of the at least one kernels, a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of the respective kernel, for each first data field (30), a second data field (40) which indicates, for each predetermined discrete weight value indicated by the respective first data field, kernel positions (552) at which the kernel for which the first data field (30) is, has a weight which assumes the respective predetermined discrete weight value.

20. Apparatus of any of claims 8 to 18, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons (16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Κι; K2) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the apparatus is configured to code into the representation (50), a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of any of the at least one kernels, a second data field (40) comprising a portion (690) for each of the at least one kernels which indicates, for each predetermined discrete weight value indicated by the first data field, kernel positions (552) at which the respective kernel has a weight which assumes the respective predetermined discrete weight value.

21 . Apparatus of any of claims 8 to 18, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons (16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Κι ; K2) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the apparatus is configured to code into the representation (50) a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of the at least one kernels, code into the representation (50) a first second-data field portion (140') indicating, for each of the set (36) of predetermined discrete weight values, kernel positions (552) where the respective predetermined discrete weight value is assumed by at least one of the weights of the at least one kernels, so that, on the basis of the first second-data field portion (140') and the first data field (30), a set (566) out of the array (442) of the activations is derivable for each of the set (36) of predetermined discrete weight values, allowing for a pre- computation of a product between each of the set (566) of predetermined discrete weight values and the set (566) of activations derivable for the respective predetermined discrete weight value so as to obtain pre-computed products, and code into the representation (50) a second second-data field portion (64'; 150') which, for each of the at least one kernels, indicates a distribution of the set of predetermined discrete weight values over the kernel positions (552), or determine, for each of components of the dot product output (output'), as to which of the pre-computed products are to be summed-up according to the dot product, and code into the representation (50) the second second-data field portion (150') so that same indicates for each of the components of the dot product output, as to which of the pre-computed products are to be summed-up.

22. Apparatus of any of claims 8 to 18, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons (16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Κι ; K2) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the apparatus is configured to code into the representation (50) a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of the at least one kernels, code into the representation (50) a second-data field (40) comprising a portion (698) for each of the set (36) of predetermined discrete weight values, which indicates, for each of the one or more kernels, kernel positions (552) at which weights of the respective kernel assume the respective predetermined discrete weight value.

23. Apparatus according to any of claims 8 to 18, wherein the weights (24) of the pre- determined matrix/tensor portion are arranged in lines parallel to each other and being .one.of ro.ws. or columns,. . _ .. - _ . . ... . . . . . wherein the apparatus is configured to determine, in determining the one or more first data fields (30), for each of the lines, one first data field (30) which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the respective line, wherein the apparatus is configured to determine, for each first data field (30), the second data field (40) so that same indicates the positions of weights within the line for which the respective first data field (30) indicates the predetermined discrete weight values, which weights assume any of the predetermined discrete weight values indicated by the respective first data field (30), by transversal line indices (60) each indexing a transversal line out of transversa! lines of the predetermined weight matrix or tensor (10) which cross the lines, within which a respective one of said indicated positions is located.

24. Apparatus according to 23, configured to, in determining the one or more first data fields (30), determine a reference list (66) comprised by the one or more first data fields (30) and listing the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion, and determine, for each of the lines, a weight index list (90) comprised by the first data field (30) for the respective line, which indexes the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in the respective line, out of the reference list (66).

25. Apparatus according to 24, wherein the apparatus is configured to concatenate the weight index lists (90) comprised by the first data field (30) for each line to from a concatenated weight index list (92), and, wherein the apparatus is configured to determine, for each first data field (30), the second data field (40) such that same comprises, for each of the predetermined discrete weight values indicated by the respective first data field (30), a second data field portion (68) in- dicating the positions of weights (24) assuming the respective predetermined discrete weight value, in the line for which for which the respective first data field (30) indicates the predetermined discrete weight values, by transversal line indices (60) each indexing a transversal line out of the transversal lines, within which a respective one of said positions is located,

5 wherein the apparatus is configured to concatenate the second data field portions (68) comprised by the second data field (40) determined for the first data fields (30) so as to from a concatenated transversal line index list (64), and wherein the apparatus is configured to generate, for each line, a set (106) of one or more 0 pointers pointing, within the concatenated weight index list (92), to the weight index list (90) of the first data field (30) determined for the respective line, and within the concatenated transversal line index iist (64) to the second data field portions (68) comprised by the second data field (40) associated with the respective line. 5 26. Apparatus according to any of claims 8 to 18, wherein the weights (24) of the predetermined matrix/tensor portion are arranged in lines parallel to each other and being one of rows or columns, wherein the apparatus is configured to determine, in determining the one or more first 0 data fields, a first data field (30) which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in any of the lines, wherein the apparatus is configured to determine the second data field (40) so that same 5 comprises, for each line, a second data field portion (690; 203) which indicates positions of weights within the respective line which assume any of the predetermined discrete weight values indicated by the first data field (30) by transversal line indices each of which indexes a transversal line out of transversal lines of the predetermined weight matrix or tensor which cross the lines, within which a respective one of said positions is located. 0

27. Apparatus according to claim 26, wherein, for each line, the second data field portion (690; 203) is composed of one subportion (68) for each predetermined discrete weight value indicated by the first data field (30) which is assumed by at least one of the weights within the respective line, which subportion (68) indicates the positions of weights 5 within the respective line which assume the respective predetermined discrete weight val-

, . . tie by the transversal line, indices, ...... . . .. ........ . , _.

28. Apparatus according to 27, wherein the apparatus is configured to concatenate, for each line, the subportions (68) comprised by the second data field portion (203) determined for the respective line to form a concatenated transversal line index list (203), and the apparatus is configured to generate, for each line, a set (202) of pointers pointing within the concatenated transversal line index list (203) to the subportions (68) associated with the respective line.

29. Apparatus according to any of claims 27 to 28, wherein

The apparatus is configured to concatenate the subportions (68) comprised by the second data field portions (203) determined for the lines to form a concatenated transversal line index list (64), and the apparatus is configured to generate, for each line, a set of one or more pointers (202; 99) pointing within the concatenated transversal line index list (64) to the subportions (68) associated with the respective line.

30. Apparatus according to any of claims 1 to 18, wherein the weights (24) of the predetermined matrix/tensor portion are arranged in lines parallel to each other and being one of rows or columns, wherein the apparatus is configured to determine, in determining the one or more first data fields, a first data field (30) which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in any of the lines, wherein the apparatus is configured to determine the second data field (40) so that same comprises, for each of the predetermined discrete weight values indicated by the one first data field, a transversal line index list (142) listing transversal line indices of those transversal lines out of transversal lines of the predetermined weight matrix or tensor which cross the lines, within which any weight (24) is located which assumes the respective predetermined discrete weight value, and, for each line, an index list (152) indexing, for each weight within the respective line

_. . which assumes any predetermined discrete weight value, the transversal line index out of the transversal line index list (142) for the respective predetermined discrete weight value which indexes a transversal line out of transversal lines of the predetermined weight matrix or tensor which cross the lines, which the respective weight is located in.

31 . Apparatus according to claim 30, wherein the apparatus is configured to concatenate transversa! line index lists (142) determined for the predetermined discrete weight values indicated by the first data field (30) to form a concatenated transversal line index list (140), and concatenate the index lists (152) determined for each line to obtain a concatenated index list list (150), and determine, for each line, a pointer (161 ) pointing to the index list (152) determined for the respective line.

32. Apparatus according to any of claims 8 to 31 , wherein the apparatus is configured to select a format for the representation (50) out a set of formats supported by the apparatus and provide the representation with a format type indication 620 indicating the selected format.

33. Apparatus according to any of claims 8 to 32, wherein the apparatus is configured to provide the representation with an indication (628) of a set of discrete weight values onto which preactivations of neurons of a portion of the neural network are to be quantized to yield the activations.

34. Apparatus for performing a prediction using a neural network, configured to receive a representation (604; 50) of the neural network, the representation (604) comprising one or more matrices or tensors of weights which control an amplification at which activations of neurons (16) of the neural network are forwarded , derive (610a, 610b) from the representation a set (36) of predetermined discrete weight values actually assumed by at least one of weights (24) of a predetermined matrix/tensor portion of a predetermined matrix or tensor (10) and, for each predetermined discrete weight value of the set (36) of predetermined discrete weight values, an indication of positions of weights (24) assuming the respective predetermined discrete weight value within the predetermined matrix/tensor portion, and compute (614) a dot product (650) of the predetermined matrix or tensor (10) with an array (442) of activations (652) of neurons using the set (36) of predetermined discrete weight values und the indication of the positions.

35. Apparatus of claim 34, wherein the apparatus is configured to receive the representation (604; 50) of the neural network from a digital storage medium, via digital communication, or from a lossless data decompressor.

36. Apparatus of claim 34 or 35, wherein the digital storage medium is a a programmable read only memory, a non-transitory solid state memory, or a volatile memory.

37. Apparatus according to any of claims 34 to 36, configured to compute the dot product (650) of the predetermined matrix or tensor (10) and the array (442) of activations of neurons involving first sums (654) over component wise products between weights (24) of the predetermined matrix or tensor ( 10) and activations (652) of neurons of the array (442) of activations by determining at least one of the first sums by identifying (658) component wise products concerning weights of equal predetermined discrete weight value and executing a multiplication (662) of the equal predetermined discrete weight value with a second sum (660) of the activations of neurons which the component wise products concerning the weights of equal predetermined discrete weight value concern so as to obtain a partial sum of the at least one first sum, and/or identifying (668) weights of the predetermined matrix or tensor (10) which are of equal predetermined discrete weight value and are subject, in accordance with the dot product (650) of the predetermined matrix or tensor (10) and the array (442) of activations of neurons, to a component wise multiplication with a same one of the activations (652) of the array of activations, computing (670) the component wise product between the equal predetermined discrete weight value and the same activation once, and using (672) the computed component wise product as addend in more than one of the first sums (654), and/or decomposing (678) the dot product (650) of the predetermined matrix or tensor (10) and the array (442) of activations of neurons into a weighted sum (123) over dot products of modified versions (127) of the predetermined matrix or tensor (10) and the array (442) of activations of neurons, wherein each of the modified versions (123) is a binary valued matrix or tensor (124) which is associated with a respective one of the predetermined discrete weight values and differs from the predetermined matrix or tensor (10) by, within the predetermined matrix/tensor portion, setting all weights in the predetermined matrix/tensor portion having any but the respective predetermined discrete weight value of the set (36) pf predetermined discrete weight values set to zero and setting all weights in the predetermined matrix/tensor portion having the respective predetermined discrete weight value to one, and the dot product involving the respective version is weighted using the respective predetermined discrete weight value or a difference (125) of the respective predetermined discrete weight value and an reference value (34).

38. Apparatus according to claim 37, configured to derive from the representation a most oftenly assumed discrete weight value (34) not included in the set (36) of predetermined discrete weight values, and configured to in determining the at least one of the first sums, use an offset of the predetermined discrete weight values to the most oftenly assumed discrete weight value for the multiplication with the second sum and add to the partial sum a product of a sum of the activations forming a factor in the component wise products of the at least one first sum, and the most oftenly assumed discrete weight value (34), and/or use an offset of the equal predetermined discrete weight values to the most oftenly assumed discrete weight value for the component wise product between the equal predetermined discrete weight value and the same activation, and add in all first sums an addend of a product of a sum of the activations forming a factor in the component wise products of the first sums, and the most oftenly assumed discrete weight value (34), and/or decompose (678) the dot product (650) of the predetermined matrix or tensor (10) and the array (442) of activations of neurons such that the weighted sum also comprises an addend formed by a dot product of an all-ones matrix and the array of (442) of activations, weighted with the most oftenly assumed discrete weight value.

39. Apparatus according to any of claims 34 to 38, configured to derive from the representation a most oftenly assumed discrete weight value and perform multipiications with any of the set (36) of predetermined discrete weight values using their respective offsets to the most oftenly assumed discrete weight value.

40. Apparatus according to claim 39, configured to decode the set (36) of predetermined discrete weight values in form of the offsets to the most oftenly assumed discrete weight value

41 . Apparatus of any claims 34 to 40, configured so that the derivations involve decoding from digital data which forms the representation; and/or differential decoding; and/or entropy decoding.

42. Apparatus of any claims 34 to 41 , configured to decode from the representation (50) one or more first data fields (30) which indicate the set (36) of predetermined discrete weight values, decode from the representation (50), for each first data field (30), a second data field (40) which indicates positions of weights (24) assuming any predetermined discrete weight value within the predetermined matrix/tensor portion, indicated by the respective first data field (30), with using the indication of the predetermined discrete weight values by the re- spective first data field (30) in order to associate the positions indicated by the second data field (40) to the weights of the predetermined matrix/tensor portion indicated by the respective first data field (30).

43. Apparatus of any claims 34 to 42, configured to decode from the representation (50) one or more data fields (30) which indicate the set (36) of predetermined discrete weight values, decode from the representation (50), for each first data field (30), a second data field (40) which indicates positions of weights assuming any predetermined discrete weight value within the predetermined matrix/tensor portion, indicated by the respective first data field (30), with deducing from a number of the predetermined discrete weight values indicated by the respective first data field (30), a number of scans using which the positions indicated by the second data field (40) for the respective first data field (30) are assigned to the weights (24) of the predetermined matrix/tensor portion.

44. Apparatus of any of claims 34 to 43, wherein the predetermined matrix or tensor (10) controls an amplification at which activations of neurons ( 16) of the neural network are forwarded by computation of a dot product (650) of the predetermined matrix or ten- sor ( 10) and an array (442) of activations, the dot product (650) involving first sums (654) over component wise products between weights (24) of the predetermined matrix or tensor ( 10) and activations (652) of neurons (16) of the array (442) of activations, wherein, for each of the first sums (654), the weights related to the component wise products over which the respective sum is formed, are arranged in lines of the predetermined matrix or tensor (10), the lines being one of rows and columns of the predetermined matrix or tensor (10).

45. Apparatus of any of claims 34 to 44, wherein the predetermined matrix or tensor controls ( 10) an amplification at which a activations of neurons ( 16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Ki ; K_∑) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output¹) representing the convolution, wherein the apparatus is configured to decode from the representation (50), for each of the at least one kernels, a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of the respective kernel, for each first data field (30), a second data field (40) which indicates, for each predeter- mined discrete weight value indicated by the respective first data field, kernei positions (552) at which the kernel for which the first data field (30) is, has a weight which assumes the respective predetermined discrete weight value.

46. Apparatus of any of claims 34 to 44, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons ( 16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Ki ; K_∑) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the apparatus is configured to decode from the representation (50), a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of any of the at least one kernels, a second data field (40) comprising a portion (690) for each of the at least one kernels which indicates, for each predetermined discrete weight value indicated by the first data field, kernel positions (552) at which the respective kernel has a weight which assumes the respective predetermined discrete weight value.

47. Apparatus of claim 45 or 46, configured to map kernel positions (552) indicated by the second data field (40) or a portion (690) thereof as being populated in a respective kernel by a weight assuming a respective predetermined discrete weight value for each of a set of components of the dot product output which are related to the respective kernel, onto a set of activations out of the array of activations, form a partial sum over this set of activations and weight same using the respective predetermined discrete weight value.

48. Apparatus of any of claims 34 to 44, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons (16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Κι; K2) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the apparatus is configured to decode from the representation (50) a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of the at least one kernels, decode from the representation (50) a first second-data field portion (140') indicating, for each of the set (36) of predetermined discrete weight values, kernel positions (552) where the respective predetermined discrete weight value is assumed by at least one of the weights of the at least one kernels, on the basis of the first second-data field portion (140') and the first data field (30), derive a set (566) out of the array (442) of the activations for each of the set (36) of predeter- mined discrete weight values, pre-compute a product between each of the set (566) of predetermined discrete weight values and the set (566) of activations derivable for the respective predetermined discrete weight value so as to obtain pre-computed products, and decode from the representation (50) a second second-data field portion (64'; 150') which, for each of the at least one kernels, indicates a distribution of the set of predetermined discrete weight values over the kernel positions (552), and deduce therefrom, for each of components of the dot product output (output ), as to which of the pre-computed products are to be summed-up according to the dot product, or decode from the representation (50) _ a second second-data field portion (150') that indicates, for each of components of the dot product output (output'), as to which of the pre-computed products are to be summed-up according to the dot product.

49. Apparatus of any of claims 34 to 44, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons (16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Κι; K2) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the apparatus is configured to decode from the representation (50) a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of the at least one kernels, decode from the representation (50) a second-data field (40) comprising a portion (698) for each of the set (36) of predetermined discrete weight values, which indicates, for each of the one or more kernels, kernel positions (552) at which weights of the respective kernel assume the respective predetermined discrete weight value.

50. Apparatus of claim 49, configured to deduce, for each of the set (36) of predetermined discrete weight values, from the portion of the second data field for the respective predetermined discrete weight value, a binary matrix indicating with ones positions of the predetermined matrix or tensor (Κ') assuming the respective predetermined discrete weight value, form a dot product of the binary matrix with the array of activations and weight same using the predetermined discrete weight value.

51 . Apparatus according to any of claims 34 to 44, wherein the weights (24) of the predetermined matrix/tensor portion are arranged in lines parallel to each other and being one of rows or columns, wherein the apparatus is configured to decode from the representation (50), for each of the lines, a first data field (30) which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the respective line, wherein the apparatus is configured to decode from the representation (50), for each first data field (30), a second data field (40) which indicates the positions of weights within the line for which the respective first data field (30) indicates the predetermined discrete weight values, which weights assume any of the predetermined discrete weight values indicated by the respective first data field (30), by transversal line indices (60) each indexing a transversal line out of transversal lines of the predetermined weight matrix or tensor which cross the lines, within which a respective one of said positions is located.

52. Apparatus of claim 51 ,

wherein, for each first data field (30), the second data field (40) comprises, for each of the predetermined discrete weight values indicated by the respective first data field, a second data field portion (68) indicating the positions of weights assuming the respective predetermined discrete weight value, in the line for which for which the respective first data field (30) indicates the predetermined discrete weight values, wherein the apparatus is configured to decode from the representation (50), for each line, a set (106) of one or more pointers comprised by the second data field (40) and pointing to the second data field portion (68) which indicates the positions of the weights assuming any of the predetermined discrete weight values indicated by the respective first data field (30), wherein the apparatus is configured to access, for each line, the second data field portions (68) indicating the positions of weights within the respective line assuming any of the predetermined discrete weight values indicated by any of the one or more first data fields, using the set of one or more pointers decoded for the respective line.

53. Apparatus of claim 52, wherein the apparatus is configured to access, for each line, the first data field (30) indi- eating the predetermined discrete weight values for the respective line, in a manner depending on the set (106) of one or more pointers decoded for the respective line.

54. Apparatus according to 51 to 53, configured to compute the dot product (650) of the predetermined matrix or tensor (10) and the array (442) of activations of neurons involving first sums (654) over component wise products between weights of the predetermined matrix or tensor and activations of neurons of the array of activations by determining at least one of the first sums (654) which corresponds to a corresponding line of the predetermined matrix or tensor (10) by, for each of the predetermined discrete weight values indicated by the respective first data field (30) which is assumed by any weight within the predetermined matrix/tensor portion in the respective line,

forming (660) a partial sum over activations addressed using the transversal line indices by way of which the second data field (40) associated with the correspond- ing line indicates the positions of the weights, and weighting (662) the partial sum using the respective predetermined discrete weight value.

55. Apparatus according to any of claims 51 or 54, configured to decode a reference list (66) from the representation which lists the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion, and decode from the representation (50), for each of the lines, a weight index list (92) comprised by the respective first data field (30) for the respective line, which indexes the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in the respective line, out of the reference list (66), and identify, for each line, the predetermined discrete weight values indicated by the first data field (30) for the respective line which is assumed by any weight within the predetermined matrix/tensor portion in the respective line, using the weight index list (92) and the refer- ence list (66).

56. Apparatus according to 55, wherein the weight index lists (92) comprised by the first data field for each line are concatenated in the representation to from a concatenated weight index list (90), and, for each first data field, the second data field (40) comprises, for each of the predetermined discrete weight values indicated by the respective first data field, a second data field portion (203) indicating the positions of weights assuming the respective predetermined discrete weight value, in the line for which for which the respective first data field (30) indicates the predetermined discrete weight values, by transversal line indices (60) each indexing a transversal line out of the transversal lines, within which a respective one of said positions is located,

the second data field portions (203) comprised by the second data field (40) determined for the first data fields are concatenated in the representation so as to from a concatenat- ed transversal line index list (64), and wherein the apparatus is configured to decode from the representation (50), for each line, a set of one or more pointers pointing, within the concatenated weight index list (90), to the weight index list (92) of the first data field decoded for the respective line, and within the concatenated transversal line index list (64) to the second data field portions (68) comprised by the second data field associated with the respective line.

57. Apparatus according to 56, configured to compute the dot product of the predetermined matrix or tensor and the array of activations of neurons involving first sums over component wise products between weights of the predetermined matrix or tensor and activations of neurons of the array of activations by determine at least one of the first sums which corresponds to a corresponding line of the predetermined matrix or tensor by using the set of one more pointers decoded for the corresponding line to access within the concatenated weight index list (90) the weight index list (92) for the corresponding line so as to determine the predetermined discrete weight val- ues which is assumed by any weight within the predetermined matrix/tensor portion in the corresponding line, and access within the concatenated transversal line index list (64) the second data filed portions (68) comprised by the second data field associated with the correspond- ing line, and for each of the predetermined discrete weight values in the weight index list accessed, forming (660) a partial sum over activations addressed using the transversal line indices (60) of the second data filed portion (68) associated with the respective predetermined discrete weight value and comprised by the second data field for the corresponding line, and weight the partial sum using the respective predetermined discrete weight value.

58. Apparatus according to any of claims 34 to 44, wherein the weights of the predetermined matrix/tensor portion are arranged in lines parallel to each other and being one of rows or columns, wherein the apparatus is configured to decode from the representation a first data field (30) which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in any of the lines, wherein the apparatus is configured to decode from the representation, for each line, a second data field portion composed of one subportion for each predetermined discrete weight value indicated by the first data field which is assumed by at least one of the weights within the respective line, which subportion indicates the positions of weights within the respective line which assume the respective predetermined discrete weight value by transversal line indices each of which indexes a transversal line out of transversal lines of the predetermined weight matrix or tensor which cross the lines, within which a respective one of said positions is located.

59. Apparatus according to claim 58, configured to compute the dot product of the predetermined matrix or tensor and the array of activations of neurons involving first sums over component wise products between weights of the predetermined matrix or tensor and activations of neurons of the array of activations by determining at least one of the first sums which corresponds to a corresponding line of the predetermined matrix or tensor by, for each of the predetermined discrete weight values indicated by the first data field which is assumed by any weight within the predetermined matrix/tensor portion in the respective line,

forming a partial sum over activations addressed using the transversal line indices by way of which the subportion for the respective predetermined discrete weight value comprised by the second data field portion associated with the corresponding line indicates the positions of the weights, and

weighting the partial sum using the respective predetermined discrete weight value.

60. Apparatus according to 58 or 59, wherein for each line, the subportions comprised by the second data field portion determined for the respective line are concatenated in the representation to form a concatenated trans- versa I line index list, and the apparatus is configured to derive from the representations, for each line, a set of pointers pointing within the concatenated transversal line index list to the subportions associated with the respective line.

61 . Apparatus according to claim 60, configured to compute the dot product of the predetermined matrix or tensor and the array of activations of neurons involving first sums over component wise products between weights of the predetermined matrix or tensor and activations of neurons of the array of activations by determining at least one of the first sums which corresponds to a corresponding line of the predetermined matrix or tensor by, for each of the predetermined discrete weight values indicated by the first data field which is assumed by any weight within the predetermined matrix/tensor portion in the respective line,

forming a partial sum over activations addressed using the transversal line indices by way of which the subportion for the respective predetermined discrete weight value comprised by the second data field portion associated with the correspond- ing line indicates the positions of the weights, with accessing the subportion within WO 2019/086104 - ! 46 - PCT/EP2017/077838

the concatenated transversal line index list of the corresponding line using the set of pointers decoded for the corresponding line, and

62. Apparatus according to any of claims 58 to 61 , wherein the apparatus is configured to decode from the representation weight association data, and use the weight association data to associate each subportion with the predetermined discrete weight value assumed by the weights the positions of which the respective sub- portion indicates.

63. Apparatus according to any of claims 58 to 62, wherein the subportions comprised by the second data field portions determined for the lines are concatenated in the representation to form a concatenated transversal line index list, and the apparatus is configured to decode from the representation, for each line, a set of pointers pointing within the concatenated transversal line index list to the subportions as- sociated with the respective line, and use the set of pointers the access the subportions.

64. Apparatus according to any of claims 34 to 44, wherein the weights of the predetermined matrix/tensor portion are arranged in lines parallel to each other and being one of rows or columns, wherein the apparatus is configured to decode from the representation a first data field which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in any of the lines, wherein the apparatus is configured to decode from the representation a second data field which comprises, for each of the predetermined discrete weight values indicated by the first data field, a transversal line index list listing transversal line indices of those transversal lines out of transversal lines of the predetermined weight matrix or tensor which cross the lines, within which any weight is located which assumes the respective predetermined discrete weight value, and, for each line, an index list indexing, for each weight within the respective line which assumes any predetermined discrete weight value, the transversal line index out of the transversal line index list for the respective predetermined discrete weight value which indexes the transversal line which the respective weight is located in.

65. Apparatus according to claim 64, configured to

compute the dot product of the predetermined matrix or tensor and the array of activations of neurons involving first sums over component wise products between weights of the predetermined matrix or tensor and activations of neurons of the array of activations by computing, for each of the predetermined discrete weight values indicated by the first data field, a pre-computed product of the respective discrete weight value and activations addressed by the transversal line indices listed by the transversal line index list de- coded for the respective discrete weight value, and using the index list decoded for the corresponding line, so as to form a sum over pre- computed products having been computed with those activations as a factor which are addressed by transversal line indices indexed by the index list decoded for the correspond line.

66. Apparatus according to claim 64 or 65, wherein the transversal line index lists decoded for the predetermined discrete weight values indi- cated by the first data field are concatenated in the representation to form a concatenated transversal line index list, and the index lists decoded for each line are concatenated in the representation to form a concatenated index list list, and the apparatus is configured to decode from the representation, for each line, a pointer pointing to the index list determined for the respective line, and use the pointer to access the respective line in the concatenated index list list.

67. Apparatus according to any of claims 34 to 44, wherein the weights of the predetermined matrix/tensor portion are arranged in lines parallel to each other and being one of rows or columns, wherein the apparatus is configured to decode from the representation a first data field which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in any of the lines, wherein the apparatus is configured to decode from the representation a second data field which comprises, for each predetermined discrete weight value indicated by the first data field, a second data field portion which indicates the positions of weights within the predetermined matrix/tensor portion which assume the respective predetermined discrete weight value.

68. Apparatus according to claim 67, configured to compute the dot product of the predetermined matrix or tensor and the array of activations of neurons involving first sums over component wise products between weights of the predetermined matrix or tensor and activations of neurons of the array of activations by decomposing the dot product of the predetermined matrix or tensor and the array of activations of neurons into a weighted sum over dot products of modified versions of the predetermined matrix or tensor and the array of activations of neurons, and forming each of the modified versions

by associating same with a respective one of the predetermined discrete weight values,

as a binary valued matrix having positions indicated by the second data field portion comprised by the second data filed for the associated predetermined discrete weight value, and zero otherwise, and

and

weighting same using the respective predetermined discrete weight value or a difference of the respective predetermined discrete weight value and a reference offset value.

69. Apparatus according to 67 or 68, wherein for each predetermined discrete weight value indicated by the first data field, the second data field portion is composed of one subportion for each line within which any weight of the respective predetermined discrete weight value is located in, which subportion indi- cates the positions of weights within the respective line which assume the respective predetermined discrete weight value, by transversal line indices each indexing a transversal line out of transversal lines of the predetermined weight matrix or tensor which cross the lines, within which a respective one of said positions is located.

70. Apparatus according to claim 69, configured to

compute the dot product of the predetermined matrix or tensor and the array of activations of neurons involving first sums over component wise products between weights of the predetermined matrix or tensor and activations of neurons of the array of activations by decomposing the dot product of the predetermined matrix or tensor and the array of activations of neurons into a weighted sum over dot products of modified versions of the pre- determined matrix or tensor and the array of activations of neurons, and forming each of the modified versions

by associating same with a respective one of the predetermined discrete weight values using which the respective modified version is to be weighted, and as a binary valued matrix having ones at positions indicated, per line, by the sub- portion comprised by the second data field portion for the respective line, which, in turn, is comprised by the second data filed for the associated predetermined discrete weight value, and zero otherwise.

71 . Apparatus according to any of claims 67 to 70, wherein wherein the apparatus is configured to decode from the representation line association data, which associates each subportion with the line within which the respective subpor- tion indicates the positions of weights assuming the predetermined discrete weight value for which the respective subportion is, and use the line association data in in accessing the subportions within the second data filed portions.

72. Apparatus according to any of claims 67 to 71 , wherein the subportions comprised by the second data field portions determined for the predetermined discrete weight values indicated by the first data field are concatenated to form a concatenated transversal line index list, and the apparatus is configured to decode from the representation, for each line, a set of one or more pointers pointing within the concatenated transversal line index list to the subpor- tions associated with the respective line, and use the one set of one or more pointers to access the subportions.

73. Digital data parametrizing a neural network which comprising one or more matrices or tensors of weights which control an amplification at which activations of neurons of the neural network are forwarded , wherein digital data comprises one or more first data fields indicating predetermined discrete weight values of a actually assumed by at least one of the weights of a predetermined matrix/tensor portion of the predetermined matrix or tensor of the neural network, and, for each of the one or more first data fields, a second data field, disjoint to the respective first data field, which indicates for each predetermined discrete weight value indicated by the respective first data field, positions of weights in the predetermined matrix/tensor portion assuming the respective predetermined discrete weight value.

74. Digital data of 73, wherein the predetermined matrix/tensor portion comprises the predetermined matrix or tensor entirely, or a subarray thereof.

75. Digital data of claim 73 or 74, further comprising an indication of a most often Iy assumed discrete weight value, the most often ly assumed discrete weight value indicating as a discrete weight value which assumed most offtenly by weights of the predetermined matrix/tensor portion, wherein the set of predetermined discrete weight values includes all discrete weight values within the predetermined matrix/tensor portion excluding the most often!y assumed discrete weight value.

76. Digital data of claim 73 or 74, wherein the set of predetermined discrete weight values includes all discrete weight values within the predetermined matrix/tensor portion excluding zero.

77. Digital data of any of claims 73 to 76, wherein the predetermined matrix or tensor controls an amplification at which activations of neurons of the neural network are forwarded by computation of a dot product of the predetermined matrix or tensor and an array of activations, the dot product involving first sums over component wise products between weights of the predetermined matrix or tensor and activations of neurons of the array of activations, wherein, for each of the first sums, the weights related to the component wise products over which the respective sum is formed, are arranged in lines of the predetermined matrix or tensor, the lines being one of rows and columns.

78. Digital data of any of claims 73 to 77, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons (16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Ki; K2) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the representation (50) comprises for each of the at least one kernels, a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of the respective kernel, for each first data field (30), a second data field (40) which indicates, for each predetermined discrete weight value indicated by the respective first data field, kernel positions (552) at which the kernel for which the first data field (30) is, has a weight which assumes the respective predetermined discrete weight value.

79. Digital data of any of claims 73 to 77, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons (16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Ki; K2) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the representation (50) comprises a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of any of the at least one kernels, a second data field (40) comprising a portion (690) for each of the at least one kernels which indicates, for each predetermined discrete weight value indicated by the first data field, kernel positions (552) at which the respective kernel has a weight which assumes the respective predetermined discrete weight value.

80. Digital data of any of claims 73 to 77, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons (16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Ki ; K2) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the representation (50) comprises a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of the at least one kernels, a first second-data field portion (140') indicating, for each of the set (36) of predetermined discrete weight values, kernel positions (552) where the respective predetermined discrete weight value is assumed by at least one of the weights of the at least one kernels, so that, on the basis of the first second-data field portion (140') and the first data field (30), a set (566) out of the array (442) of the activations is derivable for each of the set (36) of predetermined discrete weight values, allowing for a pre-computation of a product be- tween each of the set (566) of predetermined discrete weight values and the set (566) of activations derivable for the respective predetermined discrete weight value so as to obtain pre-computed products, and a second second-data field portion (64'; 150') which, for each of the at least one kernels, indicates a distribution of the set of predetermined discrete weight values over the kernel positions (552), or determine, for each of components of the dot product output (output'), as to which of the pre-computed products are to be summed-up according to the dot product, and code into the representation (50) the second second-data field portion (150') so that same indicates for each of the components of the dot product output, as to which of the pre-computed products are to be summed-up.

81 . Apparatus of any of claims 73 to 77, wherein the predetermined matrix or tensor controls (10) an amplification at which a activations of neurons (16) of the neural network are forwarded in a manner emulating a convolution of an array (553) at which the neurons are arranged with one or more kernels (Ki; K₂) of equal size so that a dot product between the predetermined matrix or tensor (Κ') and an array (I) of the activations results in an dot product output (output') representing the convolution, wherein the representation (50) comprises a first data field (30) which indicates the set (36) of predetermined discrete weight values actually assumed by at least one of the weights (24) of the at least one kernels, a second-data field (40) comprising a portion (698) for each of the set (36) of predetermined discrete weight values, which indicates, for each of the one or more kernels, kernel positions (552) at which weights of the respective kernel assume the respective predeter- mined discrete weight value.

82. Digital data according to any of claims 73 to 81 , wherein the weights of the predetermined matrix/tensor portion are arranged in lines parallel to each other and being one of rows or columns, wherein the one or more first data fields comprise, for each of the lines, one first data field which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the respective line, wherein, for each first data field, the second data field indicates the positions of weights within the line for which the respective first data field indicates the predetermined discrete weight values, which weights assume any of the predetermined discrete weight values indicated by the respective first data field, by transversal line indices each indexing a transversal line out of transversal lines of the predetermined weight matrix or tensor which cross the lines, within which a respective one of said positions is located.

83. Digital data according to 82, wherein

the one or more first data fields comprise

a reference list listing the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion, and for each of the lines, a weight index list for the respective line which indexes the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in the respective line, out of the reference list.

84. Digital data according to 83, wherein the weight index lists comprised by the first data field for each line are concatenated to from a concatenated weight index list, and, wherein, for each first data field, the second data field comprises, for each of the predetermined discrete weight values indicated by the respective first data field, a second data field portion indicating the positions of weights assuming the respective predetermined discrete weight value, in the line for which for which the respective first data field indicates the predetermined discrete weight values, by transversal line indices each indexing a transversal line out of the transversal lines, within which a respective one of said positions is located, wherein the second data field portions comprised by the second data field determined for the first data fields are concatenated so as to from a concatenated transversa! line index list, and wherein the digital data comprises, for each line, a set of one or more pointers pointing, within the concatenated weight index list, to the weight index list of the first data field determined for the respective line, and within the concatenated transversal line index list to the second data field portions comprised by the second data field associated with the respective line.

85. Digital data according to any of claims 73 to 84, wherein the weights of the prede- termined matrix/tensor portion are arranged in lines parallel to each other and being one of rows or columns, wherein the one or more first data fields comprises one first data field which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in any of the lines, wherein the second data field comprises, for each line, a second data field portion composed of one subportion for each predetermined discrete weight value indicated by the first data field which is assumed by at least one of the weights within the respective line, which subportion indicates the positions of weights within the respective line which as- sume the respective predetermined discrete weight value by transversa! line indices each of which indexes a transversa! line out of transversal lines of the predetermined weight matrix or tensor which cross the lines, within which a respective one of said positions is located.

86. Digital data according to 85, wherein for each line, the subportions comprised by the second data field portion determined for the respective line are concatenated to form a concatenated transversal line index list, and for each line, the digital data comprises a set of pointers pointing within the concatenated transversal line index list to the subportions associated with the respective line.

87. Digital data according to 85 or 86, wherein the second data field comprises weight association data, which associates each subportion with the predetermined discrete weight value assumed by the weights the positions of which the respective subportion indicates.

88. Digital data according to any of claims E9 to E1 1 , wherein the subportions comprised by the second data field portions determined for the lines are concatenated to form a concatenated transversal line index list, and wherein the digital data comprises, for each line, a set of pointers pointing within the concatenated transversal line index list to the subportions associated with the respective line.

89. Digital data according to any of claims 73 to 88, wherein the weights of the predetermined matrix/tensor portion are arranged in lines parallel to each other and being one of rows or columns, wherein the one or more first data fields comprises one first data field which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in any of the lines, wherein the second data field comprises

for each of the predetermined discrete weight values indicated by the one first data field, a transversal line index list listing transversal line indices of those transversal lines out of transversal lines of the predetermined weight matrix or tensor which cross the lines, within which any weight is located which assumes the respective predetermined discrete weight value, and, for each line, an index list indexing, for each weight within the respective line which assumes any predetermined discrete weight value, the transversal line index out of the transversal line index list for the respective predetermined discrete weight val- ue which indexes the transversal line which the respective weight is located in.

90. Apparatus according to claim 89, wherein wherein transversal line index lists for the predetermined discrete weight values indicated by the one first data field are concatenated to form a concatenated transversal line index list, and the index lists for each line, are concatenated to form a concatenated index list list and the digital data comprises, for each line, a pointer pointing to the index list determined for the respective line.

91 . Digital data according to any of claims 73 to 90, wherein the weights of the predetermined matrix/tensor portion are arranged in lines parallel to each other and being one of rows or columns, wherein the one or more first data fields comprise one first data field which indicates the predetermined discrete weight values actually assumed by at least one of the weights within the predetermined matrix/tensor portion in any of the lines, wherein the second data field comprises, for each predetermined discrete weight value indicated by the first data field, a second data field portion which indicates the positions of weights within the predetermined matrix/tensor portion which assume the respective predetermined discrete weight value.

92. Digital data according to 91 , wherein

for each predetermined discrete weight value indicated by the first data field, the second data field portion is composed of one subportion for each line within which any weight of the respective predetermined discrete weight value is located in, which subportion indicates the positions of weights within the respective line which assume the respective predetermined discrete weight value , by transversal line indices each indexing a transversal line out of transversal lines of the predetermined weight matrix or tensor which cross the lines, within which a respective one of said positions is located,

93. Digital data according to 91 or 92, wherein the second data field comprises line association data which associates each subportion with the line within which the respective subportion indicates the positions of weights assuming the predetermined discrete weight value for which the respective subportion is.

94. Digital data according to any of claims 91 to 93, wherein the subportions comprised by the second data field portions for the predetermined discrete weight values indicated by the first data field are concatenated to form a concatenated transversal line index list, and the digital data comprises, for each line, a set of one or more pointers pointing within the concatenated transversal line index list to the subportions associated with the respective line.

95. Digital data according to any of claims 73 to 31 , wherein the representation (50) comprises a format type indication 620 indicating a format of the representation.

96. Digital data according to any of claims 73 to 32, wherein the representation (50) comprises an indication (628) of a set of discrete weight values onto which preactivations of neurons of a portion of the neural network are to be quantized to yield the activations.

97. Methods performed by apparatuses according to any of claims 1 to 72.

98. Computer program having a program code for performing, when running on a computer, a method according to claim 97.