DE102021124203A1 - Method and device for training a neural network with quantized parameters - Google Patents

Abstract

A device for training a neural network for a target function is described. The apparatus is configured to quantize one or more activations of the one or more layers of the neural network using an activation quantization function to determine one or more corresponding quantized activations, and the one or more sets of parameters of the one or more layers using a quantize a parameter quantization function to determine one or more corresponding sets of quantized parameters. The device is also set up to determine a predicted output of the neural network based on the one or more quantized activations and based on the one or more sets of quantized parameters. Furthermore, the device is set up to adapt the activation quantization function, the parameter quantization function and the one or more sets of parameters on the basis of the predicted output and on the basis of an optimization function.

Description

The invention relates to a device and a corresponding method for training a neural network that has quantized parameters and/or quantized activations.

An at least partially automated (motor) vehicle (e.g. a passenger car, a truck or a bus) can be set up to evaluate sensor data from one or more environmental sensors of the vehicle (e.g. an image camera, a lidar sensor, a radar sensor, etc.), e.g. to detect one or more objects in the vicinity of the vehicle. The vehicle can then be guided longitudinally and/or transversely in an automated manner depending on the one or more detected objects.

Objects can be detected on the basis of the sensor data from one or more environmental sensors using specifically trained neural networks, in particular so-called deep neural networks and/or convolutional neural networks. A neural network typically has a large number of parameters (which are also referred to as neuron parameters in this document) which must be stored and which must be processed with sensor data in order to carry out object recognition based on the sensor data. The storage and processing of the large number of neuron parameters typically leads to a relatively large amount of storage and computing resources (especially on an embedded system, such as on a control unit in a vehicle).

The present document deals with the technical task of training a neural network, in particular the large number of (neural) parameters of the neural network, in such a way that the resource requirements of the neural network are reduced without affecting the quality (e.g. the object recognition quality) of the neural network (significantly) to affect. Alternatively or in addition, the present document deals with determining the resource requirements of a neural network in an efficient and precise manner.

The object is solved in each case by the independent claims. Advantageous embodiments are described inter alia in the dependent claims. It is pointed out that additional features of a patent claim dependent on an independent patent claim without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim can form a separate invention independent of the combination of all features of the independent patent claim, which can be made the subject of an independent claim, a divisional application or a subsequent application. This applies equally to the technical teachings described in the description, which can form an invention independent of the features of the independent patent claims.

According to one aspect, an apparatus (e.g., a computer or a server) for training a neural network for a target function is described. The neural network can be trained on the basis of training data, with the training data comprising a large number of training data sets. In this case, a training data set can have an activation for an input layer of the neural network (which is also referred to in this document as the input value or input data of the neural network). Furthermore, the training data set can include a target output (i.e. a target output value or target output data) which the neural network should provide for the corresponding activation at the output layer of the neural network. The target function (e.g. object recognition or classification) for which the neural network is to be trained can be described by the large number of pairings of activation (the input layer) and target output (the output layer).

The neural network has a set of one or more layers, each with a set of parameters (in particular weights). In particular, the neural network may comprise L consecutive layers, with L greater than 10, in particular greater than 100. The first layer (l=1) may be referred to as the input layer and the last layer (l=L) may be referred to as the output layer.

A layer can be a convolution layer that is designed to bring about a convolution between the respective set of (possibly quantized) parameters and the respective (possibly quantized) activation in order to produce an output of the respective layer determine. In particular, the neural network can be a convolutional neural network (CNN).

For example, the activation for the input layer can comprise one or more data matrices, in particular a data tensor (which eg comprises one or more images from a camera). This activation can be processed sequentially through the different layers of the neural network to provide an output (ie output data) at the output layer. The output can also include one or more data matrices, in particular a data tensor. For example, the output can display frames (ie, bounding boxes) around one or more detected objects in a camera image.

The device can be designed to use the output of a layer l as an activation for the immediately following layer l+1. Furthermore, the device can be set up to use the output of the output layer of the neural network as the predicted output of the neural network.

The device is set up, in an iteration k of a learning algorithm, to quantize one or more activations of the one or more slices using an activation quantization function in order to determine one or more corresponding quantized activations. In this case, the activation quantization function can indicate for each individual layer how the activation of the respective layer is to be quantized in each case. In particular, the respective number of (possibly evenly distributed) quantization values can be displayed. The quantization, in particular the number of quantization values, can be different in the different layers. The activation quantization function can have been defined in the directly preceding iteration k−1.

Furthermore, the device can be set up to quantize the one or more sets of (non-quantized) parameters of the one or more slices using a parameter quantization function in order to determine one or more corresponding sets of quantized parameters. In this case, the parameter quantization function can indicate for each individual layer how the set of (non-quantized) parameters of the respective layer is to be quantized in each case. In particular, the respective number of (possibly evenly distributed) quantization values can be displayed. The quantization, in particular the number of quantization values, can be different in the different layers. The parameter quantization function can have been defined in the directly preceding iteration k−1.

Furthermore, the device can be set up to determine a predicted output of the neural network (at the output layer of the neural network) based on the one or more quantized activations and based on the one or more sets of quantized parameters. A (non-quantized) output of the respective layer l can be determined sequentially for the L layers, starting with the layer l=1, on the basis of the quantized activation and on the basis of the set of quantized parameters of the respective layer l. The (non-quantized) output of layer l may correspond to the (non-quantized) activation of layer l+1 immediately following. The (unquantized) output of layer l can then be quantized using the activation quantization function (for the immediately following layer l+1) to provide the quantized activation for the immediately following layer l+1.

The device is further configured to adapt the activation quantization function, the parameter quantization function and/or the one or more sets of (non-quantized) parameters based on the predicted output and based on an optimization function. The number of quantization values can be adjusted individually for each individual layer (possibly differently).

In particular, the device can be set up to determine a respective predicted output of the neural network for a large number of training activations (i.e. for a large number of training data sets). The training activations can be directed to one or more recognition and/or classification tasks in an autonomous vehicle. The activation quantization function, the parameter quantization function, and/or the one or more sets of (unquantized) parameters can then be adjusted based on the plurality of predicted outputs (of the neural network) and based on the optimization function.

The device can also be set up to carry out a large number of iterations of the above steps (e.g. until a certain termination, in particular convergence, criterion is reached).

In particular, the device can be set up to carry out a large number of iterations of the above-mentioned steps within the framework of one epoch of the learning algorithm. As part of an iteration, a specific Batch of training data, ie a specific subset of the training data, are used. In an iteration, forward and backward propagation can be performed by the neural network in order to adapt the one or more sets of parameters and the quantization functions. A further batch of training data, ie a further subset of training data, can then be used for a subsequent iteration.

An epoch of the learning algorithm is typically complete when all of the training data has been run through. A further epoch of the learning algorithm (with the same training data and again with a large number of iterations) can then be carried out. A large number of epochs of the learning algorithm can be carried out in a corresponding manner, e.g. until a termination criterion, in particular a convergence criterion, is reached.

A device is thus described which is designed to simultaneously learn the quantization functions and the parameters of the neural network (individually for each individual layer) within the framework of an iterative learning algorithm (using a combined optimization function). In this way, an optimized compromise between resource expenditure and quality of the neural network can be achieved.

The device can be set up to compare the predicted output (of the neural network) with a target output (from the respective training data set). The activation quantization function, the parameter quantization function and/or the one or more sets of parameters can then be adjusted in a particularly precise manner based on the comparison (in particular to train the neural network with respect to the target function).

In particular, the device can be set up to determine a deviation of the predicted output from the target output. Typically, a large number of deviations (for the large number of training data sets) can be determined. A gradient of the optimization function for the deviation (in particular for the large number of deviations) can then be determined. The activation quantization function, the parameter quantization function, and/or the one or more sets of parameters can then be adjusted based on the gradient (in particular using a backpropagation algorithm). In this way, the neural network (including the quantization functions) can be trained in a particularly precise and robust manner.

The optimization function can include a first sub-function aimed at adapting the neural network, in particular the one or more sets of parameters, in relation to the target function. For example, the first sub-function can depend on a deviation of the predicted output from the target output (for the plurality of training data sets).

Furthermore, the optimization function can include a second sub-function which is aimed at adapting (in particular reducing) the resource expenditure of the neural network. In this case, in particular, the resource expenditure associated with the quantization of the one or more activations and the one or more sets of parameters caused by the activation quantization function and by the parameter quantization function can be considered. A second sub-function can thus be taken into account within the optimization function, which indicates how resource-intensive (in terms of the number of arithmetic operations and/or in terms of the required memory requirements and/or in terms of the required computing time (latency) to determine an output of the neural network) is the neural network.

The optimization function (in English, loss function) can thus take into account both the objective function and the resource expenditure. In this way, a particularly advantageous compromise can be achieved between quality and resource expenditure.

As already explained above, the activation quantization function and/or the parameter quantization function for a layer can each have the number | U | of (possibly evenly distributed) quantization values with which the activation of this layer or with which the set of parameters of this layer are quantized. The available quantization values are preferably “unique” or unique. The device can be set up to adapt the number of quantization values (individually for each individual layer) in order to adapt the activation quantization function and/or the parameter quantization function. Thus iteratively the number | U | adapted from available quantization values for quantization of the data in the different layers, and in particular learned, become. In this way, the resource requirements of the neural network can be adjusted in an efficient and precise manner.

The device can be set up, the number | U | of quantization values (for the activation quantization function and/or for the parameter quantization function in a respective layer) from a set of values. The value set can include the possible values for the number |U| establish. In this case, it is typically advantageous to provide a quantization function at the end of the iterative learning algorithm that uses a specific number of bits for quantization. In other words, the set of values for the number |U| of quantization values of a learned quantization function is preferably {2 ^b } with base two exponential values, where b is a natural number (greater than zero) (eg b=1, 2, 3, 4, 5,...).

The device can be set up to reduce the cardinality of the set of values (for selecting the number 1 U 1 of quantization values) as the number of iterations of the learning algorithm increases. For example, the set of real numbers can initially be used as the set of values. If necessary, this initial set of values (after a certain iteration) can be reduced to a set of real numbers with a limited number of decimal places. Furthermore, the set of values (after a certain iteration) can be limited to the set of natural numbers. Finally, if necessary (after a certain iteration), the value set {2 ^b } can be used.

By progressively reducing the available set of values as part of the iterative learning algorithm, the quantization functions can be adjusted, in particular optimized, in a particularly precise and robust manner. In particular, it can be achieved in this way that a gradient of the optimization function can be determined and used for the adjustment.

As already explained above, the device can be set up to determine different numbers of quantization values of the activation quantization function and/or the parameter quantization function for different layers of the neural network. In this way, a particularly advantageous compromise between quality and resource efficiency can be achieved.

The activation quantization function and/or the parameter quantization function can each have a scaling value c, which can be used to define the value range of quantization values for quantization of an activation or a set of parameters. The scaling value can be different for different layers. In particular, an individual scaling value (each individually for the quantization of the activation and for the quantization of the parameters) can be learned for each layer. The device can be set up to adapt the scaling value c (for the respective layer and/or for the respective quantization function) on the basis of the predicted output (of the neural network) and on the basis of the optimization function. In this way, the quality of the neural network, in particular of the quantization functions, can be further increased.

As already explained above, the optimization function can include a sub-function which is designed to take into account the resource requirements of the neural network. The optimization function can in particular include a sub-function (e.g. an estimation function) which is designed to assign a value to the resource requirements of the neural network on the basis of one or more characteristic variables for describing the neural network, the activation quantization function and/or the parameter quantization function predict. The estimator can be trained in advance (on the basis of resource training data). In a preferred example, the estimator comprises a learned Gaussian process. In this way, the resource expenditure of the neural network can be optimized in a particularly precise manner.

In one aspect, an apparatus (e.g., a computer or a server) for determining an estimate of the resource cost of a neural network is described. The aspects described in connection with this device are also applicable, alone or in combination, to the one or more other devices described in this document. Conversely, the aspects of the one or more other devices are also applicable to this device, alone or in combination. Furthermore, the aspects described in relation to a neural network can also be used individually or in combination in connection with this device.

As already explained above, the neural network can have a set of one or more layers, each with a set of (possibly quantized) parameters. Furthermore, it can neural network may be configured to quantize one or more activations of the one or more layers using an activation quantization function to determine one or more corresponding quantized activations. In addition, the neural network can be designed to determine a predicted output of the neural network based on the one or more quantized activations and based on the one or more sets of quantized parameters.

The device can be set up, values of one or more characteristic variables for describing the neural network (in particular for describing the structure of the neural network and/or for describing the number of arithmetic operations of the neural network), the activation quantization function and/or the parameters - determine the quantization function with which the one or more sets of quantized parameters were quantized.

• • zumindest eine Größe in Bezug auf die Anzahl von Rechenoperationen, insbesondere Multiplikationen, die in den ein oder mehreren Schichten des neuronalen Netzes durchzuführen sind; Beispielsweise kann die Größe des Faltungs-Kernels einer Schicht beschrieben werden; und/oder
• • zumindest eine Größe in Bezug auf die Anzahl 1 U 1 von unterschiedlichen Quantisierungswerten, mit denen die ein oder mehreren quantisierten Aktivierungen repräsentiert werden und/oder mit denen die ein oder mehreren Aktivierungen durch die Aktivierungs-Quantisierungsfunktion quantisiert werden; und/oder
• • zumindest eine Größe in Bezug auf die Anzahl 1 U 1 von unterschiedlichen Quantisierungswerten, mit denen die ein oder mehreren Mengen von quantisierten Parametern repräsentiert werden und/oder mit denen die ein oder mehreren Mengen von quantisierten Parametern durch die Parameter-Quantisierungsfunktion quantisiert wurden.

The one or more characteristic quantities may include:

• • at least one variable in relation to the number of arithmetic operations, in particular multiplications, to be carried out in the one or more layers of the neural network; For example, the size of the convolution kernel of a layer can be described; and or
• • at least one size related to the number 1 U 1 of different quantization values with which the one or more quantized activations are represented and/or with which the one or more activations are quantized by the activation quantization function; and or
• • at least one size related to the number 1 U 1 of different quantization values with which the one or more sets of quantized parameters are represented and/or with which the one or more sets of quantized parameters were quantized by the parameter quantization function.

The device can also be set up to predict the estimated value of the resource requirement of the neural network based on an estimation function and based on the values of the one or more characteristic variables. The estimator can be trained in advance on the basis of resource training data. The resource training data can have a large number of resource data sets. A resource data record can display a combination of values of one or more characteristic variables on the one hand and a target value of the resource expenditure on the other.

A machine-learned estimator can thus be provided and used to efficiently determine an accurate estimate of the resource cost of a neural network. This estimated value can then advantageously be used as part of a learning algorithm for training a neural network for a specific target function, in order to provide a neural network with a high target function quality and with a low outlay on resources.

In a preferred example, the estimator includes a Gaussian process, which was trained using a regression method based on the resource training data, for example. The use of such an estimator makes it possible to determine a gradient of the estimator, which can then be used for a robust and precise optimization of the parameters and/or the quantization functions of a neural network.

ist. Dabei kann m(ρ) ein angelernter Mittelwert des Gauß-Prozesses $\mathcal{G}\mathcal{P}\left(\right)$

sein. K (ρ, ρ') kann eine angelernte Kovarianz des Gauß-Prozesses $\mathcal{G}\mathcal{P}\left(\right)$

sein. Ferner kann ρ ein Skalar oder ein Vektor sein, der die Werte der ein oder mehreren charakteristischen Größen zur Beschreibung des neuronalen Netzes, der Aktivierungs-Quantisierungsfunktion und/oder der Parameter-Quantisierungsfunktion umfasst.The estimator can be designed to determine an estimate of the resource requirements, which is proportional to $\mathcal{G}\mathcal{P}\left(m\left(\rho \right),K\left(\rho ,\rho \text{'}\right)\right)$

is. Here, m(ρ) can be a learned mean value of the Gaussian process $\mathcal{G}\mathcal{P}\left(\right)$

be. K(ρ,ρ') can be a learned covariance of the Gaussian process $\mathcal{G}\mathcal{P}\left(\right)$

be. Furthermore, ρ can be a scalar or a vector that includes the values of the one or more characteristic variables for describing the neural network, the activation quantization function and/or the parameter quantization function.

mit einer angelernten Amplitude σ und einem angelernten Skalierungsfaktor ℓ.In a preferred example, the covariance K(ρ,ρ') is given by $$K\left(\rho ,\rho \text{'}\right)={\sigma }^{2}exp\left(-\frac{{\Vert \rho -\rho \text{'}\Vert }^2}{2{\mathcal{l}}^2}\right)$$

with a learned amplitude σ and a learned scaling factor ℓ.

In this way, estimated values for the resource expenditure can be provided in a particularly efficient and precise manner.

The device can be set up to learn the activation quantization function, the parameter quantization function and/or the one or more sets of (possibly quantized) parameters as part of a learning algorithm using the estimation function and/or on the basis of the estimated value determined. In particular, the device can be set up to iteratively learn the activation quantization function, the parameter quantization function and/or the one or more sets of (possibly quantized) parameters using the estimation function as part of an optimization function.

The estimator is preferably designed in such a way that the estimator can be differentiated, in particular can be differentiated according to the one or more characteristic variables. This is the case, for example, for the above-mentioned Gaussian process-based estimator. The ability to differentiate enables particularly efficient and reliable use as part of an optimization process (e.g. for training a neural network).

According to a further aspect, a method for training a neural network for a target function is described. The neural network comprises a set of one or more layers, each with a set of parameters. A layer can have 1000 or more parameters. The method includes, in one iteration of a learning algorithm, quantizing one or more activations of the one or more layers using an activation quantization function to determine one or more corresponding quantized activations. The method further includes quantizing the one or more sets of parameters of the one or more layers using a parameter quantization function to determine one or more corresponding sets of quantized parameters

The method further includes determining, based on the one or more quantized activations and based on the one or more sets of quantized parameters, a predicted output of the neural network. The method also includes adjusting the activation quantization function, the parameter quantization function, and the one or more sets of parameters based on the predicted output and based on an optimization function.

According to a further aspect, a method for determining an estimated value of the resource expenditure of a neural network is described. The method includes determining values of one or more characteristic variables for describing the neural network (in particular the arithmetic operations and/or the structure of the neural network), the activation quantization function (for quantization of the activations) and/or the parameter quantization function ( for quantization of the parameters).

The method also includes determining the estimated value of the resource expenditure of the neural network using an estimator and using the values of the one or more characteristic variables. The estimator preferably includes a Gaussian process that was learned using a regression method based on resource training data.

According to a further aspect, a software (SW) program is described. The SW program can be set up to be executed on a processor and thereby to perform one or more of the methods described in this document.

According to a further aspect, a storage medium is described. The storage medium can comprise a SW program which is set up to be executed on a processor and thereby to carry out one or more of the methods described in this document.

It should be noted that the methods, devices and systems described in this document can be used both alone and in combination with other methods, devices and systems described in this document. Furthermore, any aspects of the methods, devices and systems described in this document can be combined with one another in a variety of ways. In particular, the features of the claims can be combined with one another in many different ways. Furthermore, features listed in brackets are to be understood as optional features.

• 1 beispielhafte Komponenten eines Fahrzeugs;
• 2a ein beispielhaftes neuronales Netz;
• 2b ein beispielhaftes Neuron;
• 3 ein beispielhaftes Verfahren zum Anlernen eines neuronalen Netzes;
• 4 ein Ablaufdiagramm eines beispielhaften Verfahrens zum Anlernen eines neuronalen Netzes unter Berücksichtigung des Ressourcenaufwands; und
• 5 ein Ablaufdiagramm eines beispielhaften Verfahrens zur Ermittlung eines Schätzwertes des Ressourcenaufwandes eines neuronalen Netzes.

The invention is described in more detail below using exemplary embodiments. show it

• 1 exemplary components of a vehicle;
• 2a an exemplary neural network;
• 2 B an exemplary neuron;
• 3 an exemplary method for training a neural network;
• 4 a flowchart of an exemplary method for training a neural network, taking into account the resource requirements; and
• 5 a flow chart of an exemplary method for determining an estimated value of the resource expenditure of a neural network.

As explained at the outset, the present document deals with training a neural network for a specific target function in such a way that an optimized compromise is made available in relation to the quality of the neural network and in relation to the resource requirements of the neural network. Exemplary target functions of a neural network are: classification, object recognition and/or segmentation. Furthermore, the present document deals with determining the resource expenditure of a neural network in an efficient and precise manner.

1 FIG. 12 shows exemplary components of a vehicle 100. The vehicle 100 includes one or more surroundings sensors 102 which are set up to record surroundings data (ie sensor data) in relation to the surroundings of the vehicle 100. FIG. Example surroundings sensors 102 are an image camera, a radar sensor, a lidar sensor and/or an ultrasonic sensor. Furthermore, the vehicle 100 typically includes one or more vehicle sensors 103 which are set up to acquire status data (ie sensor data) in relation to a status variable of the vehicle 100 . Exemplary state variables are the driving speed and/or the (longitudinal) acceleration of vehicle 100. Vehicle 100 also includes one or more actuators 104 for automatic longitudinal and/or lateral guidance of vehicle 100. Exemplary actuators 104 are a drive motor, a braking device and /or a steering device.

A control unit 101 of the vehicle 100 can be set up to operate one or more actuators 104 automatically on the basis of the environment data and/or on the basis of the status data. In particular, control unit 101 can be set up to detect an object in the vicinity of vehicle 100 on the basis of the sensor data from one or more surroundings sensors 102 . The one or more actuators 104 can then be operated as a function of the detected object, in particular in order to guide the vehicle 100 at least partially in an automated manner.

At least one neural network can be used to recognize an object on the basis of the sensor data from one or more surroundings sensors 102, which has been trained on the basis of training data for the task of object recognition. 2a and 2 B show exemplary components of a neural network 200, in particular a feedforward network. In the example shown, the neural network 200 comprises two input neurons or input nodes 202, which at a specific point in time t each record a current value of an input variable as the input value 201 (ie an activation). The one or more input nodes 202 are part of an input layer 211. In general, the neural network 200 can be designed to record input data (ie an activation) with one or more input values 201 (eg the pixels of an image) from a data sample.

The neural network 200 further comprises neurons 220 in one or more hidden layers 212 of the neural network 200. Each of the neurons 220 can have as input values the individual output values (i.e. the output) of the neurons of the previous layer 212, 211 (or at least a part thereof ). Processing is carried out in each of the neurons 220 in order to determine an output value of the neuron 220 depending on the input values. The output values of the neurons 220 of the last hidden layer 212 can be processed in an output neuron or output node 220 of an output layer 213 in order to determine the one or more output values 203 (i.e. the predicted output) of the neural network 200. In general, the network 200 may be configured to provide output data (i.e., an output) having one or more output values 203 .

2 B 1 illustrates exemplary signal processing within a neuron 220, particularly within the neurons 202 of the one or more hidden layers 212 and/or the output layer 213. The one or more input values 221 of the neuron 220 are assigned individual weights 222 in order to determine a weighted sum 224 of the input values 221 in a summation unit 223 (possibly taking into account a bias or offset 227). The weighted sum 224 can be mapped to an output value 226 of the neuron 220 by an activation function 225 . In this case, the activation function 225 can be used, for example, to limit the value range. For example, a sigmoid function or a hyperbolic tangent (tanh) function or a rectified linear unit (ReLU), eg f(x)=max(0, x) can be used as activation function 225 for a neuron 220 . If necessary, the value of the weighted sum 224 can be shifted with an offset 227 .

A neuron 220 thus has weights 222 and/or possibly an offset 227 as neuron parameters. The neuron parameters of the neurons 220 of a neural network 200 can be trained in a training phase in order to cause the neural network 200 to approximate a specific target function and/or to model a specific behavior.

A neural network 200 can be trained using a backpropagation algorithm, for example. For this purpose, in a first phase of a k ^th iteration and/or epoch of a learning algorithm for the input values 201 (ie for activation) at the one or more input nodes 202 of the neural network 200, output values corresponding to a specific training set of data samples can be generated 203 (ie an output) of the one or more output neurons 220 can be determined. The error value of an optimization or error function can be determined on the basis of the output values 203 . In particular, the deviations between the output values 203 calculated by the network 200 (ie the predicted output) and the target output values (ie a target output) can be calculated from the data samples as error values.

In a second phase of the k ^th iteration and/or epoch of the learning algorithm, the error or the error value is backpropagated from the output layer 213 to the input layer 211 of the neural network 200 in order to change the neuron parameters of the neurons 220 layer by layer. In this case, the determined error function at the output can be partially derived according to each individual neuron parameter of the neural network 200 in order to determine an extent and/or a direction for the adjustment of the individual neuron parameters. This learning algorithm can be repeated iteratively for a large number of epochs or iterations until a predefined convergence and/or termination criterion is reached.

A neural network 200 can thus have L different layers 211, 212, 213, each with a set of (neuron) parameters 222, 227, in particular weights. The input value 201 of a layer 211, 212, 213 can be a matrix or a tensor, for example. The input value 201 can also be referred to as activation of the respective layer 211, 212, 213. The activation Al ^-1 to a layer l ∈ {1,...,L} can be processed with the parameters ^Wl of layer l to find the output ^Al of layer l. The output A ^l of layer l then represents the activation for the subsequent layer l+1. A° thus corresponds to the input value 201 of the neural network 200, and A ^L corresponds to the output value 203 of the neural network 200. In the individual layers l ∈ {1, ..., L} a convolution can take place between the activation A ^l-1 (ie the activation tensor) and the parameters W ^l (ie the parameter tensor).

(für den Parameter-Tensor W^l) und/oder als $b_a^l$

(für den Aktivierungs-Tensor A^l) bezeichnet werden.The accuracy with which the arithmetic operations are carried out in the individual layers can differ in different layers. In particular, the number of bits used to represent the activation Al ^-1 and/or the parameters ^Wl can be different in different layers. The number of bits per tensor entry can be specified as $b_w^l$

(for the parameter tensor W ^l ) and/or as $b_a^l$

(for the activation tensor A ^l ).

wobei x_q der quantisierte Tensoreintrag ist.The number of bits for a tensor entry x (eg for a parameter or for an activation) can be eg b. The tensor entry x can then be represented by 2 ^b different quantization values. The different quantization values can be in a value range [-c, c] (eg for parameters) or [0, c] (eg for activations), where c is a scaling value that can be learned for the individual layers. The quantization of a tensor entry x can then be effected using the following formula $${\text{x}}_{\text{q}}=RO\mathrm{and}n\mathrm{i.e}\left(\text{clip}\left(x\mathrm{,0,}c\right)\cdot \frac{\left(2^b-1\right)}{c}\right)\cdot \frac{c}{\left(2^b-1\right)};$$

where x _q is the quantized tensor entry.

wobei x̃_q der quantisierte Tensoreintrag ist.The number of different quantization values that can be taken into account in the above quantization formula can only assume the values 2 ^b , with b=1,2,3,4,.... This discontinuity in terms of the possible number of different quantization values can lead to convergence problems in the context of the iterative learning algorithm. To train the parameters of a neural network, it can therefore advantageously be assumed that any number 1 U 1 of different quantization values can be used for quantization, where |U| is a real number (and thus can take on any value from the set of real numbers). The quantization formula for a tensor entry x is then: $${\tilde{\text{x}}}_{\text{q}}=RO\mathrm{and}n\mathrm{i.e}\left(\text{clip}\left(x\mathrm{,0,}c\right)\cdot \frac{\left|u\right|}{c}\right)\times \frac{c}{\left|u\right|}$$

where x̃ _q is the quantized tensor entry.

As part of the backpropagation algorithm, the gradient g _|U| gradient formula following the quantization operation can be used, $$G_{\left|u\right|}\stackrel{\text{ISTE}}{=}{G}_{{\tilde{\text{x}}}_{\text{q}}}\cdot \left(\frac{-c}{{\left|u\right|}^2}\cdot RO\mathrm{and}n\mathrm{i.e}\left(\text{clip}\left(x\mathrm{,0,}c\right)\cdot \frac{\left|u\right|}{c}\right)+\frac{\text{clip}\left(x\mathrm{,0,}c\right)}{\left|u\right|}\right)$$

To ensure that after convergence of the learning algorithm, the number |U| of different quantization values only assumes values 2 ^b , eg with b=1,2,3,4, ...., the cardinality of the possible set of values for |U| be reduced. For example, the cardinality can first be reduced from the value set of the real numbers to the value set of the natural numbers and finally to 2 ^b , with b = 1,2,3,4,.... For this purpose, the exponent b can be calculated as Round(log ₂ (|U|)).

• • eine erste Komponente bzw. Teilfunktion, die darauf ausgerichtet ist, die Güte der Zielfunktion des neuronalen Netzes 200 zu erhöhen;
• • eine zweite Komponente bzw. Teilfunktion, die darauf ausgerichtet ist, die Komplexität des neuronalen Netzes 200 (z.B. in Bezug auf Speicher- und/oder Rechenressourcen) zu reduzieren; und/oder
• • eine dritte Komponente bzw. Teilfunktion, die darauf ausgerichtet ist, einen Informationsverlust aufgrund der Quantisierung der Tensoreinträge zu reduzieren.

As explained above, an optimization function (in particular an error function) can be used to train a neural network 200 . The optimization function can have several components in order to be able to take different aspects into account when training the neural network 200 . In particular, the optimization function can include

• • a first component or sub-function which is designed to increase the quality of the target function of the neural network 200;
• • a second component or sub-function aimed at reducing the complexity of the neural network 200 (eg in terms of memory and/or computational resources); and or
• • a third component or sub-function which is aimed at reducing information loss due to the quantization of the tensor entries.

311, und aktuelle Parameter-Quantisierungsfunktionen $b_w^l$

312 vor (für alle Schichten des neuronalen Netzes 200). Die Parameter-Tensoren W^l 310 können dabei in nicht-quantisierter Form gespeichert werden. Die Quantisierungsfunktionen 311, 312 können jeweils die Anzahl |U| von unterschiedlichen Quantisierungswerten und den Skalierungs-Wert c (individuell für jede einzelne Schicht) anzeigen. Die Parameter-Tensoren 310 und die Quantisierungsfunktionen 311, 312 können für alle L Schichten 211, 212, 213 des neuronalen Netzes 200 bereitgestellt werden. 3 FIG. 3 illustrates an example method 300 for training a neural network 200. The method 300 can be iterated to repeatedly adjust the parameter tensors and the quantization functions in the different layers. In a particular iteration of the method 300 are current parameter tensors ^Wl 310, current activation quantization functions $b_a^l$

311, and current parameter quantization functions $b_w^l$

312 (for all layers of the neural network 200). The parameter tensors W ¹ 310 can be stored in non-quantized form. The quantization functions 311, 312 can each have the number |U| of different quantization values and the scaling value c (individually for each single layer). The parameter tensors 310 and the quantization functions 311, 312 can be provided for all L layers 211, 212, 213 of the neural network 200.

The block 325 comprises steps performed sequentially for all layers 211, 212, 213 of the neural network 200, from the input layer 211, l = 1, to the output layer 213, l = L. The output 303 (also called the prediction designated) of the block 325 for a layer l is used as input or activation 301 for the subsequent layer l+1. The output 303 for the output layer l = L is used to determine the optimization function (i.e. the loss function) and thus to adapt the parameter tensors 310 and/or the quantization functions 311, 312 (in block 330).

The arithmetic operations for a layer 1 are described below. Based on the current activation quantization function 311 for layer l, a set 308 of quantization values for the activation 301 of layer l (ie for A ^l-1 ) is determined. The activation 301 is quantized in the quantization block 321 using the set 308 of quantization values, so that a quantized activation 302 for the layer l is provided.

The parameters 305 for layer 1 are extracted from the current parameter tensor 310 for the entire neural network 200 . Furthermore, based on the current parameter quantization function 312 for layer l, a set 306 of quantization values for the parameters 305 of layer l (ie for W ^l ) is determined. The parameters 305 are then quantized in the quantization block 323 using the set 306 of quantization values, so that quantized parameters 307 are provided for layer l.

The quantized activation 302 for layer l is processed (e.g., by a convolution) against the quantized parameters 307 for layer l to determine the output 303 of layer l. As already explained above, the output 303 of layer l is used as an activation 301 for the following layer l+1 (for l=1,...,L-1). For the output layer l=L, the output 303 is used to adjust the quantization functions 311, 312 and/or the parameter tensor 310 for the neural network 200.

The adaptation block 330 receives the predicted output 303 (from the neural network 200). Furthermore, the adjustment block 330 records a corresponding target output 304 . Based on the predicted output 303 and the target output 304, a value of the optimization function can be determined. Furthermore, a gradient of the optimization function can be back-propagated through the neural network 200 in order to adapt the quantization functions 311, 312 and/or the parameter tensor 310 (as described above as part of the back-propagation algorithm). In this case, the optimization function has at least one component relating to the target function of the neural network 200 to be learned and at least one component relating to the resource efficiency or the resource expenditure of the neural network 200 .

As part of gradient descent optimization (using the backpropagation algorithm), the gradient of the optimization function (i.e. the loss function) is determined. In order to make this possible, the optimization function, in particular the resource cost component, must be derivable. The following describes a hardware resource optimization function that enables gradient descent optimization to be performed efficiently and accurately.

A Gaussian process can be taught, which describes the connection between the optimization functions 311, 312 and the hardware load or the hardware efficiency or the resource expenditure. The Gaussian process can be learned using a regression method. In other words, Gaussian process regression can be used to provide a derivable component of the optimization function that describes the resource cost of the neural network 200 to be trained.

• • der Anzahl von Rechenoperationen, die in der jeweiligen Schicht 211, 212, 213 auszuführen sind; bei einer sogenannten „convolutional“, d.h. Faltungs-, Schicht ist die Anzahl von Rechenoperationen wiederum von der Anzahl von Reihen (rows) und Spalten (columns) der Aktivierung 301, sowie von der Tiefe (depth) des inneren Produkts abhängig;
• • der Anzahl |U | von unterschiedlichen Quantisierungswerten der Aktivierungs-Quantisierungsfunktion, insbesondere des Exponenten b_a zur Ermittlung der Anzahl 2^ba von unterschiedlichen Quantisierungswerten; und/oder
• • der Anzahl |U | von unterschiedlichen Quantisierungswerten der Parameter-Quantisierungsfunktion, insbesondere des Exponenten b_w zur Ermittlung der Anzahl 2^bw von unterschiedlichen Quantisierungswerten.

The resource expenditure of a layer 211, 212, 213 of the neural network 200 typically depends on the following characteristic values,

• • the number of arithmetic operations to be performed in the respective layer 211, 212, 213; in a so-called "convolutional", ie convolutional, layer, the number of arithmetic operations is in turn dependent on the number of rows (rows) and columns (columns) of the activation 301, and on the depth (depth) of the inner product;
• • the number |U | of different quantization values of the activation quantization function, in particular of the exponent b _a to determine the number 2 ^{b a} of different quantization values; and or
• • the number |U | of different quantization values of the parameter quantization function, in particular of the exponent b _w to determine the number 2 ^{b w} of different quantization values.

The covariance of the Gaussian process to be learned can be formulated as follows, $$\begin{array}{l}K\left(\rho ,\rho \text{'}\right)={\sigma }^{2}exp\left(-\frac{{\Vert \rho -\rho \text{'}\Vert }^2}{2{\mathcal{l}}^2}\right),\text{where}\rho =\left(\mathrm{right}Ow,\mathrm{i.e}eptH,cOl,b_{w,}{b}_a\right)\\ {\phi }_{HW}\sim \mathcal{G}\mathcal{P}\left(m\left(\rho \right),K\left(\rho ,\rho \text{'}\right)\right)\end{array}$$

In this case, p is a vector that describes the resource expenditure of the neural network 200 based on one or more of the aforementioned characteristic variables of the neural network 200. The amplitude σ and the scaling factor ℓ can be taught. The predicted resource cost φ _HW for a neural network 200 described by the vector p is proportional to a value provided by the Gaussian process with the mean m(ρ) and the covariance K(ρ, ρ′).

The different parameters of the Gaussian process can be learned using resource training data, the resource training data having a large number of pairs of vectors ρ and corresponding resource expenditure values.

The above-mentioned function for describing the resource requirements of a neural network 200 can be derived and can therefore be used as part of a gradient descent optimization.

4 shows a flowchart of a (possibly computer-implemented) method 400 for training a neural network 200 for a target function (eg for detecting objects based on the sensor data from one or more surroundings sensors 102 of a vehicle 100). In this case, the target function can be described by training data for training the neural network 200 (in particular by the nominal outputs 304 of the neural network 200 indicated in the training data).

The neural network 200 comprises a set of one or more layers 211, 212, 213 (in particular convolutional layers), each with a set 305 of parameters 222. Typically, the neural network 200 comprises 10 or more, or 100 or more layers 211, 212 , 213, which are arranged in series with one another, so that the output 303 of one layer serves as activation 301 of the following layer.

In one iteration of a learning algorithm, the method 400 includes the quantization 401 of one or more activations 301 of the one or more layers 211, 212, 213 using an activation quantization function 311 in order to determine one or more corresponding quantized activations 302. The activation quantization function 311 can indicate the number of quantization values to be used for each layer.

Furthermore, the method 400 includes the quantization 402 of the one or more sets 305 of parameters 222 of the one or more layers 211, 212, 213 using a parameter quantization function 312 in order to determine one or more corresponding sets 307 of quantized parameters 222. The parameter quantization function 312 can indicate the number of quantization values to be used for each layer.

The method 400 also includes determining 403, based on the one or more quantized activations 302 and based on the one or more sets 307 of quantized parameters 222, a predicted output 303 of the neural network 200. It can sequentially layer by layer based on the respective activation 302 the respective output 303 can be determined until finally the output layer 313 provides the predicted output 303 of the neural network 200 .

In addition, the method 400 includes adjusting 404 the activation quantization function, the parameter quantization function and/or the one or more sets 305 of parameters 222 based on the predicted output 303 and based on an optimization function. In this case, the optimization function can be designed to optimize the quality of the neural network 200 in relation to the target function and to reduce the resource expenditure of the neural network 200 . In this way, a neural network 200 can be provided in an efficient and reliable manner, which has an optimized compromise between objective function quality and resource expenditure.

5 shows a flow chart of a (possibly computer-implemented) method 500 for determining an estimated value of the resource expenditure of a neural network 200. The aspects described in relation to the methods 300, 400 are also applicable to the method 500 (and vice versa). In particular, method 500 may be used as part of methods 300,400.

As already explained, the neural network 200 includes a set of one or more layers 211, 212, 213, each with a set 307 of (possibly quantized) parameters 222. The neural network 200 can be designed, one or more activations 301 of a or more slices 211, 212, 213 using an activation quantization function 311 to determine one or more corresponding quantized activations 302. Furthermore, the neural network 200 can be designed to determine a predicted output 303 of the neural network 200 based on the one or more quantized activations 302 and based on the one or more sets 307 of (possibly quantized) parameters 222 .

The method 500 includes determining 501 values of one or more characteristic variables for describing the neural network 200, the activation quantization function 311 and/or a parameter quantization function 312 with which the one or more sets 307 of quantized parameters 222 were quantized .

Furthermore, the method 500 includes determining 502 the estimated value of the resource expenditure of the neural network 200 based on an estimation function and based on the values of the one or more characteristic variables. The estimator preferably includes a Gaussian process that was learned using a regression method based on resource training data. A precise estimated value for the resource expenditure of the neural network 200 can thus be determined in an efficient manner.

The present invention is not limited to the exemplary embodiments shown. In particular, it should be noted that the description and the figures are only intended to illustrate the principle of the proposed methods, devices and systems by way of example.

Claims (16)

Device for training a neural network (200) for a target function; wherein the neural network (200) comprises a set of one or more layers (211, 212, 213) each having a set (305) of parameters (222); wherein the device is set up, in an iteration of a learning algorithm, - quantizing one or more activations (301) of the one or more layers (211, 212, 213) using an activation quantization function (311) to determine one or more corresponding quantized activations (302); - quantizing the one or more sets (305) of parameters (222) of the one or more layers (211, 212, 213) using a parameter quantization function (312) to obtain one or more corresponding sets (307) of quantized parameters ( 222) to determine; - to determine a predicted output (303) of the neural network (200) based on the one or more quantized activations (302) and based on the one or more sets (307) of quantized parameters (222); and - adapt the activation quantization function, the parameter quantization function and the one or more sets (305) of parameters (222) based on the predicted output (303) and based on an optimization function. Device according to claim 1 , wherein the optimization function comprises - a first sub-function aimed at adapting the neural network (200), in particular the one or more sets (305) of parameters (222), with respect to the objective function; and - a second sub-function aimed at calculating a resource cost of the neural network (200), in particular a resource cost associated with the quantization effected by the activation quantization function (311) and by the parameter quantization function (312) of the one or more Activations (301) and the one or more sets (205) of parameters (222) associated. Device according to one of the preceding claims, wherein the device is set up - to compare the predicted output (303) with a target output (304); and - adjust the activation quantization function, the parameter quantization function and/or the one or more sets (305) of parameters (222) based on the comparison. Device according to claim 3 , wherein the device is set up - to determine a deviation of the predicted output (303) from the target output (304); - determine a gradient of the optimization function for the deviation; and - adjust the activation quantization function, the parameter quantization function and/or the one or more sets (305) of parameters (222) based on the gradient. Device according to one of the preceding claims, wherein - the activation quantization function and/or the parameter quantization function for a layer (211, 212, 213) respectively indicates a number of quantization values with which the activation (301) of this layer (211, 212, 213) or the quantity ( 205) are quantized from parameters (222) of this layer (211, 212, 213); and - the device is set up to adapt the number of quantization values in order to adapt the activation quantization function and/or the parameter quantization function. Device according to claim 5 , wherein the device is set up to - select the number of quantization values from a set of values; and - to reduce a cardinality of the set of values with an increasing number of iterations of the learning algorithm, in particular - from the set of real numbers; - optionally over a set of real numbers with a limited number of decimal places; - optionally over the set of natural numbers; - to a set of values {2 ^b } with base-two exponential values, where b is a natural number. Device according to one of Claims 5 until 6 , wherein the device is set up to determine different numbers of quantization values of the activation quantization function and/or the parameter quantization function for different layers (211, 212, 213) of the neural network (200). Device according to one of the preceding claims, wherein - the activation quantization function and/or the parameter quantization function each have a scaling value c, which defines a value range of quantization values for quantization of an activation (301) or a set (205) of parameters (222); and - the device is set up to adapt the scaling value c on the basis of the predicted output (303) and on the basis of the optimization function. Device according to one of the preceding claims, wherein the optimization function comprises a sub-function which is formed on the basis of one or more characteristic variables for describing the neural network (200), the activation quantization function (311) and/or the parameter quantization function ( 312) predict a value of a resource cost of the neural network (200). Device according to claim 9 , wherein the one or more characteristic quantities comprise, - at least one quantity relating to a number of arithmetic operations, in particular multiplications, to be carried out in the one or more layers (211, 212, 213); - at least one variable in relation to a number of different quantization values with which the one or more quantized activations (302) are represented and/or with which the one or more activations (301) are quantized by the activation quantization function (311); and/or - at least one quantity in relation to a number of different quantization values with which the one or more sets (307) of quantized parameters (222) are represented and/or with which the one or more sets (305) of parameters ( 222) can be quantized by the parameter quantization function (312). Device according to one of claims 9 until 10 , wherein - the sub-function for predicting a value of the resource cost of the neural network (200) comprises a Gaussian process; and - the Gaussian process was trained using a regression method based on resource training data. Device according to claim 11 , where - the sub-function is designed to determine an estimated value of the resource expenditure, which is proportional to $\mathcal{G}\mathcal{P}\left(m\left(\rho \right),K\left(\rho ,\rho \text{'}\right)\right)$

is; - m(ρ) a learned mean of the Gaussian process $\mathcal{G}\mathcal{P}\left(\right)$

is; - K(ρ, ρ') a learned covariance of the Gaussian process $\mathcal{G}\mathcal{P}\left(\right)$

is; and - ρ is a scalar or a vector comprising values of one or more characteristic quantities describing the neural network (200), the activation quantization function (311) and/or the parameter quantization function (312). Device according to claim 12 , where the covariance K(ρ,ρ') is given by $$K\left(\rho ,\rho \text{'}\right)={\sigma }^{2}exp\left(-\frac{{\Vert \rho -\rho \text{'}\Vert }^2}{2{\mathcal{l}}^2}\right)$$

with a learned amplitude σ and a learned scaling factor ℓ. Device according to one of the preceding claims, wherein the device is set up - to determine a predicted output (303) of the neural network (200) for a plurality of training activations (301); and - adapt the activation quantization function, the parameter quantization function and the one or more sets (305) of parameters (222) based on the plurality of predicted outputs (303) and based on the optimization function. Device according to one of the preceding claims, wherein - the neural network (200) comprises L consecutive layers (211, 212, 213), with L greater than 10, in particular greater than 100; and or - a layer (211, 212, 213) is a convolution layer arranged to cause a convolution between the respective set (307) of quantized parameters (222) and the respective quantized activation (302) to produce an output ( 303) of the respective layer (211, 212, 213); and or - the device is designed - to use an output (303) of a layer l as an activation (301) for a directly following layer l+1; and or - using an output (303) of an output layer (213) of the neural network (200) as the output (303) of the neural network (200). Method (400) for training a neural network (200) for a target function; wherein the neural network (200) comprises a set of one or more layers (211, 212, 213) each having a set (305) of parameters (222); wherein the method (400) comprises in an iteration of a learning algorithm, - quantizing (401) one or more activations (301) of the one or more layers (211, 212, 213) using an activation quantization function (311) to determine one or more corresponding quantized activations (302); - quantizing (402) the one or more sets (305) of parameters (222) of the one or more layers (211, 212, 213) using a parameter quantization function (312) to obtain one or more corresponding sets (307) of quantized determine parameters (222); - determining (403), based on the one or more quantized activations (302) and based on the one or more sets (307) of quantized parameters (222), a predicted output (303) of the neural network (200); and - adjusting (404) the activation quantization function, the parameter quantization function and the one or more sets (305) of parameters (222) based on the predicted output (303) and based on an optimization function.

Images