DE102019214546B4 - Computer-implemented method and apparatus for optimizing an artificial neural network architecture - Google Patents

Abstract

Computer-implemented method for optimizing an architecture of an artificial neural network, an architecture of the artificial neural network being specified in which arithmetic operations of the artificial neural network are combined in function layers, characterized in that depending on the architecture, a reciprocal of a number of neurons of the artificial neural network in an output layer of one of the function layers is determined (202), a factor depending on a window size of a max-pooling window in one of the function layers following the function layer for this function layer depending on the architecture -layers is determined (204), with a measure for a sensitivity of the architecture to hardware memory errors being determined as a function of a product of the reciprocal value and the factor (208), with it being checked whether the measure falls below a threshold value (210) and where a hardware description configuring the hleuniger (106) with neurons, connections between neurons, weights and/or activation functions specified by the architecture if the metric falls below the threshold (210) and otherwise determining the metric for another architecture.

Description

State of the art

The invention is based on a device and a method for optimizing an architecture of an artificial neural network.

It is desirable to provide artificial neural networks that are particularly robust to random hardware memory failures. Random hardware memory errors are, for example, bit flip errors that are triggered by electromagnetic radiation affecting a memory. However, an evaluation of an artificial neural network architecture with a large number of neurons against such errors is difficult to present.

From SCHORN, Christoph; GUNTORO, Andre; ASCHEID, Gerd: An efficient bit-flip resilience optimization method for deep neural networks. In: 2019 Design, Automation & Test in Europa Conference & Exhibition (DATE). IEEE, 2019. pp. 1507-1512 and ELSKEN, Thomas; METZEN, Jan Hendrik; HUTTER, Frank: Efficient multiobjective neural architecture search via lamarckian evolution. Version 4, pp. 1-23, URL: https://arxiv.org/pdf/1804.09081 methods for optimizing the architecture of artificial neural networks are known.

Disclosure of Invention

The device and the method according to the independent claims enable a particularly good evaluation of the robustness of the architecture of artificial neural networks with regard to random hardware memory errors. An optimization of an architecture of the artificial neural network is thus possible in a particularly efficient manner.

In a computer-implemented method for optimizing an architecture of an artificial neural network, an architecture of the artificial neural network is specified, in which arithmetic operations of the artificial neural network are combined in functional layers, with a reciprocal of a number of neurons of the artificial neural network depending on the architecture is determined in an output layer of one of the functional layers, a factor depending on a window size of a max pooling window in one of the functional layers following the functional layer being determined for this functional layer depending on the architecture, wherein a measure of a sensitivity of the architecture to hardware memory errors is determined as a function of a product of the reciprocal and the factor, it being checked whether the measure falls below a threshold value and a hardware accelerator with neurons, connections between neurons, weights and/or activation functions dictated by the architecture are configured if the metric falls below the threshold and otherwise the metric is determined for a different architecture. This measure represents a parameter for the robustness of the artificial neural network. Here, for example, unlike in typical deep learning software tools, a function layer is defined as follows: A function layer contains all computing operations that a dedicated deep neural network hardware accelerator sequentially before storing the result in an output feature map in memory for computation by the next function layer. The arithmetic operations are, for example, the partial arithmetic operations for a convolutional layer, a pooling layer, a batch normalization, or an activation function. In the example, the memory is a hardware memory. The last sub-arithmetic operation of the function layer supplies a result in the output layer. For the hardware memory, for example, a constant and evenly distributed bit flip probability for the hardware memory is taken as a basis. The bit flip probability indicates the probability that a bit in memory will change its value, e.g. in response to external influences. The measure is used for a calculation of an optimized architecture for an artificial neural network. This measure represents a parameter of the architecture that can be evaluated favorably in terms of computing time and computing resources. The parameter makes it possible to evaluate the robustness of neural networks with regard to bit flip errors and to increase it by optimizing the architecture in this regard. In order to maximize the robustness, the error sensitivity of the architecture is minimized in particular by the fact that the architecture that is implemented in the memory falls below the threshold value. This automatically generates an optimized artificial neural network.

In one aspect, the factor for the function layer is dependent on a particular arithmetic mean of the individual window sizes of max-pooling windows for a plurality of the following next to the other functional layers. This enables a particularly efficient calculation for more complex architectures.

In one aspect, an artificial neural network architecture is selected from a plurality of artificial neural network architectures depending on the metric. For example, the metric can be used in a cost function for sampling architectures for evaluation, choosing the architecture that minimizes the cost function.

An architecture for the artificial neural network is preferably determined which minimizes a cost function, the cost function being defined as a function of the measure.

Preferably, the cost function is defined as a function of a number of arithmetic operations or a number of memory transfers required for the hardware accelerator 106 to determine a result. This automatically generates a particularly fast, efficient and robust artificial neural network.

In one aspect, the architecture defines an artificial neural network for the detection or classification of objects depending on sensor data, in particular video data, data for optical distance or speed measurement or radar data, an input variable for the artificial neural network being determined depending on the sensor data, where depending on an output signal of the artificial neural network, a control signal for controlling an at least partially autonomous robot, an at least partially autonomous vehicle, an actuator, a machine, a household appliance or a home improvement device is determined.

In one aspect, the architecture defines an artificial neural network for detecting or classifying objects depending on sensor data, in particular video data or radar data, with an input variable for the artificial neural network being determined depending on sensor data, with an output signal of the artificial neural network determining a Information about a pedestrian and/or a traffic sign and/or a vehicle and/or their position relative to one another is determined.

In one aspect, the architecture defines an artificial neural network for segmenting sensor data, in particular video data, data for optical distance or speed measurement or radar data, with an input variable for the artificial neural network being determined depending on the sensor data, with depending on segmented sensor data are determined from an output signal of the artificial neural network.

In a first phase, an architecture for the artificial neural network is preferably determined for which the measure falls below the threshold value, with training data being provided in a second phase and this architecture being trained as a function of the training data, and with the sensor data being used in a third phase processed in the artificial neural network trained in this way.

A device comprises a processor and a memory for an artificial neural network, which are designed to carry out the method.

• 1 eine schematische Darstellung von Teilen einer Recheneinrichtung,
• 2 Schritte in einem Verfahren für die Recheneinrichtung,
• 3 ein Berechnungsschema für das Verfahren.

Further advantageous embodiments result from the following description and the drawing. In the drawing shows:

• 1 a schematic representation of parts of a computing device,
• 2 steps in a method for the computing device,
• 3 a calculation scheme for the procedure.

In 1 a device 100 is shown schematically.

The apparatus 100 comprises a processor 102, a memory 104, an artificial neural network hardware accelerator 106, a sensor 108 and an output device 110. The artificial neural network comprises a plurality of neurons. The hardware accelerator 106 can be configured with neurons, connections between neurons, weights, and/or activation functions according to an artificial neural network architecture.

The memory 104 can be used for storing the configurations of neural networks as well as for writing/reading input and output tensors of the artificial neural network. The same memory 104 need not necessarily be used for storing neural network configurations and writing/reading input and output tensors of the artificial neural network. Different memories can be used for this.

In the example, the hardware accelerator 106 is configured to write results from an output layer of the artificial neural network to the memory 104 . Corresponding data may flow from hardware accelerator 106 to memory 104 via processor 102 . In 1 arrows are shown representing data flows for this and other exemplary data flows. The hardware accelerator 106 is designed to output the result to the output device 110 via the processor 102 .

Memory access for writing/reading input and output tensors is not necessarily done through the processor 102. A direct access from the hardware accelerator 106 to the memory 104, a so-called “direct memory access”, can be provided. In this case, for example, a central data bus is provided, to which the processor 102, the memory 104, the hardware accelerator 106, the input device 108 and the output device 110 are connected and which regulates the data flow.

The sensor 108 is designed to deliver sensor data. The output device 110 is designed to output an output signal.

The device 100 is designed to carry out the method described below. The method evaluates the robustness of architectures for the artificial neural network and configures the artificial neural network in memory 104 according to a particularly robust architecture determined in the method.

In particular, robustness against random hardware memory errors, i. H. especially bit flip errors that can occur in the memory 104 evaluated. These errors can be triggered, for example, by electromagnetic radiation acting on a memory 104 . These errors occur, for example, with a bit flip probability that is assumed to be constant and uniformly distributed.

In the method, a measure f _ASI (N) for a sensitivity of an architecture of the artificial neural network with N functional layers is determined. A low sensitivity, in particular to the random hardware memory errors, in particular to the bit flip errors that are triggered by the electromagnetic radiation acting on the memory 104, means a high level of robustness. In this context, sensitivity means that an output of a neural network with bit-flip errors deviates significantly from an unaffected variant of the network. A significant deviation exists, for example, if other conclusions are drawn from the output, such as changed classification results.

In the example, the artificial neural network comprises at least one function layer l, which is defined by a large number of arithmetic operations of the artificial neural network, which can be carried out one after the other in particular by a hardware accelerator 106 to determine a result in an output layer of the function layer l are. The result can be defined as an output tensor or an output feature map. The result is stored in memory 104 when using the artificial neural network. A function layer following this function layer 1 can load the result from the memory 104 for subsequent processing. A large number of functional layers 1 can be provided.

In the example, the method comprises three phases in which the artificial neural network is first determined, then trained and then used. These three phases can be performed in the order described below or independently.

In a first phase, in a step 202 for the function layer l, a reciprocal of a number n ^(l) of neurons of the artificial neural network in the output layer of the function layer l is determined. If a large number of functional layers 1 are provided, an individual reciprocal value is determined for each of the layers.

In the first phase, in a step 204 for the function layer l, a factor λ ^(l) is determined as a function of a window size of a max pooling window for the result in the subsequent function layer. This factor λ ^(l) is also referred to below as the max pooling factor λ ^(l) .

If the result in the output layer of function layer l is determined for subsequent processing in a multiplicity of function layers following function layer l, the max pooling factor λ ^(l) is determined as a function of a particularly arithmetic mean value that is defined via individual window sizes of the max-pooling window for the majority of the subsequent functional layers.

If a large number of functional layers l are provided, an individual max pooling factor λ ^(l) is determined for each of the layers.

In the first phase, in an optional step 206 for the functional layer l, a further factor ς ^(l) can be determined, which is defined as a function of a number of functional layers, the result of which is in the same subsequent functional layer, in particular by component-wise addition of Input tensors read from memory 104 are merged. In the example, the input tensors correspond to the output tensors that were previously saved.

If a large number of functional layers l are provided, an individual factor ς ^(l) is determined for each of the layers. For function layers l that are not included in a component-wise addition, i.e. if none of the subsequent function layers carries out a component-wise addition, this factor is ς ^(l) = 1.

In the first phase, in a step 208, the measure f _ASI (N) for the sensitivity is determined as a function of a product of the reciprocal and the max pooling factor λ ⁽¹⁾ .

When a plurality of the functional layers l are provided, the measure f _ASI (N) is determined depending on a sum of a plurality of the products determined for individual functional layers l.

For a number N of function layers l, the measure f _ASI (N) is determined, for example, as: $${ƒ}_{ASI}\left(N\right)={\mathrm{\Sigma }}_{i\in N}\frac{1}{n^l}{\lambda }^{\left(l\right)}{\varsigma }^{\left(l\right)}$$

In a step 210, an architecture for the artificial neural network is determined from a plurality of architectures for the artificial neural network as a function of the measure f _ASI (N).

For example, an individual measure f _ASI N) is determined for each architecture of a large number of architectures for the artificial neural network. The memory 104 is configured in this example with neurons, connections between neurons, weights and/or activation functions dictated by the architecture with the smallest of the individual metrics.

The hardware accelerator 106 is configured in step 210 with neurons, connections between neurons, weights and/or activation functions according to an architecture for which the measure f _ASI (N) falls below a threshold value. The threshold value is, for example, in the range between 0 and 1 and is, for example, 0.1 or 0.2 or 0.5. For a comparison of one architecture with another architecture, the threshold value can also be defined by the measure f _ASI (N) determined for one of the two architectures to be compared. If the measure does not fall below the threshold value, these steps can be repeated, for example, until a certain number of architectures have been determined and evaluated with the measure. Provided that the size is not undercut, the architecture with the smallest size can be used. The architectures define a topology for an artificial neural network that includes neurons, connections between neurons, weights, and/or activation functions.

The measure f _ASI (N) is used, for example, as a cost function in a calculation scheme for the architecture of artificial neural networks. An example of such a calculation scheme is in the previously published DE 102017219282 A1 specified. In particular, an evaluation unit uses the measure f _ASI (N) as a cost function to evaluate the performance of the artificial neural network works or by further developments of the artificial neural network. In this, the artificial neural network is defined by the parameter p.

The artificial neural network or the developments of the artificial neural network are optimized by varying the respective parameters p. In this example, the resulting dimensions f _ASI (N) mean that the artificial neural network is retained or that one of the developments of the artificial neural network is selected whose dimension f _ASI (N) is minimal compared to the other dimensions f _ASI (N). . For this purpose, a topology of a respective neural network N, for example a graph that defines the artificial neural network N and the parameterization of its layers as parameter p, can be transferred to a function for calculating the measure f _ASI (N). As a return value, this function returns a numeric value that indicates the architecture sensitivity. In this example, high sensitivity corresponds to low robustness to bit flip errors. This value is used in an example for sampling parent architectures and child architectures and for calculating a Pareto front. In addition to the measure f _ASI (N), further cost functions which, for example, estimate an efficiency of the neural network for a given hardware accelerator 106 can be used. This further cost function takes into account, for example, a number of arithmetic operations or a number of memory transfers required for determining the result by this hardware accelerator 106.

The hardware accelerator 106 may be or include a graphical processing unit, a digital signal processing unit, a programmable FPGA, or a dedicated computing device.

In a second phase, training data is provided in a step 212 and this architecture is trained depending on the training data.

In a first aspect, the architecture defines, for example, an artificial neural network for detecting objects depending on sensor data, in particular video data, data for optical distance or speed measurement, or radar data. In this case, in the example, the sensor data is specified as training data, an input variable for the artificial neural network is determined as a function of the sensor data, and the artificial neural network is trained to recognize the objects. Instead of recognizing objects, a classification of objects can also be provided.

In a second aspect, the architecture defines, for example, an artificial neural network for segmenting sensor data, in particular video data, data for optical distance or speed measurement, or radar data. In this case, in the example, the sensor data is specified as training data, an input variable for the artificial neural network is determined as a function of the sensor data, and the artificial neural network is trained to segment the sensor data.

In a third phase, in a step 214, the sensor data are processed with the artificial neural network trained in this way. In the example, an input variable for the artificial neural network is determined depending on the sensor data. In the example, the output signal is determined depending on the input variables.

In the first aspect, it can be provided that a control signal for controlling an at least partially autonomous robot, an at least partially autonomous vehicle, an actuator, a machine, a household appliance or a do-it-yourself device is determined as an output signal depending on an output signal of the artificial neural network. In the first aspect, it can be provided that information about a pedestrian and/or a road sign and/or a vehicle and/or their position relative to one another is determined as an output signal depending on an output signal of the artificial neural network.

In the second aspect, segmented sensor data is determined as an output signal depending on an output signal of the artificial neural network.

In 3 a calculation scheme is shown as an example for an artificial neural network 300 with a network input at a first function layer 302 with n=1000 neurons in an output layer of the first function layer 302. The first function layer 302 contains the arithmetic operations 2D convolution and activation, for example. A layer index l=1 is assigned to the first functional layer 302 .

A hidden second function layer 304 is arranged immediately after the first function layer 302 . In this context, arranged immediately following means that an output tensor of the first function layer 302, which is stored in the memory 104, for example, defines the input tensor of the hidden second function layer 304, which is read from the memory 104, for example. The hidden second function layer 304 includes, for example, n=1000 neurons in an output layer of the second function layer 304. The second function layer 304 contains, for example, the arithmetic operations 2D convolution and activation. A layer index l=2 is assigned to the hidden second function layer 304 .

A hidden third function layer 306 is arranged immediately after the first function layer 302 . In this context, arranged immediately following means that the output tensor of the first function layer 302 defines the input tensor of the hidden third function layer 306, which is read from the memory 104, for example. The hidden third function layer 306 includes, for example, n=100 neurons in an output layer of the third function layer 306. The hidden third function layer is designed for 2×2 max pooling and for example 2D convolution and activation. A layer index l=3 is assigned to the hidden third function layer 304 .

A hidden fourth function layer 308 is arranged immediately after the hidden second function layer 304 , the input tensor of which is defined by an output tensor of the hidden second function layer 304 . The hidden fourth function layer 308 includes, for example, n=100 neurons in an output layer of the fourth function layer 308. The hidden fourth function layer 308 is designed for 2×2 max pooling and for example 2D convolution and activation. A layer index l=4 is assigned to the hidden fourth function layer 308 .

A hidden fifth function layer 310 is arranged immediately after the hidden third function layer 306 , the input tensor of which is defined by an output tensor of the hidden third function layer 306 . The hidden fifth function layer 310 includes, for example, n=100 neurons in an output layer of the fifth function layer 310. The fifth function layer 310 contains, for example, the arithmetic operations 2D convolution and activation. A layer index l=5 is assigned to the hidden fifth function layer 310 .

The hidden fourth function layer 308 and the hidden fifth layer 310 is followed directly by a hidden addition layer 312 . In the hidden addition layer 312 , a component-wise addition of input tensors is carried out, which are read from the memory 104 , for example, and are defined as a function of the output tensors of the hidden fourth function layer 308 and the hidden fifth function layer 310 . In the example, these input tensors are added to form a tensor.

A hidden sixth function layer 314 is arranged immediately after the hidden addition layer 312 , the input tensor of which is defined by an output tensor of the hidden addition layer 312 . The hidden sixth function layer 314 includes, for example, n=100 neurons in an output layer of the sixth function layer 314. The hidden sixth function layer 314 is designed for 4×4 max pooling and for example 2D convolution and activation. A layer index l=6 is assigned to the hidden sixth function layer 314 .

A seventh function layer 316 is arranged immediately after the hidden sixth function layer 314 , the input tensor of which is defined by an output tensor of the hidden sixth function layer 314 . The seventh functional layer 316 includes, for example, n=10 neurons in an output layer of the seventh functional layer 316. The seventh functional layer 316 contains, for example, the arithmetic operations fully connected and activation. The seventh functional layer 316 output layer provides a network output. A layer index l=7 is assigned to the seventh functional layer 316 .

In general, the function layers with the layer indices 1-7 are each function layers that can contain several arithmetic operations, eg 2D convolution, activation, max pooling, which are calculated by the hardware accelerator 106 in one computing step of the hardware accelerator 106 . In this example, hardware accelerator 106 must provide for each of the in 3 function layers shown, a result can be written into the memory 104. Memory 104 is error prone. In a function layer that is arranged downstream of another function layer, the value from the Memory 104 to be reloaded. Thus, each arrow in the figure is associated with a load from memory 104. The function layers cannot be calculated together by the hardware accelerator 106 due to these loading processes. However, a layer could also contain further arithmetic operations if the implementation of the hardware accelerator 106 allows this without intermediate storage in a faulty memory 104.

den Max-Pooling Faktor λ^(l), und den weiteren Faktor ς^(l) sowie das daraus berechnete jeweilige Produkt $\frac{1}{n^{\left(l\right)}}{\lambda }^{\left(l\right)}{\varsigma }^{\left(l\right)}$

an, die sich für diese Architektur ergeben. Die Additions-Schicht 312 geht in diesem Beispiel nicht direkt in die Berechnung des Maßes f_ASI(N) ein. Der Einfluss der Additions-Schicht 312 wird durch weitere Faktoren in den Schichten mit Layer Index l = 4 und l = 5 modelliert. Dadurch geht die Additions-Schicht 312 indirekt das Maß f_ASI(N) ein. I $\frac{1}{n^{\left(l\right)}}$

λ^(l) ζ^(l) Produkt 1 0,001 2,5 1 0,0025 2 0,001 4 1 0,004 3 0,01 1 1 0,01 4 0,01 16 2 0,32 5 0,01 16 2 0,32 6 0,01 1 1 0,01 7 0,1 1 1 0,1 The following table gives the values for the reciprocal for each of the layers I individually $\frac{1}{n^{\left(l\right)}},$

the max-pooling factor λ ^(l) , and the further factor ς ^(l) as well as the respective product calculated from it $\frac{1}{n^{\left(l\right)}}{\lambda }^{\left(l\right)}{\varsigma }^{\left(l\right)}$

that arise for this architecture. In this example, the addition layer 312 is not directly included in the calculation of the measure f _ASI (N). The influence of the addition layer 312 is modeled by further factors in the layers with layer index l=4 and l=5. As a result, the addition layer 312 indirectly enters into the measure f _ASI (N). I $\frac{1}{n^{\left(l\right)}}$

λ ^(l) ζ ^(l) product 1 0.001 2.5 1 0.0025 2 0.001 4 1 0.004 3 0.01 1 1 0.01 4 0.01 16 2 0.32 5 0.01 16 2 0.32 6 0.01 1 1 0.01 7 0.1 1 1 0.1

In this example, the resulting value of the dimension f _ASI (N) is f _ASI (N) = 0.7665.

The artificial neural network is a deep neural network running on the hardware accelerator 106, for example. The artificial neural network is designed, for example, to process sensor data. For example, the network input is determined depending on the sensor data. The artificial neural network determined in this way is particularly robust against hardware errors and is particularly suitable for use in safety-critical applications, for example in automobiles, in which very high robustness is to be achieved with little or limited hardware resources. Hardware resources are, for example, power consumption, latency or memory bandwidth for processing data through the artificial neural network. Applications in medical or industrial environments in which radiation is used can also be designed to be very robust. Other applications concern the use for robots, machine tools, production machines or control systems.

Claims (12)

Computer-implemented method for optimizing an architecture of an artificial neural network, with an architecture of the artificial neural network being specified in which arithmetic operations of the artificial neural network are combined in function layers, characterized in that depending on the architecture, a reciprocal of a number of neurons of the artificial neural network in an output layer of one of the function layers is determined (202), a factor depending on a window size of a max-pooling window in one of the function layers following the function layer for this function layer depending on the architecture -layers is determined (204), with a measure for a sensitivity of the architecture to hardware memory errors being determined as a function of a product of the reciprocal value and the factor (208), with it being checked whether the measure falls below a threshold value (210) and where a hardware bes accelerating (106) with neurons, connections between neurons, weights and/or activation functions specified by the architecture if the metric falls below the threshold (210) and otherwise determining the metric for another architecture. procedure after claim 1 , characterized in that the factor for the functional layer is determined (204) as a function of an in particular arithmetic mean of the individual window sizes of max-pooling windows for a plurality of the subsequent other functional layers. Method according to one of the preceding claims, characterized in that depending on the measure an architecture for the artificial neural network is selected from a plurality of architectures for the artificial neural network (210). Method according to one of the preceding claims, characterized in that an architecture for the artificial neural network is determined which minimizes a cost function, the cost function being defined as a function of the measure. procedure after claim 4 , characterized in that the cost function is defined as a function of a number of arithmetic operations or a number of memory transfers required for a determination of a result by the hardware accelerator (106). Method according to one of the preceding claims, characterized in that the architecture defines an artificial neural network for the detection or classification of objects depending on sensor data, in particular video data, data for optical distance or speed measurement or radar data, an input variable depending on the sensor data is determined for the artificial neural network, a control signal for controlling an at least partially autonomous robot, an at least partially autonomous vehicle, an actuator, a machine, a household appliance or a DIY appliance being determined as a function of an output signal of the artificial neural network (214). Procedure according to one of Claims 1 until 5 , characterized in that the architecture defines an artificial neural network for detecting or classifying objects depending on sensor data, in particular video data or radar data, an input variable for the artificial neural network being determined depending on sensor data, and depending on an output signal of the artificial neural Network information about a pedestrian and / or a traffic sign and / or a vehicle and / or their position relative to each other is determined (214). Procedure according to one of Claims 1 until 5 , characterized in that the architecture defines an artificial neural network for the segmentation of sensor data, in particular video data, data for optical distance or speed measurement or radar data, with an input variable for the artificial neural network being determined as a function of the sensor data, with dependent sensor data segmented from an output of the artificial neural network is determined (214). Procedure according to one of Claims 6 until 8th , characterized in that in a first phase (202, 204, 206, 208, 210) an architecture for the artificial neural network is determined for which the measure falls below the threshold value, with training data being provided in a second phase (212) and this architecture is trained depending on the training data, and in a third phase (214) the sensor data are processed with the artificial neural network trained in this way. Device (100) for optimizing an architecture of an artificial neural network, characterized in that the device (100) comprises a processor (102) and a memory (104) for the artificial neural network, which are designed using the method according to one of Claims 1 until 9 to execute. Computer program, characterized in that the computer program comprises computer-readable instructions, when executed on a computer, a method according to one of Claims 1 until 9 expires. Computer program product, characterized by a machine-readable storage medium on which the computer program claim 11 is saved.

Images