DE102019215255A1 - Device and method for processing data from a neural network - Google Patents

Abstract

Device (200) and method (100) for processing, in particular non-normalized, multidimensional, data of a neural network, in particular deep neural network, in particular for detecting objects in an input image, the data for a plurality of positions in the input image in each case at least comprise a first classification value, wherein a classification value quantifies a presence of a class, the method comprising the following steps: evaluating (102) the data as a function of a threshold value, wherein a first classification value for a respective position in the input image, which is either below or above of the threshold value is discarded (104a) and a first classification value for a respective position in the input image, which is either above or below the threshold value, is not discarded (104b).

Description

State of the art

The disclosure relates to a computer-implemented method for processing, in particular non-normalized, multidimensional, data of a neural network, in particular deep neural network.

The disclosure also relates to a device for processing, in particular non-normalized, multidimensional, data of a neural network, in particular deep neural network.

In the field of image processing, in particular object detection, neural networks, in particular folding neural networks, are often used. Convolutional Neural Network. Basically, the structure of such a network consists of several folding layers. Convolutional Layer.

For object detection, such a network is used to make a decision about the presence of classes, in particular target object classes, for a large number of positions in an input image. In this way, a large number, for example up to 10 ^7, decisions per input image are made. On the basis of these decisions, a final network output of the neural network, also referred to as prediction, can then be calculated.

In a so-called bounding box method, the prediction for an object is usually processed in such a way that a so-called bounding box, that is to say a box surrounding the object, is calculated for a detected object. The coordinates of the bounding box correspond to the position of the object in the input image. At least one probability value of an object class is output for the bounding box.

In the so-called semantic segmentation, pixel-by-pixel or super-pixel-by-pixel classes are assigned to pixels of the input image. In this context, superpixel by pixel is understood to mean a plurality of combined pixels. A pixel has a certain position in the input image.

Even smaller networks of this type can already contain several million parameters and require several billion arithmetic operations for a single execution. In particular, when neural networks are to be used in embedded systems, both the required memory bandwidth and the number of computing operations required are often limiting factors.

Conventional compression methods are often not suitable for reducing the required memory bandwidth due to the characteristic frequency distribution of the final network output of a neural network.

It is desirable to provide a method with which both the number of required arithmetic operations and a required memory bandwidth can be reduced.

Disclosure of the invention

Preferred embodiments relate to a computer-implemented method for processing, in particular non-normalized, multidimensional, data of a neural network, in particular deep neural network, in particular for detecting objects in an input image, the data for a plurality of positions in the input image in each case at least one first classification value, wherein a classification value quantifies a presence of a class, wherein the method comprises the following steps: evaluating the data as a function of a threshold value, wherein a first classification value for a respective position in the input image, which is either below or above the threshold value, is discarded and a first classification value for a respective position in the input image, which is either above or below the threshold value, is not discarded.

A first classification value is, for example, the non-normalized result of a filter, in particular a convolutional layer, of the neural network. A filter that is trained to quantify the presence of a class is also referred to below as a class filter. It is therefore proposed to evaluate the non-normalized results of the class filters and to discard the results of the class filters as a function of a threshold value.

In further preferred embodiments it is provided that the threshold value is zero, and that a first classification value for a respective position in the input image, which is below the threshold value, is discarded and a first classification value for a respective position in the input image, which is above the threshold value is not discarded. It is therefore proposed to discard negative classification values and not to discard positive classification values.

In further preferred embodiments it is provided that discarding a first classification value for a respective position in the input image further comprises: setting the first classification value to a fixed value, especially zero. The fixed value is preferably a freely definable value. The fixed value is preferably zero. Then a compression method such as run length coding can be applied to the classification values. Since the non-normalized, multi-dimensional, data of the neural network after setting the first classification values to the fixed value, in particular zero, predominantly comprise this fixed value, high compression rates, in particular of 10 ³ -10 ⁴ , can be achieved.

In further preferred embodiments it is provided that the first classification value is the non-normalized result of a class filter of the neural network, in particular for a background class, for a respective position in the input image, the discarding of a first classification value for a respective position in the input image being the discarding of the Includes the result of the class filter.

In further preferred embodiments it is provided that the data for the respective position in the input image include at least one further classification value and / or at least one value for an additional attribute, the further classification value including the non-normalized result of a class filter for an object class, in particular a target object class, wherein the method further comprises: discarding the at least one further classification value and / or the at least one value for an additional attribute for a respective position depending on whether the first classification value for the respective position is discarded. A value for an additional attribute includes, for example, a value for a relative position.

In further preferred embodiments it is provided that discarding the at least one further classification value further comprises: setting the further classification value and / or the value for an additional attribute to a fixed value, in particular zero. Then a compression method such as run length coding can be applied to the classification values. Since the non-normalized, multidimensional, data of the neural network after setting the first and further classification values and / or the values for an additional attribute to a fixed value, in particular zero, predominantly comprise this fixed value, high compression rates, in particular from 10 ^{3 to} 10 ⁴ , reachable.

In further preferred embodiments it is provided that the method further comprises: processing the non-discarded classification values, in particular forwarding the non-discarded classification values and / or applying an activation function, in particular Softmax activation function, to the non-discarded classification values. By applying an activation function, a final network output of the neural network, also referred to as prediction, can then be calculated using the non-rejected classification values, in particular to predict whether and / or with what probability an object in a certain class is at a certain position in the Input image is located.

Further preferred embodiments relate to a device for processing, in particular non-normalized, multidimensional, data of a neural network, in particular deep neural network, in particular for detecting objects in an input image, the data for a plurality of positions in the input image each at least one first classification value, wherein the device is designed to carry out the method according to the embodiments.

In further preferred embodiments it is provided that the device comprises a computing device, in particular a processor, and a memory for at least one artificial neural network, which are designed to carry out the method according to the claims.

Further preferred embodiments relate to a system for detecting objects in an input image, comprising a device for processing, in particular non-normalized, multidimensional, data of a neural network according to the embodiments, the system further comprising a computing device for applying an activation function, in particular Softmax- Activation function, in particular for calculating a prediction of the neural network, and the device is designed to forward the non-discarded classification values to the computing device and / or to a memory device assigned to the computing device.

Further preferred embodiments relate to a computer program, the computer program comprising computer-readable instructions which, when executed by a computer, run the method according to the embodiments.

Further preferred embodiments relate to a computer program product, the computer program product comprising a memory on which a computer program according to the embodiments is stored.

Further preferred embodiments relate to a use of the Method according to the embodiments, and / or a neural network according to the embodiments and / or a device according to the embodiments and / or a system according to the embodiments, and / or a computer program according to the embodiments and / or a computer program product according to the embodiments at least partially autonomous movement of a vehicle, with an input image being captured by a sensor system, in particular a camera, radar sensor or lidar sensor, of the vehicle, with a method according to the embodiments being carried out for the input image for detecting objects, with at least one control for the vehicle is intended, in particular for automated braking, steering or acceleration of the vehicle.

Further preferred embodiments relate to a use of the method according to the embodiments, and / or a neural network according to the embodiments and / or a device according to the embodiments and / or a system according to the embodiments, and / or a computer program according to the embodiments and / or or a computer program product according to the embodiments for moving a robot system or parts thereof, with an input image being captured by a sensor system, in particular a camera, of the robot system, with a method according to the embodiments being carried out for the input image for detecting objects, depending on the result of the Object detection at least one control of the robot system, in particular for interaction with objects in the vicinity of the robot system, is determined.

• 1 Schritte eines herkömmlichen Verfahrens zur Objektdetektion;
• 2a eine typische Häufigkeitsverteilung der Ergebnisse eines Convolutional Layer eines neuronalen Netzes zur Objektdetektion;
• 2b eine typische Häufigkeitsverteilung von unnormierten Daten umfassend einen ersten und einen weiteren Klassifizierungswert;
• 2c eine typische Häufigkeitsverteilung von unnormierten Daten umfassend den ersten Klassifizierungswert;
• 2d eine typische Häufigkeitsverteilung von unnormierten Daten umfassend den weiteren Klassifizierungswert;
• 3 Schritte eines Verfahrens zum Verarbeiten von Daten;
• 4 eine schematische Darstellung einer Vorrichtung zum Verarbeiten von Daten, und
• 5 eine schematische Darstellung eines Systems zum Verarbeiten von Daten.

Further advantageous embodiments emerge from the following description and the drawing. In the drawing shows

• 1 Steps of a conventional method for object detection;
• 2a a typical frequency distribution of the results of a convolutional layer of a neural network for object detection;
• 2 B a typical frequency distribution of unnormalized data comprising a first and a further classification value;
• 2c a typical frequency distribution of unnormalized data including the first classification value;
• 2d a typical frequency distribution of unnormalized data including the further classification value;
• 3 Steps of a method for processing data;
• 4th a schematic representation of a device for processing data, and
• 5 a schematic representation of a system for processing data.

1 shows schematically steps of a known method for object detection. Usually a so-called folding neural network is used for this. Convolutional Neural Network. The structure of such a network usually comprises several convolutional layers. Filters of the convolutional layer are trained, for example, to quantify the presence of a class. Such filters are also referred to below as class filters. In one step 10 a decision about the presence of classes, in particular a background class and / or a target object class, is made using class filters for a large number of positions in an input image. The results of the class filters are also referred to below as classification values.

Then in one step 12th at each of the positions via the results of the class filters, also as, non-normalized, multidimensional data. Raw scores, referred to as the softmax function, used to determine a probability with which an object of a certain class is located at a given position. By using the Softmax function, the raw scores are normalized to the interval [0, 1], so that the so-called score vector is created for each of the positions. The score vector usually has an entry for each target object class and an entry for the background class. Then in a further step 14th by so-called score thresholding, the score vectors are filtered out in which an entry of the score vector for a target object class is greater than a predefined threshold.

Further steps for post-processing include, for example, the calculation of object boxes and the application of further standard methods, for example non-maximal suppression, to generate the final object boxes. These post-processing steps are exemplified in step 16 summarized.

Most computing devices for neural networks, particularly hardware accelerators, are not capable of performing the steps 12th to 16 to execute. For this reason, all non-normalized data, including the classification values, then have to be transferred to a further storage device, in order then to be transferred from a further one suitable computing device to be further processed.

The transfer of all data and the use of the post-processing steps mentioned require both a high memory bandwidth and a large number of required arithmetic operations.

2 B shows a typical frequency distribution of unnormalized data comprising a first and a further classification value. The first classification value is, for example, the result of a class filter for the background class. The further classification value is, for example, the result of a class filter for the pedestrian target object class.

Methods for reducing the memory bandwidth, for example based on lossless or also lossy compression, for example run length coding, are known. Such approaches can be applied to the results of a convolutional layer, for example. 2a shows a typical frequency distribution of the results of a convolutional layer of a neural network. Such approaches work for the non-normalized data of the neural network due to the frequency distribution of the numerical values of the classification values, cf. 2 B , Not.

3 shows a computer implemented method 100 for processing, in particular non-normalized, multi-dimensional, data of a neural network, in particular deep neural network, in particular for detecting objects in an input image, the data for a plurality of positions in the input image each comprising at least one first classification value, the method being the includes the following steps: Evaluate 102 of the data as a function of a threshold value, a first classification value for a respective position in the input image, which is either below or above the threshold value, being discarded, 104a, and a first classification value for a respective position in the input image, which is either above or below of the threshold is not discarded, 104b.

The neural network works, for example, according to the so-called bounding box method, with a so-called bounding box, that is to say a box surrounding the object, being calculated in the event that an object is detected. The coordinates of the bounding box correspond to the position of the object in the input image. At least one probability value of an object class is output for the bounding box.

The neural network can also work according to the so-called semantic segmentation method, according to which classes are assigned pixel-by-pixel or super-pixel-by-pixel to pixels of the input image. In this context, superpixel by pixel is understood to mean a plurality of combined pixels. A pixel has a certain position in the input image.

In the process 100 so becomes an evaluation 102 the non-standardized, multi-dimensional data, the raw scores of the neural network, based on a threshold value, so-called score thresholding.

In further embodiments, the first classification value is the non-normalized result of a class filter of the neural network, in particular for a background class, for a respective position in the input image, the discarding 104a of a first classification value for a respective position in the input image comprises discarding the result of the class filter.

In the case of a first classification value, which is the result of a class filter of the background class and which is below or above a threshold value, it is assumed that there is a background and therefore no target object instance at this position in the input image. The classification values of the background class therefore already represent a valid decision limit when considered on their own. A combination with further classification values of other class filters, as is done, for example, when using the Softmax function, is not necessary. Out 2c and 2d it can be seen that the non-normalized data of the class filter of the background class and the non-normalized data of the class filter of a target object class, for example pedestrians, are not independent.

In particular, the threshold value can be zero. In this case, it can prove to be advantageous that a first classification value for a respective position in the input image which is below the threshold value is discarded, 104a, and a first classification value for a respective position in the input image which is above the threshold value , is not discarded, 104b.

In this aspect it is provided that the first classification values, i.e. the results of the class filter of the background class, are calibrated in such a way that the value zero defines the decision limit from which it can be assumed that at a position with a classification value that is below the threshold value , i.e. negative, a background at this position in the input image and therefore not a target object instance is present. The classification values are calibrated, for example, with the help of the bias in the convolutional filter of the background class.

It can further be provided that the data for the respective position in the input image include at least one further classification value and / or at least one value for an additional attribute, the further classification value including the non-standardized result of a class filter for an object class, in particular a target object class The method further comprises: discarding the at least one further classification value and / or the at least one value for an additional attribute for a respective position as a function of whether the first classification value for the respective position is discarded. Specifically, it is therefore provided that all results of the filters for a position are discarded as a function of the first classification value, in particular the result of the class filter of the background class.

In a further aspect it is provided that the non-discarded classification values in one step 106 are processed, in particular by forwarding the non-discarded classification values and / or by applying an activation function, in particular Softmax activation function, to the non-discarded classification values. So only the classification values that have not been discarded are passed on and / or processed further. By applying the activation function, the prediction of the neural network can then be calculated on the basis of the non-rejected classification values, in particular to predict whether and / or with what probability an object in a certain class is at a certain position in the input image. By applying the activation function exclusively to non-rejected classification values and thus only to a part of the classification values, the arithmetic operations required to calculate a prediction are reduced.

In a further aspect, it can be provided that when the non-discarded classification values are forwarded, the original position of the non-discarded classification values is also forwarded. This is particularly advantageous in order to determine the position of the classification values in the input image. This means that instead of transmitting classification values for all positions, classification values and position are transmitted for a significantly smaller number of positions.

In a further aspect it can be provided that the discarding 104a a first classification value for a respective position in the input image further comprises: setting the first classification value to a fixed value, in particular zero. In this context, it can advantageously be provided that discarding the at least one further classification value and / or the at least one value for an additional attribute further comprises: setting the further classification value and / or the at least one value for an additional attribute to a fixed value, in particular zero.

Specifically, it is provided that all classification values and possibly further values for additional attributes for a position are set to a fixed value, in particular zero, as a function of the first classification value, in particular the result of the class filter of the background class. Then a compression method such as run length coding can be applied to the classification values. Since the non-normalized, multi-dimensional, data of the neural network after setting the classification values and / or the further values for additional attributes to a fixed value, in particular zero, predominantly comprise this fixed value, high compression rates, in particular of 10 ^{3 -10 4} , can be achieved.

The procedure described 100 can for example from a device 200 for processing, in particular non-normalized, multi-dimensional, data of a neural network, in particular deep neural network, in particular for detecting objects in an input image, the data for a plurality of positions in the input image each including at least one first classification value, cf. . 4th .

The device 200 comprises a computing device 210 , in particular a hardware accelerator, and a storage device 220 for a neural network.

Another aspect relates to a system 300 for detecting objects in an input image, comprising a device 200 and a computing device 310 for applying an activation function, in particular Softmax activation function, in particular for calculating a prediction of the neural network. The device 200 is for forwarding the non-discarded classification values to the computing device 310 and / or to one of the computing devices 310 associated storage device 320 educated. Data lines 330 connect these facilities in the example, cf. 5 .

In the event that the computing device 210 for the neural network is not suitable for the step 106 to execute, it proves to be advantageous to send the non-discarded classification values to the computing device 310 and / or to one of the computing devices 310 associated storage device 320 forward.

The procedure described 100 , the device described 200 and the system described 300 can be used, for example, for object detection, especially person detection, for example in the surveillance area, in robotics or in the automotive sector.

Further preferred embodiments relate to a use of the method 100 according to the embodiments and / or a device 200 according to the embodiments and / or a system 300 according to the embodiments, and / or a computer program according to the embodiments and / or a computer program product according to the embodiments for at least partially autonomous movement of a vehicle, wherein an input image is captured by a sensor system, in particular a camera, radar sensor or lidar sensor, of the vehicle, with Input image for detecting objects a method 100 is carried out according to the embodiments, at least one control for the vehicle, in particular for automated braking, steering or acceleration of the vehicle being determined as a function of the result of the object detection.

Further preferred embodiments relate to a use of the method 100 according to the embodiments and / or a device 200 according to the embodiments and / or a system 300 according to the embodiments, and / or a computer program according to the embodiments and / or a computer program product according to the embodiments for moving a robot system or parts thereof, wherein an input image is captured by a sensor system, in particular a camera, of the robot system, with the input image for detecting Objects a procedure 100 is carried out according to the embodiments, at least one control of the robot system being determined as a function of the result of the object detection.

Claims (13)

Computer-implemented method (100) for processing, in particular unnormalized, multidimensional, data of a neural network, in particular deep neural network, in particular for detecting objects in an input image, the data each comprising at least one first classification value for a plurality of positions in the input image , wherein a classification value quantifies a presence of a class, the method comprising the following steps: evaluating (102) the data as a function of a threshold value, a first classification value for a respective position in the input image, which is either below or above the threshold value, is discarded (104a) and a first classification value for a respective position in the input image, which is either above or below the threshold value, is not discarded (104b). Method (100) according to Claim 1 , wherein the threshold value zero and a first classification value for a respective position in the input image which lies below the threshold value is discarded (104a) and a first classification value for a respective position in the input image which is above the threshold value is not discarded ( 104b). The method (100) according to at least one of the preceding claims, wherein discarding (104a) a first classification value for a respective position in the input image further comprises: setting the first classification value to a fixed value, in particular zero. The method (100) according to at least one of the preceding claims, wherein the first classification value is the non-normalized result of a class filter of the neural network, in particular for a background class, for a respective position in the input image, the discarding (104a) of a first classification value for a respective Position in the input image includes discarding the result of the class filter. The method (100) according to at least one of the preceding claims, wherein the data for the respective position in the input image include at least one further classification value and / or at least one value for an additional attribute, the further classification value being the non-normalized result of a class filter for an object class, in particular Target object class, the method further comprising: discarding the at least one further classification value and / or the at least one value for an additional attribute for a respective position depending on whether the first classification value for the respective position is discarded. Method (100) according to Claim 5 wherein discarding the at least one further classification value and / or discarding the at least one value for an additional attribute further comprises: setting the further classification value and / or the value for an additional attribute to a fixed value, in particular zero. The method (100) according to at least one of the preceding claims, wherein the method further comprises: processing (106) the non-discarded classification values, in particular forwarding the non-discarded classification values and / or applying an activation function, in particular Softmax activation function, to the non-discarded classification values. Device (200) for processing, in particular unnormalized, multi-dimensional, data of a neural network, in particular deep neural network, in particular for detecting objects in an input image, the data for a plurality of positions in the input image each including at least one first classification value, wherein the apparatus for performing the method (100) according to at least one of Claims 1 to 7th is trained. System (300) for detecting objects in an input image, comprising a device (200) for processing, in particular non-normalized, multi-dimensional, data of a neural network according to Claim 8 , wherein the system (300) further comprises a computing device (310) for applying an activation function, in particular Softmax activation function, in particular for calculating a prediction of the neural network, and the device (200) for forwarding the non-discarded classification values to the computing device ( 310) and / or on a memory device (320) assigned to the computing device (310). Computer program, the computer program comprising computer-readable instructions, when executed by a computer, the method (100) according to one of the Claims 1 to 7th expires. Computer program product, the computer program product comprising a memory on which a computer program is based Claim 10 is stored. Use of a method (100) according to one of the Claims 1 to 7th and / or a device Claim 8 and / or a system Claim 9 , and / or a computer program Claim 10 and / or a computer program product Claim 11 for at least partially autonomous movement of a vehicle, with an input image being captured by a sensor system, in particular a camera, radar sensor or lidar sensor, of the vehicle, with a method (100) according to one of the Claims 1 to 7th is carried out, with at least one control for the vehicle, in particular for automated braking, steering or acceleration of the vehicle being determined as a function of the result of the object detection. Use of a method (100) according to one of the Claims 1 to 7th and / or a device Claim 8 and / or a system Claim 9 , and / or a computer program Claim 10 and / or a computer program product Claim 11 for moving a robot system or parts thereof, an input image being captured by a sensor system, in particular a camera, of the robot system, a method (100) according to one of the Claims 1 to 7th is carried out, with at least one control of the robot system being determined as a function of the result of the object detection.

Images