DE102021211257A1 - Prevent attacks on an artificial neural network - Google Patents

Abstract

Method for operating an artificial neural network, hereinafter referred to as ANN (1), during the operation of a vehicle with multiple sensors, by feeding the ANN (1) with input data and ensuring a core functionality by deriving control signals from the output data of the ANN (1). be, with the following steps: - Receiving (S1) of sensor data; - Pre-processing (S2.1) of the sensor data in order to obtain input data for feeding the ANN and/or analyzing (S2.2) the sensor data with regard to sensor data legality; - Providing (S3) an AI replacement module that is redundant to the ANN (1), the AI replacement module being set up to ensure the redundancy with means alternative to the ANN (1);- Analyzing (S4.1) the ANN output data with regard to their confidence and/or changing (S4.2) the ANN output data in order to devalue the ANN output data;- storing (S5) a selection of result data from at least one of the steps: preprocessing and/or analyzing the sensor data; Analyzing and/or changing the ANN output data; - evaluating (S6) the preceding method steps and initiating a reaction to the evaluation.

Description

FIELD OF THE INVENTION

The present invention relates to a method for ensuring a core functionality of an artificial neural network, hereinafter abbreviated to ANN.

TECHNICAL BACKGROUND

It is known to control safety-relevant applications using artificial neural networks. If ANN are used to control safety-related processes, it is desirable to ensure that the ANN can be operated safely. This means that an attacker should not or hardly be able to harm a user, a third party or the environment. In addition, it should be ensured that the ANN produces reliable output data even if the environmental conditions are not ideal. In addition, it is desirable for developers of ANN that intellectual property, for example know-how, which is contained in an ANN is secured against theft.

SUMMARY OF THE INVENTION

Against this background, the present invention is based on the object of making the operation of a safety-relevant ANN more secure.

• - ein Verfahren zum Betreiben eines künstlichen neuronalen Netzwerks während des Betriebs eines Fahrzeugs mit mehreren Sensoren, indem das KNN mit Eingangsdaten gespeist wird und eine Kernfunktionalität gewährleistet, indem aus Ausgangsdaten des KNNs Steuersignale abgeleitet werden, mit den folgenden Schritten: Empfangen von Sensordaten; Vorverarbeiten der Sensordaten, um Eingangsdaten zum Speisen des KNN zu erhalten und/oder Analysieren der Sensordaten hinsichtlich der Sensordatenlegalität; Bereitstellen wenigstens eines KI-Ersatzmoduls, das zu dem KNN redundant ist, wobei das KI-Ersatzmodul eingerichtet ist, die Redundanz mit alternativen Mitteln wie das KNN zu gewährleisten; Analysieren der KNN-Ausgangsdaten hinsichtlich deren Konfidenz und/oder Verändern der KNN-Ausgangsdaten, um die KNN-Ausgangsdaten zu entwerten; Speichern einer Auswahl an Ergebnisdaten aus wenigstens einem der Schritte: „Vorverarbeiten und/oder Analysieren der Sensordaten“, „Analysieren und/oder Verändern der KNN-Ausgangsdaten“; Auswerten der vorangehenden Verfahrensschritte und Einleiten einer Reaktion auf die Auswertung.

Accordingly, it is provided:

• - A method for operating an artificial neural network during the operation of a vehicle with multiple sensors by the ANN is fed with input data and ensures a core functionality by control signals are derived from output data of the ANNs, with the following steps: receiving sensor data; pre-processing the sensor data to obtain input data for feeding the ANN and/or analyzing the sensor data for sensor data legality; providing at least one replacement AI module that is redundant to the ANN, the replacement AI module being set up to ensure the redundancy with alternative means such as the ANN; Analyzing the KNN output data with regard to their confidence and/or changing the KNN output data in order to invalidate the KNN output data; Saving a selection of result data from at least one of the steps: “preprocessing and/or analyzing the sensor data”, “analyzing and/or changing the ANN output data”; Evaluating the preceding method steps and initiating a reaction to the evaluation.

A sensor, also known as a detector, (measuring variable or measurement) pickup or (measuring) probe, is a technical component that detects certain physical, chemical properties or states, e.g. B. temperature, humidity, pressure, speed, brightness, acceleration, pH, ionic strength, electrochemical potential and / or the material composition of its environment can be qualitatively or quantitatively recorded as a measurand. These variables are recorded using physical or chemical effects and converted into an electrical signal that can be further processed as sensor data. Vehicle sensors are mounted on a vehicle to sense a vehicle environment. Sensor data that is recorded by vehicle sensors is vehicle environment-related sensor data. Vehicle sensors are sensors suitable for use on a vehicle and are mountable or mounted on a vehicle. Vehicle sensors include optical sensors, such as camera, lidar, radar, TOF, and other sensor technologies, such as acoustic sensors. In the method according to the invention, for example, sensor data is received from one or more camera sensors, lidar sensors, radar sensors and/or acoustic sensors.

Machine learning is a generic term for the "artificial" generation of knowledge from experience. An artificial system learns from examples and can generalize them after the learning phase has ended. To do this, machine learning algorithms build a statistical model based on training data. This means that the examples are not simply learned by heart, but that patterns and regularities are recognized in the learning data. The system can also assess unknown data (learning transfer) or fail to learn unknown data (overfitting).

In machine learning, the type and power of knowledge representation play an important role. A distinction is made between symbolic approaches, in which the knowledge - both the examples and the induced rules - is explicitly represented, and non-symbolic approaches, such as neural networks (ANN), which are "trained" to behave in a calculable manner, but are not Allow insight into the learned solutions; here knowledge is implicitly represented.

In this patent application, the terms AI (artificial intelligence) and machine learning are used interchangeably.

An AI module is a computer program based on an AI, e.g. on an ANN or a symbolic AI routine. In this patent application, the term AI module refers to an upper unit with several AI submodules and an ANN. The AI module according to the invention includes an ANN, an AI replacement module and other AI submodules. An KI submodule can contain several KI routines.

Support Vector Machines (SVMs) are a method of artificial intelligence. The starting point for building a support vector machine is a set of training objects for which it is known which class they belong to. Every object is represented by a vector in a vector space. It is the task of the support vector machine to fit a hyperplane into this space, which acts as a separating surface and divides the training objects into two classes. The distance between those vectors that are closest to the hyperplane is maximized. This wide, empty border should later ensure that objects that do not exactly correspond to the training objects are classified as reliably as possible.

Decision trees are a method of artificial intelligence for the automatic classification of data objects and thus for solving decision problems. They are also used for the clear presentation of formal rules. A decision tree consists of a root node and any number of inner nodes and at least two leaves. Each node represents a logical rule and each leaf an answer to the decision problem.

An artificial neural network (ANN) is in particular a network of networked artificial neurons that is simulated in a computer program. The artificial neurons are typically arranged on different layers. Usually, the artificial neural network comprises an input layer and an output layer (output layer), whose neuron output is the only one of the artificial neural network that is visible. Layers lying between the input layer and the output layer are typically referred to as hidden layers. Typically, an architecture or topology of an artificial neural network is first initiated and then trained in a training phase for a specific task or for a plurality of tasks in a training phase.

The term "topology of an ANN" includes all aspects relating to the structure of an ANN. This includes, for example, the number of neurons in the ANN, the allocation of the neurons to the individual layers of the ANN, the number of layers in an ANN, the networking of the neurons and the weighting of the networking.

The training of the artificial neural network typically includes changing a weight of a connection between two artificial neurons of the artificial neural network. The weight contains information about the strength of the consideration of an input of a neuron. The training of the artificial neural network can also include developing new connections between artificial neurons, deleting existing connections between artificial neurons, adjusting threshold values of the artificial neurons and/or adding or deleting artificial neurons.

An example of an artificial neural network is a shallow artificial neural network, which often contains only a single hidden layer between the input layer and the output layer and is therefore relatively easy to train. Another example is a deep artificial neural network, which contains multiple nested hidden layers of artificial neurons between the input layer and the output layer. The deep artificial neural network enables improved recognition of patterns and complex relationships.

In this application, training data are data pairs consisting of input data that are to be processed by the ANN and target result data that are to be determined by the ANN. During the training, the ANN is adjusted on the basis of a comparison of target result data with the actual result data determined by the ANN, as a result of which a training effect occurs.

An actual reaction signal can be derived from actual result data. A target reaction signal can be derived from target result data.

Data legality encompasses all aspects of data integrity and data authenticity. Along with availability and confidentiality, integrity is one of the three goals of information security. Integrity is defined as "preventing unauthorized modification of information". Data authenticity means that data can be traced back to its origin at any time.

An ANN is fed with input data. Input data is based on sensor data and can be raw sensor data or pre-processed sensor data.

If an AI solves a problem by alternative means, it means that the AI can rely on an alternative technology, an alternative Structure may have been trained with other training data and / or is fed with alternative input data.

Devaluation of ANN output data means changing the data in such a way that fictitious training of another ANN with the ANN output data as target output data results in an ANN that produces output data of lower confidence that determined the ANN output data. So it is conceivable that a classifying ANN returns an incorrect class. In general, it can also be stated that the quality of a canceled issue is lower.

A central data store is, for example, a cloud, ie a storage space that is provided as a service over the Internet. Central systems include central elements such as central data storage or central evaluation units that are provided on the Internet.

The term sensor data includes raw sensor data as well as processed sensor data.

Input data deviations can be artificial (e.g. due to attacks), natural (e.g. due to sensor pollution) or technical (e.g. due to signal interference). The deviation refers to the change in the input data caused by the artificial, natural or technical disruption.

A test bench is a device or device with which a technical object can be tested for its properties in a reproducible manner. In addition to the mechanical design for positioning the test object, a test bench also includes the appropriate sensors and controls to be able to generate the properties and log the measured values.

Binning is understood to be class formation as a preprocessing technique. For example, the target sets of attributes are divided into intervals - so-called bins - in ascending order of size if the target set is a (one-dimensional) number. All attribute values are then replaced with the representative of the interval in which the value is located. This representation value is often about the average or the median. This is a form of quantization.

On the one hand, this form of binning can be used to reduce the number of values of a given attribute and thus the amount of data. Furthermore, this can reduce the consequences of small deviations in the attribute values, for example due to measurement errors.

In digital image processing, binning means combining adjacent pixels in an image file. Image noise is reduced by forming pixel blocks. In return, however, the image resolution is reduced according to the number of combined pixels, i. H. the image becomes coarser.

Different AI routines can also produce different output data for identical input data. These differences can be due to the data format, the structure of the data processing and the like. A certain degree or a certain structure of differences in the output data of different AI routines can therefore be expected. If an AI routine is unsuitable for processing certain data, the differences in the output data can increase beyond an expected level or structure.

In this patent application, redundant AI replacement module means that the redundant AI replacement module ensures that the safety-relevant operation of a control unit, which determines control data from output data, continues. Accordingly, it can be provided that the AI replacement module guarantees identical or limited functionality as the ANN. Redundancy therefore does not mean guaranteeing the entire range of functions, but only its safety-critical part. In relation to a vehicle, this means that a core functionality can relate to the autonomous control of the vehicle and the redundant AI replacement module is set up to steer the vehicle into the nearest parking bay if the ANN fails. Such an AI replacement module can be mapped with lower complexity, since the AI replacement module does not have to be able to drive at high speeds, for example.

The phrase "at runtime" means that something happens during the intended use of a computer program.

Computer program products typically include a sequence of instructions that, when the program is loaded, cause the hardware to perform a specific method that leads to a specific result.

The basic idea of the invention is to create an overarching system for defending against attacks on an artificial neural network. The system includes multiple sub-units configured to detect and/or mitigate certain aspects of an attack.

The attacks can be, for example, an attack aimed at stealing intellectual property on which an AI module is based in order to copy the AI module. An AI module is fed with input data that is as comprehensive as possible and the structure of the AI is derived from the output data produced by the AI.

The attack can also be an attempt at manipulation. If, for example, a vehicle is controlled with the AI module, the aim of an attack can be to control the vehicle illegally in order to cause damage.

One aspect of the invention relates to the handling of sensor data. Sensor data for feeding an artificial neural network can be pre-processed to ward off attacks aimed at corrupting the sensor data, or at least to reduce their impact.

Alternatively and/or additionally, it is provided to analyze the sensor data with regard to the legality of the sensor data in order to detect an attack on the sensor data and/or to simplify the processing of the sensor data by the ANN.

As part of the pre-processing of the sensor data, it is conceivable to normalize the sensor data, for example cropping sensor data to a previously known range of a sensor, changing pixel values of the sensor data, changing the resolution of the sensor data, analyzing the sensor data with regard to their frequency and determining to filter frequencies, to quantify the sensor data by binning, and/or the like.

Alternatively and/or additionally, it is also conceivable to select input data in a targeted manner in order to simplify, accelerate or make the processing by the ANN more robust.

Provision is also made to provide at least one AI replacement module that is redundant for the ANN and which can be accessed if it is determined that the ANN is producing output data of insufficient confidence or an attack on the ANN is detected. The AI replacement module can be a different ANN or an alternative AI technology, for example a machine learning concept that was trained using other means than the ANN. Furthermore, it is conceivable to provide an AI replacement module with a large number of AI replacement routines that process other input data, for example raw sensor data or pre-processed sensor data.

Provision can also be made for a suitable option to be selected from the above options depending on the cause, which means that the reaction to attacks of different types can differ, or if it is recognized that the ANN was insufficiently trained on input data at runtime.

In addition, provision is made for the ANN output data to be analyzed with regard to their confidence and/or for the ANN output data to be changed in order to invalidate the ANN output data.

The importance of the confidence of the initial data is explained below using a classification network. However, the following aspects can also be transferred to non-classifying networks.

Confidence is a measure of the quality or reliability of the output of an ANN and can be modeled using numerous models. For example, the confidence about the distance of a data point in the input data space to the next decision limits can be modeled. This allows an assessment of how close a data point is to another decision boundary. If it turns out that the distance between the data point in the input data space and another decision limit is small, this results in a low confidence in a classification result, since even a small variation in the input data would lead to a different classification result.

The assessment of the significance or reliability of a classification result of an artificial neural network is relevant for a large number of safety-critical applications.

The analysis of the confidence is based on the assumption that the classification certainty, i.e. the confidence, of an ANN is higher when the input data are in the center of a decision boundary. At this point, the data is most reliably assigned to a specific classification result, and small changes in the data do not change the classification result. Accordingly, the measure of confidence should be such that at such points the determined confidence of the ANN is high and such that the determined confidence for input data is low near a decision boundary.

In practice, the location of the decision boundaries is not always known. In this case, decision limits can be determined empirically.

The confidence can be determined in real time at the same time as the classification results of input data. Correspondingly, noisy input data is generated in real time and the classifying artificial neural network is fed with this noisy input data for the purpose of its classification. Accordingly, the confidence is high when large disturbances do not change the result, and low when small disturbances change the result. It is conceivable to determine the confidence on the basis of a measure of a difference between the classification results of the noisy and non-noisy input data if the classification results of the noisy input data differ from the non-noisy ones.

Alternatively, it is also conceivable to determine decision limits, e.g. in a training phase, before an ANN is used as intended. The confidence of the output data can then be determined from the distance between the input data and a decision limit.

The aim of changing the ANN output data is to invalidate the ANN output data, for example if it is recognized that an attempt is being made to steal know-how relating to the ANN as part of an attack. By changing the data, output data can be encoded, falsified or deleted. It is conceivable to falsify the output data in such a way that they remain usable or become unusable for a control device which is controlled by the ANN.

Provision is made for the result data described above to be stored in order to detect attacks on the ANN if a runtime-dependent change in the result data is relevant for this, and to document the attack.

It goes without saying that the method steps described are evaluated in order to initiate a reaction to the evaluation. Exemplary reactions are explained elsewhere, e.g. selecting an alternative AI replacement module, changing input data at runtime, manipulating ANN output data and/or the like.

Advantageous refinements and developments result from the further dependent claims and from the description with reference to the figures of the drawing.

According to a preferred development of the invention, it is advantageous if the sensor data are preprocessed by the sensor data being quantified by means of binning, with a bin size being selected in particular at runtime.

With binning, sensor data is sorted into so-called bins, which form a predetermined interval or a subset of an input data space. Each bin is represented by a specific value, for example a marginal value or an average value. The data size of input data can thus initially be reduced. In addition, binning makes it possible to remove a small influence of noise in the input data.

The influence of noise on input data can thus be reduced, so that attacks based on disturbed, noisy input data are made more difficult or weakened. It is also conceivable to adapt the size of a bin at runtime and thus adjust the sensitivity to noise influences. For example, it is conceivable that it is recognized that sensor data is being disturbed by an attack and that this attack is responded to by greater quantification, i.e. enlarging the bins, of the sensor data. Alternatively, it is also conceivable to select another AI replacement module instead of the ANN, which is known to be more robust in the event of noise influences.

According to a preferred development of the invention, it is advantageous to preprocess sensor data by selecting the sensor data according to their relevance for the core functionality.

Accordingly, it is advantageous if input data, with which the ANN is fed, contain only an application-specific specified section of the sensor data. If the ANN is an application for lane detection, for example, it is conceivable to cut off an upper area of a camera image, since typically only sky can be seen there. The display of the sky does not affect lane detection. Sky clipping reduces the area that can be used by an attack to inject noise or other malicious patterns into the input data.

This means that it is conceivable to examine the input data to determine which subsets are relevant for an ANN and to cut out irrelevant subsets from the input data before the ANN is fed with the input data.

It goes without saying that diverse sensor data can be changed under this aspect. For example, it is conceivable to cut out certain components from a signal at certain (non)relevant frequencies.

According to a preferred development of the invention, it is advantageous if the sensor data be analyzed with regard to sensor data legality by examining the change in a sensor data sequence.

For example, it is known that so-called black box evasion attacks feed an ANN successively with many data examples, with the data examples differing only slightly. The result can be that no difference is discernible to a human observer. By analyzing the (superordinate) structure of the data examples, it can be determined whether data examples have been modified in a manner typical of an attack.

A preferred development of the invention provides for the input data or the sensor data to be analyzed with regard to their distribution by comparing a distribution of the sensor data with a distribution of the training data with which the ANN was trained in a training phase.

This results in different results of the comparison. It is conceivable that the sensor data are distributed at runtime independently and identically to the distribution of the training data with which the ANN was trained. In this case, the ANN has been trained on representative data of this distribution and, when fed with the sensor data, will determine output data that are congruent with the input data.

Instead, it is also conceivable that the input data is distributed similarly to the training data at runtime. Accordingly, the distribution of the input data at runtime deviates slightly from the training data, so it can happen that data points lie slightly outside the training distribution. This can result in reduced confidence in the output data determined based on the input data. Since the majority of the data points are still within the distribution of the training data, the worst that can be expected is a small influence of noise on the distribution.

As a further deterioration, it is also conceivable that the sensor data deviate significantly from the distribution of the training data. Accordingly, the distribution of runtime data has shifted compared to the distribution of training data. This can result, for example, from use under changed environmental conditions. For example, it is conceivable that a driver assistance system was trained for an application in Germany and the application of the driver assistance system in another country leads to a shift in the distribution of the input data. It can be assumed that the ANN, without an adjustment, produces insufficiently confident output data when fed with the shifted distributed input data.

Furthermore, in a further deterioration, it is conceivable that the distribution of the sensor data at runtime is completely different from the distribution of the training data. In this case, the distribution is not only shifted, but differs by other characteristics, for example the type of distribution, the variance, a bias or the like. For example, it is conceivable that a driver assistance system that was developed for freeway driving is used for inner-city driving and the inner-city sensor data is distributed completely differently from the freeway-related training data. In this case, it can be assumed that the ANN will produce unusable output data if it is fed with input data that is distributed differently.

In addition, it is also conceivable that input data was deliberately changed as part of an attack on the ANN in order to deceive the ANN. Such noisy sensor data often lies at the edges of a decision region, which means that small changes in the sensor data lead to large changes in the output data. It is conceivable that such data examples were deliberately constructed as part of an attack in order to deceive an ANN so that an attacker can control the output of the ANN.

According to a preferred development of the invention, it is advantageous to provide an AI replacement module with multiple AI replacement routines and to select a suitable AI replacement routine based on a bin size. It is therefore conceivable to optimize the ANN to a standard bin size and to use an AI replacement routine if the bin size is compared to changed from the default size. One or more AI replacement routines can be optimized for specific bin sizes.

According to a preferred development of the invention, several AI replacement routines are provided which are based on at least one of the following technologies: support vector machine (SVM), decision trees, other symbolic AI components. These AI models differ in structure and functionality from an ANN. Thus, attacks that deceive an ANN by designing the attacks to exploit vulnerabilities specific to an ANN can be countered by closing vulnerabilities specific to an ANN with another technology.

According to a preferred embodiment of the invention, an AI replacement module with at least provided to a specialized AI replacement routine that is set up to produce more valuable output data on a subset of the input data space of the ANN.

For example, it is conceivable that an AI replacement module that has been trained for a large number of countries is made available and that a switch is made to this AI replacement module if it is recognized that input data fed into an ANN comes from a country for which the ANN has not been trained.

Similar applications are conceivable in poor visibility conditions such as night, fog, heavy precipitation and/or the like by providing an ANN that is specialized for such conditions.

According to a preferred development of the invention, at least one robust artificial intelligence is provided, which is set up to produce more reliable output data than the ANN for data that has been falsified according to a specific pattern. For example, it is conceivable that it is recognized that a security-critical ANN is attacked at runtime in order to cause damage. In this case, damage may be prevented or mitigated by resorting to a replacement AI module that is specifically designed to be resilient in the event of an attack. This can also include relatively simple concepts such as the safe switching off of the control unit, which is controlled by the ANN. For example, it is conceivable that the AI replacement module is set up to park a vehicle safely.

According to a preferred development of the invention, the input data and/or the sensor data are examined with regard to whether the sensor data were generated on a test bench or other test apparatus or whether a control device, in particular of a vehicle, which derives control signals from the output data of the ANN, itself is on a test bench.

An attacker who intends to steal know-how from the ANN could obtain a comprehensive, relevant input data set and feed the ANN with this input data set on a test bench in order to obtain all the result data, including control data from the control unit, for this input data set. Within the framework of such a procedure, it is possible to copy an ANN using statistical means. However, when analyzing a sequence of sensor data, it is possible to detect test bench operation. If the control unit is a vehicle control unit and the sensor data is vehicle (environment)-related sensor data, test bench operation can be recognized by abrupt changes in sensor data that relate to the vehicle itself or to the vehicle environment. For example, pedal positions (for example a brake) are monitored using suitable sensors. If the pedal position changes abruptly and a large number of different pedal positions are also run through, it can be assumed that a corresponding vehicle with an associated artificial neural network is on a test bench.

According to a preferred development of the invention, the confidence of the output data is stored and a confidence stream of an associated output data stream is analyzed. Alternatively and/or additionally, it is also conceivable to compare the confidence of the output data of the ANN with the confidence of an AI replacement module.

For example, it can happen that the confidence of the ANN or an AI replacement module changes suddenly or even suddenly. Sudden or erratic changes in confidence are not to be expected when used as intended, since the ANN is fed with a continuous stream of input data. Accordingly, it is advantageous to store the confidence over several time steps in order to correct current confidence values.

According to a preferred development of the invention, a cause for abnormalities with regard to the confidence is inferred on the basis of the confidence analysis of an output data stream if abnormalities with regard to the confidence are ascertained.

According to a preferred development of the invention, attack data or data containing information about potential attacks is transmitted to a central data store. Thus, it is possible to immediately distribute knowledge about specific attacks and thus warn other potentially compromised modules or users. It is thus possible to publish attack data or to provide information about an attack if an unnatural structure in sensor data or unnatural behavior in actuators is noticed.

According to a preferred development of the invention, artificial intelligence is used to select whether the output data of the ANN and one or more AI replacement routines are to be averaged or which of the replacement routines of the AI replacement module or the ANN produces the most reliable output data if the output data differs more than expected differ.

It is conceivable that a specific AI replacement module is selected on the basis of a particular situation-related suitability or that an average value is determined from multiple output data from multiple AI modules and is processed further. The mean can be calculated from weighted values.

If it turns out at runtime that an ANN is exposed to an attack or the output of the ANN is not reliable, provision can be made for the output data of the ANN to be overwritten. In this case, the output of the ANN can be overwritten by the outputs of the AI replacement module, with an evaluation method being used to determine a suitable weighting of several outputs from different AI replacement routines or a suitable AI replacement routine for generating replacement output data. If, for example, a know-how theft attack is detected, false output data can be generated in order to confuse an attacker or to delete the output.

According to a preferred development of the invention, the ANN ensures functionality within the scope of autonomous or assisted driving and the method is set up to detect and in particular to ward off an attack on the ANN.

According to a preferred development of the invention, output data from the ANN and the AI replacement module are analyzed with regard to differences, and a decision is made as to whether a cause for the differences is safety-critical for the operation of the AI module.

It goes without saying that an AI system that is set up to carry out the method as described above is advantageous. The AI system has an ANN, which ensures the core functionality, a computing unit, in particular an AI, which is set up to preprocess the sensor data in order to receive input data for feeding the ANN and/or to analyze the sensor data with regard to sensor data legality, an AI Replacement module with at least one, in particular a large number of AI replacement routines, which is redundant to the ANN, in that the AI replacement module is set up to ensure the core functionality with alternative means such as the ANN, a computing unit, in particular an AI, which is set up to To analyze ANN output data with regard to their confidence and/or to change the ANN output data in order to devalue the ANN output data, a data memory for storing a selection of result data from an AI, which is set up, the method steps of a method as described above , evaluate and initiate a reaction, it being ensured that the ANN can only be operated during the operation of the other components.

It goes without saying that a training method for an AI module that is set up to carry out the method as described above is advantageous. It is provided that the system is trained with a training data set comprising training input data and target output data, the training input data containing data relating to an attack on the ANN, which ensures the core functionality, of the AI module.

Provision can be made for the AI module to be trained as part of the training specifically to allocate abnormalities with regard to the confidence of initial data to possible causes for the abnormalities in confidence.

Provision can also be made for the AI module to be specifically trained as part of the training to allocate abnormalities in an input data stream or differences in an output data stream of multiple AI routines or the ANN to possible causes for the abnormalities or the differences.

A computer program product according to a method of an embodiment of the invention performs the steps of a method as described above when the computer program product runs on a computer, in particular an in-vehicle computer. If the program in question is used on a computer, the computer program product causes an effect, namely the closing of security gaps in an ANN.

character list

• 1 ein schematisches Blockdiagramm einer Ausführungsform der Erfindung;
• 2 ein schematisches Blockdiagramm einer Ausführungsform der Erfindung.

The present invention is explained in more detail below with reference to the exemplary embodiments given in the schematic figures of the drawings. They show:

• 1 a schematic block diagram of an embodiment of the invention;
• 2 Figure 12 is a schematic block diagram of an embodiment of the invention.

The accompanying drawings are provided to provide a further understanding of embodiments of the invention. They illustrate embodiments and, together with the description, serve to explain principles and concepts of the invention. Other embodiments and many of the advantages mentioned arise in look at the drawings. The elements of the drawings are not necessarily shown to scale with respect to one another.

In the figures of the drawings, elements, features and components that are identical, have the same function and have the same effect--unless stated otherwise--are each provided with the same reference symbols.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

1 shows an AI module 10 for carrying out a method as described above. The AI module 10 is part of a driver assistance system for controlling a motor vehicle. The AI module 10 includes an ANN 1, which ensures the core functionality of the AI module 10, namely controlling a vehicle. The KI module 10 also includes several KI submodules 2, 3, 4 and 6 in order to ensure the safety-critical operation of the KI module 10 if it turns out that the ANN generates output data of insufficient confidence or an attack on the AI module or the ANN 1 takes place, as well as a data memory 5.

In order to ensure the safety-critical operation of the AI module 10 , raw sensor data 73 is analyzed by an AI routine 22 of the AI submodule 2 . The analysis of the input data aims to detect structural anomalies or aberrations in the input data that indicate an attack or sensor data of insufficient quality. Furthermore, the AI submodule 2 includes an AI routine 21 in order to preprocess the raw sensor data 73 . As part of the pre-processing of the raw sensor data 73, provision is made for the raw sensor data to be selected in an application-specific manner. Accordingly, it is provided that an upper part of an image containing the sky is discarded from the sensor data if lane detection of the AI module 10 is active. Alternatively or additionally, there are other preprocessing methods for other applications, for example when a parking aid is activated. The analysis and the pre-processing of the input data 73 by means of the AI routines 21, 22 take place in parallel.

The results of the AI routine 22 with regard to the structural analysis of the raw sensor data 73 are first stored in a sample memory 51 of the data memory 5 . The AI routine 63 of the AI submodule 6 accesses the pattern memory 51 of the data memory 5 in order to evaluate the results of the structural analysis of the raw sensor data 73 that are stored in the memory routine 51 . Results of the evaluation of the AI routine 63 are forwarded to the AI routine 64 of the evaluation AI 6 in order to initiate a suitable reaction to the result of the structural analysis. In the normal, favorable case, an appropriate response to the structure analysis is to take no further action, since the structure of the input data indicates that the legality of the input data is guaranteed. If the AI routine 64 comes to the conclusion that sufficient data legality is not guaranteed, the AI routine 63 can initiate reactions that are described below.

In the normal case, i.e. if no further reactions are initiated by the AI routine 64, the pre-processed input data is forwarded by the AI routine 21 to the ANN 1 in order to feed the ANN 1 with the pre-processed input data in order to obtain output data therefrom. Normally, 1 control signals for controlling the vehicle are derived from the output data of the ANN. Furthermore, the output data of the ANN 1 are monitored by the AI routine 41 of the AI submodule 4 with regard to their confidence. Accordingly, an associated confidence stream for the output data stream is determined for an output data stream of the ANN 1 and stored in the confidence memory 52 of the data memory 5 . The AI routine 61 of the AI module 6 accesses the confidence memory 52 in which the confidence flow is stored in order to analyze the confidence flow. If the AI routine 61 detects a conspicuous confidence flow, for example erratic or rapid changes in confidence, the AI routine 64 can initiate a suitable reaction to the abnormalities in the confidence flow. Abnormalities in the confidence stream can result from various causes, for example from insufficient data legality due to an attack on the AI module 10 or from a sudden sensor pollution or a sudden degradation of the visibility conditions, for example if a vehicle is driven through a fog sink and is inside the fog sink suddenly degrades visibility. Accordingly, the AI routine is trained in such a way that the AI routine 64 can further narrow down the cause of the confidence jump based on the type of confidence jumps, their frequency and/or their strength and can determine a suitable reaction to the confidence jump.

The initiated reaction is sent to the AI module 4 for further implementation by the AI routine 42, by means of which output data from the ANN 1 and/or the AI replacement routines 31, 32, 33 can be changed. If the AI routine 64 comes to the conclusion that a conspicuous confidence pattern results from an attack on the AI module 10, provision can be made for the output data to be changed change, for example to deceive an attacker and thus repel the attack. If the AI routine 64 determines that abnormalities in the confidence pattern have other non-critical causes, for example sudden sensor contamination, or that there are no abnormalities in the confidence pattern, it can be provided that the output data from the ANN 1 or the AI replacement routines 31, 32, 33 not to be changed.

In addition to the ANN 1, a number of AI replacement routines 31, 32, 33 are also supplied with the input data in order to ensure redundancy with the ANN 1. Accordingly, the AI replacement routine 31 is fed with the pre-processed input data, whereas the AI replacement routines 32 and 33 are fed with the unprocessed raw sensor data. Accordingly, the AI replacement routine 31 is a symbolic AI, which means that the AI replacement routine 31 was trained with symbolic decision rules instead of with a training data set that contains data sets that are as similar as possible to future sensor data. Accordingly, the output data of the ANN 1 and the output data of the AI replacement routine 31 should not differ too greatly from one another. If the output data of the ANN 1 and the output data of the AI surrogate routine 31 differ from each other by a predetermined amount, this indicates that the ANN 1 or the AI surrogate routine 31 is more suitable to process the input data. Significant differences in the output data of the ANN and the AI replacement routine 31 indicate that at least the ANN or the AI replacement routine 31 was not trained or was insufficiently trained on this type of input data. However, since the training was designed in such a way that all conceivable constellations of input data were covered, a significant difference in the output data of the ANN 1 and the output data of the AI replacement routine 31 indicates that sensor data have been artificially changed.

Furthermore, the AI replacement module 3 contains a further AI routine 32 which is in the form of an ANN which is fed with raw sensor data 73 . The ANN 1 and the AI replacement routine 32 are trained in such a way that they produce output data that is as similar as possible or preferably identical for sensor raw data 73 or preprocessed sensor raw data 73 . Accordingly, differences in the output data between the ANN 1 and the AI replacement routine 32 that go beyond an expected level are almost always due to a difference between the raw sensor data 73 and the pre-processed sensor data. The result can be that the raw sensor data is disturbed and the disturbance was at least partially averaged out by the pre-processing of the raw sensor data. This can indicate, among other things, insufficient data legality due to an attack.

The AI routine 62 can be trained to infer specific causes in the input data from differences in the output data of the ANN 1 and the AI replacement routines 31, 32, 33. The AI routine 62 is preferably trained in such a way that harmless differences in the output data that do not affect the core functionality, namely the control of a motor vehicle, differ from safety-critical differences in the output data, i. H. that, for example, the security of the AI module 10 is not sufficiently guaranteed.

Furthermore, the AI replacement module 3 contains a further symbolic AI routine 33 which is fed with raw sensor data 73 . Accordingly, it can be expected that the AI routines 31 and 33 produce output data that differ little or not at all from one another. Differences in the output data of the AI routines 31 and 33 are almost always due to a difference between the raw sensor data 73 and the pre-processed sensor data. Furthermore, differences between the KI routine 32 and the KI routine 33 are almost always due to the different structure of the KI routines 32 and 33, since the KI routine 32 is an artificial neural network and the KI routine 33, is a symbolic AI.

All of the output data from the AI replacement module 3 is monitored with regard to its confidence and its confidence is analyzed analogously to the confidence in the output data from the ANN 1 . Furthermore, all of the output data from the AI replacement module 3 and the ANN 1 is transmitted to the AI routine 62, which is trained based on differences in the output data from the ANN 1 or the AI routines 31, 32, 33 to a cause for the differences in the output data, the AI routine 64 being set up to decide whether the cause is safety-critical or not.

If the AI routine 64 detects a security risk for the AI module 10, an alarm message is sent, which is distributed to a large number of interested parties via a central data store (not shown). The interested parties can be other road users, other AI modules, developers or surveillance systems that ensure the safety of a variety of AI modules.

Finally, the AI module 10 determines a result 74 from the previous steps. In the favorable normal case, the result 74 is based on the output data of the ANN, from which a control signal for a control device is derived. However, if the AI module 10 recognizes an attack the output data can be changed as described above.

In 1 indicates that the raw sensor data 73 can be manipulated by a manipulation attack 71 or generated as part of a theft attack 72, with the purpose of the KN module 10 being to prevent a manipulation attack 71 or a theft attack 72 using the AI submodules 2, 4, 6 and the AI replacement module 3 to recognize and fend off.

• In dem Schritt S1 werden Sensordaten empfangen.
• In dem Schritt S2.1 werden die Sensordaten vorverarbeitet, um Eingangsdaten zum Speisen des KNN zu erhalten. Alternativ oder zusätzlich werden in dem Schritt S2.2 die Sensordaten hinsichtlich der Sensordatenlegalität analysiert. Werden Sensordaten nicht vorverarbeitet kann das KNN auch mit Sensorrohdaten gespeist werden.
• In dem Schritt S3 wird ein KI-Ersatzmodul 3 bereitgestellt, welches zu dem KNN redundant ist, wobei das KI-Ersatzmodul 3 eingerichtet ist, die Redundanz mit zu dem KNN 1 alternativen Mitteln zu gewährleisten. Die alternativen Mittel können als unterschiedliche Eingangsdaten, beispielsweise Sensorrohdaten oder vorverarbeitete Sensordaten bzw. anderweitig vorverarbeitete Sensordaten ausgebildet sein oder als eine KI-Routine, die auf einer anderen Technologie oder auf einer anderen Struktur wie das KNN 1 beruht, ausgebildet sein.
• In dem Schritt S4.1 werden die KNN-Ausgangsdaten hinsichtlich deren Konfidenz analysiert. Alternativ oder zusätzlich werden die KNN-Ausgangsdaten in dem Schritt S4.2 verändert, um die KNN-Ausgangsdaten zu entwerten.
• In dem Schritt S5 wird eine Auswahl an Ergebnisdaten aus wenigstens einem der Schritte S2.1, S2.2, S4.1, S4.2 gespeichert.
• In dem Schritt S6 werden die vorangehenden Verfahrensschritte S1 bis S5 ausgewertet und eine Reaktion auf die Auswertung wird eingeleitet.

2 FIG. 12 shows a schematic block diagram of a method for operating an ANN during operation of a vehicle with multiple sensors. The ANN 1 is fed with input data and ensures a core functionality in that control signals are derived from the output data of the ANN. The procedure includes the following steps:

• In step S1 sensor data are received.
• In step S2.1, the sensor data are pre-processed in order to obtain input data for feeding the ANN. Alternatively or additionally, in step S2.2, the sensor data are analyzed with regard to the legality of the sensor data. If sensor data is not pre-processed, the ANN can also be fed with raw sensor data.
• In step S3, an AI replacement module 3 is provided, which is redundant to the ANN, the AI replacement module 3 being set up to ensure the redundancy with alternative means to the ANN 1 . The alternative means can be embodied as different input data, for example raw sensor data or preprocessed sensor data or otherwise preprocessed sensor data, or as an AI routine that is based on a different technology or on a different structure than the ANN 1 .
• In step S4.1, the ANN output data are analyzed with regard to their confidence. Alternatively or additionally, the KNN output data are changed in step S4.2 in order to invalidate the KNN output data.
• In step S5, a selection of result data from at least one of steps S2.1, S2.2, S4.1, S4.2 is stored.
• In step S6, the preceding method steps S1 to S5 are evaluated and a reaction to the evaluation is initiated.

Reference List

S1-S6

process steps

1

CAN

2

AI submodule

3

AI replacement module

4

AI submodule

5

data storage

6

AI submodule

10

AI module

21

AI routine

22

AI routine

31

AI replacement routine

32

AI replacement routine

33

AI replacement routine

41

AI routine

42

AI routine

51

pattern store

52

confidence memory

61

AI routine

62

AI routine

63

AI routine

64

AI routine

74

Result

71

manipulation attack

72

theft attack

73

sensor raw data

Claims (20)

Method for operating an artificial neural network, hereinafter referred to as ANN (1), during the operation of a vehicle with multiple sensors, by feeding the ANN (1) with input data and ensuring a core functionality by deriving control signals from the output data of the ANN (1). with the following steps: - receiving (S1) sensor data; - Preprocessing (S2.1) the sensor data in order to obtain input data for feeding the ANN and/or analyzing (S2.2) the sensor data with regard to the sensor data legality; - Providing (S3) an AI replacement module that is redundant to the ANN (1), the AI replacement module being set up to ensure the redundancy with the ANN (1) alternative means; - Analysis (S4.1) of the ANN output data obviously their confidence and/or changing (S4.2) the ANN output data in order to invalidate the ANN output data; - Saving (S5) a selection of result data from at least one of the steps: preprocessing and/or analyzing the sensor data; Analyzing and/or changing the ANN output data; - Evaluation (S6) of the preceding method steps and initiation of a reaction to the evaluation. procedure after claim 1 , the sensor data being pre-processed by the sensor data being quantified by means of binning, with a bin size being selected at runtime in particular. Method according to one of the preceding claims, wherein the sensor data are pre-processed by the sensor data being selected according to relevance for the core functionality. A method according to any one of the preceding claims, wherein the sensor data is analyzed for sensor data legality by examining the change in sensor data time series. Method according to one of the preceding claims, in which the sensor data are analyzed by comparing a distribution of the sensor data with a distribution of training data with which the ANN (1) was trained in a training phase. Method according to one of the preceding claims, provided that claim 2 back-related, providing an AI replacement module with multiple AI replacement routines and an AI replacement routine based on a bin size claim 2 is selected. Method according to one of the preceding claims, wherein an AI replacement module is provided with a plurality of AI replacement routines based on at least one of the following technologies: support vector machines, decision trees, decision-based AI components that have not been trained on data; wherein at least one AI replacement routine is selected based on the pre-processing and/or analyzing of the sensor data in order to reduce the influence of input data deviations on the output data. Method according to one of the preceding claims, wherein at least one specialized AI replacement module is provided, which is set up to produce more valuable output data than the ANN on a subset of the input data space of the ANN (1). Method according to one of the preceding claims, wherein at least one robust AI replacement module is provided which is set up to produce more confidential output data than the ANN (1) for data which has been corrupted according to a specific pattern. Method according to one of the preceding claims, in which the input data and/or the sensor data are examined to determine whether the sensor data were generated on a test bench or whether a control device, in particular for a vehicle, which derives control signals from the output data of the ANN (1) is on a test bench. procedure after claim 9 , the output data being invalidated if test bench operation that was not authorized was detected. Method according to one of the preceding claims, wherein the confidence in the output data is stored and a confidence curve of a time series of output data is analyzed and/or the confidence in the output data of the ANN (1) is compared with the confidence in an AI replacement module. procedure after claim 12 , a cause for abnormalities with regard to the confidence being inferred on the basis of the confidence analysis and/or based on the confidence comparison if abnormalities with regard to the confidence are determined. Method according to one of the preceding claims, wherein attack data or data about a potential attack are transmitted to a central data store. Method according to one of the preceding claims, wherein it is selected whether output data of the ANN (1) and output data of the AI replacement module are to be averaged or whether the AI replacement module or the ANN (1) produces the most confidential output data if the output data changes more strongly than expected differentiate. Method according to one of the preceding claims, wherein the core functionality relates to autonomous or assisted driving, in particular the recognition and tracking of lanes, lane boundaries, lane markings, other motor vehicles, traffic signs and / or light signals (systems), and the method is set up to attack to detect and, in particular, to fend off the ANN (1). Method according to one of the preceding claims, wherein output data of the ANN (1) and the AI replacement module (3; 31; 32; 33) are analyzed with regard to differences and a decision is made as to whether a cause for differences is safety-critical for the operation of the AI module (10). AI module (10), which is set up to carry out the method according to any one of the preceding claims, with - an ANN (1), which ensures the core functionality; - A computing unit, in particular an AI submodule (2; 21; 22), which is set up to preprocess the sensor data in order to receive input data for feeding the ANN (1) and/or to analyze sensor data with regard to sensor data legality; - An AI replacement module (3; 31; 32; 33), in particular with a large number of AI replacement routines (31; 32; 33), which is redundant to the ANN (1) in that the AI replacement module (3) is set up to ensure the redundancy to the ANN (1), the AI replacement module determining the output data with alternative means such as the ANN (1); - A computing unit, in particular an AI submodule (4; 41; 42), which is set up to analyze the KNN output data with regard to their confidence and/or to change the KNN output data; - a data memory (5; 51; 52) for storing relevant result data; - An AI submodule (6; 61; 62; 63; 64), which is set up to evaluate the method steps of a method according to one of the preceding claims and to initiate a reaction, it being ensured that the ANN (1) only during operation the other components can be operated. Training method for an AI module, which is set up, the method according to one of the above Claims 1 until 14 to be carried out, the system being trained with a training data set comprising training input data and target output data, the training input data containing data relating to an attack on the ANN (1) of the AI module. Computer program product with program code means to implement the method according to one of Claims 1 - 17 to perform.

Images