DE102020215885A1 - PROCEDURE AND SYSTEM FOR DETECTING AND MITIGATION OF INTERFERENCE - Google Patents

Abstract

One aspect of the present disclosure relates to a computer-based method for detecting a malfunction in a machine learning system for environmental control. The method includes receiving data from the machine learning system for environment control and/or data from an environment model of the machine learning system for environment control, receiving context information from one or more external data sources, the context information being the environment that the machine learning system for environment control monitors, describe and decide whether there is a fault based on the data from the machine learning system for environment control and/or the data from the environment model and the context information. If there is a fault, the method includes triggering a response to the fault.

Description

technical field

The present disclosure relates to methods and systems for detecting and mitigating faults in a machine learning system for checking the surroundings, in particular of machine learning systems for checking the surroundings of at least partially autonomously operating vehicles.

background

Machine learning systems for environment control are being used to an increasing extent in various application areas. For example, such systems (such as machine vision systems) are used in autonomous vehicles. In this case, an environment of a vehicle is monitored by means of the machine learning system for environment control. As can easily be seen, a misjudgment of the machine learning system can have significant consequences. The same applies in other areas of application of machine learning systems for environment control, especially when they are embedded in highly automated system environments. One cause of misjudgments can be disturbances such as so-called adversarial attacks (in German "enemy attacks", the common English term will be used below). As part of an adversarial attack, an attacker tries to deceive the machine learning system for environment control by deliberately changing its input data. For example, this can be done by manipulating the environment that the environmental control machine learning system monitors (such as pasting over a street sign or applying false lane markings). In some prior art methods, the machine learning system for environment control is designed to recognize the disturbance itself (e.g. to classify it as such). In some cases, this can require complex hardware, reduce the scope for designing the defense and/or only be effective to a limited extent. It is therefore desirable to improve the methods for detecting a malfunction in a machine learning system for environmental control.

Summary of the Invention

A first general aspect of the present disclosure relates to a computer-based method for detecting a malfunction of a machine learning system for environment control. The method includes receiving data from the machine learning system for environment control and/or data from an environment model of the machine learning system for environment control, receiving context information from one or more external data sources, the context information being the environment that the machine learning system for environment control monitors, describe and decide whether there is a fault based on the data from the machine learning system for environment control and/or the data from the environment model and the context information. If there is a fault, the method includes triggering a response to the fault.

A second general aspect of the present disclosure relates to a system for detecting a malfunction of a machine learning system for environmental control, which is configured to carry out the methods according to the first general aspect.

The techniques according to the first and second general aspects of the present disclosure may have one or more advantages.

First, by using data from the machine learning system for environment control and/or data from an environment model of the machine learning system for environment control, as well as context information, the detection of disruptions (especially adversarial attacks) can be more reliable than in some prior art methods. With some state-of-the-art methods, which take place within a machine learning system for environment control (e.g. in a neural network of an image classifier), certain disorders such as adversarial attacks can only be recognized with difficulty (which are ultimately designed precisely for the machine learning system to deceive). The techniques of the present disclosure utilize data sources that are not available within a machine learning system for environmental control, which may facilitate fault detection.

Second, and as a consequence, by including a variety of external sources, the likelihood of misdetecting a disorder (“false positive result”) can be reduced. With the internal detection in some machine learning systems for environment control, a conservative design for reliable detection of disturbances can lead to a higher number of false positive results.

Third, by using an external system to detect faults, a complexity, and thus error susceptibility, and a price of a machine learning system for environmental control (including the external system to detect faults) can be lower than in solutions in which the faults are in the machines -Learning system for environment control itself should be detected. A single Sys tem to detect a malfunction in a device (e.g. a vehicle) operate several machine learning systems for environment control.

character list

• 1 shows schematically a disruption in the form of an adversarial attack on an at least partially autonomous vehicle;
• 2 Fig. 12 is a flow chart showing steps of the methods of the present disclosure;
• 3 illustrates an exemplary fault detection system of the present disclosure.

Detailed description

In 1 1 schematically shows a fault in an at least partially autonomous vehicle 100 (eg an autonomous vehicle of levels 3 to 5). The techniques of the present disclosure are explained below using examples with at least partially autonomous vehicles. However, the techniques of the present disclosure are not limited to this particular application. Other areas of application are listed below.

The at least partially autonomous vehicle 100 contains a machine learning system for environment control 200 (e.g. a machine learning system for machine vision). The machine learning system for environment control 200 can be coupled to one or more sensors 300 and fed with input data by them. The sensors 300 may provide sensor data regarding the environment that the environmental control machine learning system 200 is monitoring. In particular, the sensors can be camera-based sensors (for the visible and/or infrared wavelength range, e.g. a thermal imaging camera), radar sensors, lidar sensors or ultrasonic sensors. Additionally or alternatively, the input data can be image data (e.g. video image data or frame data). The environmental control machine learning system 200 may include one or more classifiers for assessing the environment (e.g., a classifier based on one or more neural networks). The environmental control machine learning system 200 may include an image classifier for processing image data. However, machine learning systems that are not based on image data can also be used for environment control 200 .

In the example of 1 there is a disturbance in the application of a wrong marking 500 on a roadway. The idea behind this disruption is that the machine learning system for environment control 200 of the at least partially autonomous vehicle 100 interprets the wrong marking 500 as a real marking and the at least partially autonomous vehicle 100 is therefore controlled accordingly. For example, the at least partially autonomous vehicle 100 could be brought to a standstill at the wrong stop line as part of a robbery in order to subsequently rob the occupants.

The methods or systems of this disclosure can be used to detect various types of faults. A disturbance may include a random change in an environment that environmental control machine learning system 200 is monitoring and/or an input signal to environmental control machine learning system 200 (a change in environment will of course also affect an input signal). The disruption of the machine learning system for environment control 200 can be an adversarial attack, for example. As mentioned, an adversarial attack is based on feeding a specially generated input signal (also referred to as an "adversarial example") into the machine learning system for environment control (e.g. an artificial neural network), which is intended to deceive it and, for example, generate a misclassification. The manipulation can be carried out in such a way that a human observer does not notice it or does not recognize it as such. Examples of adversarial attacks are changing the environment that the machine learning system monitors for environmental control (e.g. by attaching, changing, removing and/or covering elements relevant to environmental control, such as the markings in 1 ). In other examples, one or more sensors 300 coupled to environmental control machine learning system 200 may be tampered with or may receive a tampered signal. Disorders other than Adversarial Attacks may also be detected (additionally or alternatively). For example, a disturbance may include a random change in an environment and/or an input signal.

In the example in 1 the machine learning system for environment control 200 is arranged in the at least partially autonomous vehicle 100 (ie in a device whose environment is to be monitored). In other examples, the environmental control machine learning system 200 may be external to the device whose environment is to be monitored, or distributed among the device and the environment.

According to an example of the techniques of the present disclosure, a fault detection system 600 is arranged in the at least semi-autonomous vehicle 100 . The system for detecting a fault 600 is designed to process data from the machine learning system for environment control 200 and/or data from an environment mo dell (not shown). In addition, the fault detection system 600 is designed to receive context information from one or more external data sources 700a,b.

"Context information" describes a current environment that the machine learning system monitors for environmental control. The term “context information” is used in the present disclosure in distinction from machine learning system input data for environmental control (i.e., machine learning system input data for environmental control is not context information). In some examples, the context information may describe a property of the environment or particular aspects of the environment. In addition or as an alternative, the context information can contain information about elements and/or events that are located in or take place in the environment. Context information can be the result of a processing and/or interpretation process (e.g. a classification result of another machine learning system or a human classifier).

In some examples, the context information includes environmental map data. In the present disclosure, “map data” means any data in which certain information is associated with location information (e.g., world coordinates). It is possible, but not necessary, for the map data to also be stored so that it can be displayed as a map. For example, map data can be one or more tuples of coordinates at which a certain element (e.g. a traffic route, a marking or a traffic control element such as a sign) is located. or a specific event has taken place (e.g. an adversarial attack).

The map data may be stored locally on a device in which the environmental control machine learning system 200 is deployed (the environment of which the environmental control machine learning system 200 monitors). In other examples, the map data may be stored on a remote device that is connected via a network to the device in which the machine learning system for environmental control is deployed.

Additionally or alternatively, the context information may include information about known or suspected disruptions (e.g., adversarial attacks) in the environment that the environmental control machine learning system is monitoring. In some examples, there may also be a confidence rating for the known or suspected disruptions (e.g., adversarial attacks). For example, information about known or suspected faults may be collected (in some examples in the form of, or associated with, map data) in a central location (e.g., in a database). The information can then be queried from the central location and, for example, integrated into a local map of the device (e.g. the semi-autonomous vehicle 100).

Additionally or alternatively, the context information can include sensor data from a sensor that monitors the surroundings. In particular, this can be a sensor that is not used by the machine learning system for monitoring the environment. In examples, the sensor may be included in a device other than the device in which the machine learning system is deployed for environmental control (e.g., in another vehicle or piece of infrastructure through which the vehicle 100 is moving).

The external data sources may be external data sources related to environmental control machine learning system 200 . In other words, an external data source can be any data source other than the environmental control machine learning system 200 itself (including its input data). It is not necessary for the external data sources 700a,b to also be outside of a device (in 1 the at least partially autonomous vehicle 100) in which the machine learning system for environment control 200 is used. For example, an external data source 700a can be an (electronic) card that is stored in another system of the at least partially autonomous vehicle 100 . Other examples of external data sources are described below.

In some examples, external data sources may be other devices that collect and/or collect contextual information about the environment that the environmental control machine learning system is monitoring. For example, an external data source can be another at least semi-autonomous vehicle or an infrastructure component or any other device that is located, moves in or has access to the environment that the machine learning system for environmental control monitors (e.g. a smartphone of a road user) In some examples, an external data source can include another machine learning system for environment control (e.g. in another at least semi-autonomous vehicle or in an infrastructure component).

In 2 FIG. 12 is a flow chart showing steps of methods for detecting a disturbance (eg, adversarial attack) of a machine learning system for environmental control of the present disclosure. The methods of the present disclosure are computer-based methods, that is, they are performed on a suitable computer System running (substantially or even fully automatically, ie, without user interaction).

The method includes receiving 202 data from the machine learning system 200 for environment control and/or data from an environment model of the machine learning system for environment control 200. The method also includes receiving 204 context information from one or more external data sources 700a, b and deciding 206 whether there is a fault based on the data from the machine learning system for environment control and/or data from the environment model and the context information, and if there is a fault, triggering 208 a reaction to the fault.

Each of these steps and possible configurations are explained in more detail below.
The decision 206 can include matching the data from the machine learning system for environment control and/or data from the environment model and the context information. For example, a classification result from the machine learning system for environment control can be compared with corresponding context information. In the example of 1 For example, the environmental control machine learning system 200 may recognize the marker 500 (or other element in the environment) as a result of a classification process. A comparison can now be made with map data from the external data source 700a,b (which can contain the position of known markers or other elements). It can be assumed that there is a fault if the comparison shows that the data from the machine learning system for environment control and/or the data from an environment model and the context information do not match with regard to one or more parameters. For example, no marker may be recorded in the map data at the location where the marker 500 was detected.

In other examples (or in addition), the step of deciding may include reading information about known or suspected disturbances in the environment that the machine learning system monitors for environmental control (which may be stored in a map, for example). In addition, a confidence rating for the known or suspected interference can be read out. The information and the optional confidence score can feed into the decision-making process (e.g. the higher a confidence score, the more likely it is to confirm the presence of a fault).

In yet other examples, the decision may include using another machine learning module to analyze the data from the machine learning system for environment control and/or the data from the environment model and/or a rule-based check of the data from the machine learning system for environment control and/or or the data from the environment model.

For example, the further machine learning module (e.g. one or more neural networks) can be trained to recognize known patterns of disturbances (e.g. a dedicated machine learning module for recognizing the disturbances).

In one example, the further machine learning module can receive input data from the machine learning system for environmental control 200 as input data (e.g. sensor data or image data as discussed above) and decide whether there is a fault (e.g., the attachment, removal and/or occlusion of for elements relevant to environmental control). Alternatively or additionally, the further machine learning module can receive data from processing stages of the machine learning system for environment control 200 . For example, the environmental control machine learning system 200 may include various sensor modalities (e.g., two or more of a camera-based sensor, a lidar sensor, and/or a radar sensor). Input data from each of the different sensor modalities can first be processed separately (in a first processing stage) and then merged (in a second processing stage) in the machine learning system for environment control 200 . The further machine learning module can receive the separately processed data from one or more of the sensor modalities and decide whether there is a fault. For example, the further machine learning module can recognize relationships between the processed signals of different sensor modalities that indicate a fault (for example, to distinguish the reflection properties of a real element in the environment from those of a counterfeit and/or manipulated element). Furthermore, additionally or alternatively, the further machine learning module can receive output data from the machine learning system for environment control 200 and/or data from the environment model of the machine learning system for environment control 200 and can decide whether there is a fault. For example, an implausible classification result can be recognized (e.g. an element in the environment with implausible characteristics or implausible behavior, such as an element with an unused label or unphysical behavior when photographed from different directions and/or at different angles).

In addition to or instead of another machine learning module, a rule-based module can also receive the data described above and decide whether there is a fault based on a set of rules. For example, a rule-based plausibility check can be carried out.

Deciding 206 whether a fault is present can produce a variety of outcomes. In some examples, the disorder detection system 600 may be configured to make a binary decision as to whether a disorder (e.g., an adversarial attack) is present. This can be done separately for one or a plurality of types of faults. In other examples, a probability that a fault is present can be determined. Again, this can be done separately for one or a plurality of types of faults. In both cases, after one of the steps described above has been carried out, a probability or another evaluation parameter can be increased or decreased, depending on whether signs of a fault are detected or not. In the case of a binary decision, the presence of a fault can be determined if an evaluation parameter exceeds (or falls below) a certain threshold value.

The steps described above for deciding whether there is a fault can also be combined in any way. In this way, the decision can be made in several stages. For example, in one or more first steps, the data and context information that is present within a device (e.g. a vehicle) can be processed (e.g. map data and/or data from the machine learning system for environment control and/or data from the environment model of the machine learning system for environmental control). Then, in further steps, context information from external sources outside the device (e.g. other vehicles or road users, infrastructure elements or remote databases) can be received and processed. The first step can be used to estimate the probability of an error, while the second step for plausibility checking can only be carried out if the probability of an error is above a certain probability (if the probability after the first step is too low, the system can decide that there is no fault).

If the existence of a fault has been determined, an assessment of the situation can be made.

This step can include, for example, a criticality assessment of the fault (e.g. whether the fault is critical for the operation of the at least semi-autonomous vehicle and/or for the safety of its occupants). In other examples, the step may include a confidence assessment of the likelihood of interference. A response to the presence of a disturbance may be selected taking into account the criticality score and/or the confidence score. For example, a fault that endangers the operational safety of the vehicle (e.g. could provoke an accident) and a fault that endangers the safety of the occupants or the cargo (e.g. brings the vehicle to a standstill in order to rob the occupants) can be distinguished and corresponding mitigation methods (described in more detail in the next section) can be selected.

In response to the presence of a failure, one or more mitigation methods to mitigate the failure may be performed.

In some examples, a mitigation method to mitigate the interference may include adjusting a selection and/or weighting of sensors coupled to the environmental control machine learning system. For example, a particular sensor that is (particularly) affected by the fault may be temporarily disabled or its input data may not be considered by the environmental control machine learning system 200 . Alternatively or additionally, a configuration of a particularly affected sensor can be changed in order to ward off the disturbance. In still other examples, a first set of sensors (e.g., one or more camera-based sensors) may be switched to a second set of sensors (e.g., a lidar and/or radar sensor) to mitigate the interference.

In other examples or additionally, the environment model of the machine learning system can be adapted for environment control. For example, an incorrectly recognized or manipulated element of the environment cannot be adopted or replaced by another element (e.g. an incorrectly recognized speed limit with a correct speed limit).

In other examples or additionally, one or more operating parameters of the environmental control machine learning system or a device that the environmental control machine learning system is used to control and/or monitor may be adjusted. This can be, for example, a behavioral adaptation of the machine learning system for controlling the environment or a device for the control and/or monitoring of which the machine learning system for controlling the environment is used. In the example of an at least partially autonomous vehicle, a behavioral adjustment of a speed adaptation (e.g. braking) or carrying out a specific driving maneuver. In other examples, behavior adjustment may include route adjustment (eg, in a random fashion). The latter can happen when an attack affecting the safety of the occupants of the vehicle is detected.

In yet other examples, one or more recommended mitigation actions may be performed by an external entity (“contingency plan”).

Alternatively or additionally, in response to the presence of a fault, a warning of the fault can be generated.

In some examples, the warning may include alerting an external entity to the existence of a disruption (e.g., alerting a security service provider or official security bodies). This alerting can be done using any suitable communication channel (e.g., a cellular channel or other wireless network).

Additionally or alternatively, the environment of the machine learning system can be warned for environment control. If the machine learning system is used to monitor the surroundings in a vehicle, the warning can be transmitted to other vehicles, road users and/or infrastructure components. Again, this alert may be via any suitable communication channel (e.g., a cellular channel or other wireless network). In one example, vehicles in the vicinity of the vehicle that detected the disruption may be alerted (e.g., oncoming vehicles). As described above, an incident warning represents contextual information that may be received by an incident detection system (e.g., other vehicles, road users, infrastructure components, and/or central offices) and used according to the techniques of the present disclosure (again ) can be used to detect faults.

Additionally or alternatively, a user of a device whose environment is monitored by the machine learning system for environment control can be warned. For example, an operator, driver and/or passenger of the device can be warned. In semi-autonomous vehicles, the driver and/or passenger may be prompted to take control and/or perform other safety routines.

Additionally or alternatively, information on the presence of a fault can be transmitted to a central location (optionally with further information such as position, information on the type of fault and/or confidence information on detecting the fault). The information may be communicated to the central location using any suitable communications channel (e.g., a cellular channel or other wireless network). The information can then in turn be made available to other devices (e.g. other vehicles) for the purpose of deciding on the existence of a fault.

Further alternatively or additionally, in response to the presence of a fault, the presence of the fault can be checked for plausibility (or discarded). As already described above, the explained steps for deciding whether a fault is present can also be used for this purpose.

In some examples, validating the presence of the disturbance includes validating using an external data source (e.g., a data source connected to the device in which the machine learning system for environmental control 200 is deployed via a communication channel).

In some cases, the plausibility check can include a request for an update of context information, in particular map data. The updated context information, in particular map data, can then be made available by an external provider (e.g. an OEM, an authority or a map provider). In some examples, the update may include updated information about known or suspected disturbances in the environment that the environmental control machine learning system is monitoring. The updated context information, in particular map data, can be used with one of the techniques described herein for checking (or rejecting) the presence of the disturbances.

Alternatively or additionally, the plausibility check can include communication with a further device located outside of a device in which the machine learning system is used for environment control. This further device can be an infrastructure component (eg part of an intelligent traffic control system or other infrastructure components). The plausibility check can include receiving further context information from the infrastructure component (which can then be processed as described herein to plausibility check the existence of a fault) or a direct confirmation of the fault by the infrastructure component. In the same way, other devices can be integrated into the process as infrastructure components (e.g. other vehicles, other road users, and/or remote databases).

In addition to the processes in the device in which the machine learning system is used for environment control (whose environment the machine learning system for environment control monitors), in 2 also processes in the machine learning system for environment control (in the left column) and an external data source (in the right column) illustrated. The machine learning system for environment control can observe the environment using one or more sensors 302. The sensor data can be made available as input data to a model of the machine learning system for environment control and processed 304 in it to assess the environment (e.g. by means of one or more neural networks). As described above, data from the machine learning system for environmental control are made available to the system for detecting a fault 306. The machine learning system for environmental control can also create and/or modify an environment model based on the assessment of the environment 308. The external source can be in in some cases a database containing context information. in 2 In the case shown, the external data source also has a system for assessing the environment (eg another machine learning system for environment control with coupled sensors). In this case, sensor data of the surroundings can be collected 310, processed 312 by means of suitable evaluation steps and stored 314 as context information and/or made available 316 to the system for detecting a fault.

3 illustrates an exemplary fault detection system 800 of the present disclosure.

In general, the system for detecting a failure of a machine learning system for environmental control can be configured to carry out the methods described herein. Any suitable hardware, software or combination of both can be used for this purpose. In some examples, the fault detection system includes a memory that contains appropriate instructions for performing the methods described herein and one or more processors that execute the instructions. The system for detecting a fault can be arranged locally in a device in which the machine learning system for environment control 200 is used (e.g. in an at least semi-autonomous vehicle). In other examples, the fault detection system may be located remotely from a device employing the environmental control machine learning system 200 (and optionally communicate with the device and/or the environmental control machine learning system 200 via a network). . The system for detecting a fault can have dedicated hardware, be integrated into a higher-level system or be spread over several locations.

In general, the present disclosure is also directed to an apparatus (e.g., an at least semi-autonomous vehicle) that includes a machine learning system for environment control; and a fault detection system (as described herein). The fault detection system is designed to detect a fault on the environmental control machine learning system.

In general, the present disclosure is also directed to a computer program configured to perform all steps of the methods of the present disclosure. In general, the present disclosure is also directed to a machine-readable storage medium (e.g., a FLASH memory or other portable storage medium) on which the computer program is stored, or a signal sequence encoding the computer program.

coming back on 3 For example, the fault detection system 800 connects to various external data sources to receive appropriate context information. These external data sources include a central location 810 that provides context information regarding known or suspected faults. This context information may be stored in a dynamic map 820 (ie, a map that is regularly (continuously) updated externally) and/or processed directly by the fault detection system 800 . In addition, the fault detection system 800 can access a static map 830 (eg a map stored in the vehicle) in order to obtain further context information. In addition to the context information, the system for detecting a fault 800 accesses sensors 840, which also input data for a machine learning system for environment control (not in 3 shown) deliver, too. In the example of 3 sensors 840 include a camera-based sensor, a radar sensor, and a lidar sensor. In addition to the sensors, the system for detecting a fault 800 accesses an environment model 850 of the device 860 in which the machine learning system is used for environment control. The available data is processed in the fault detection system 800 as described in the present disclosure to decide if there is a fault. The system for Depending on the outcome of the decision, detection of a fault 800 can initiate mitigation methods (eg by adapting the environment model 850 and/or adapting a behavior of the device 860). In addition, the result of the decision can be fed back to the central office 810 .

In general, the data sources of the present disclosure may be updated periodically (i.e., continuously). This is particularly true for data sources located in the device (e.g., a vehicle) in which the environmental control machine learning system 200 is deployed (e.g., map data). The process of updating may include an update via an appropriate communication channel (e.g., a wireless communication channel, possibly while the vehicle is in operation).

In the foregoing discussion, the example of an at least semi-autonomous vehicle (particularly an autonomous vehicle) has often been used to explain the techniques of the present disclosure. The vehicle may be, for example, an automobile, bus, truck, ship or boat, airplane (including helicopters and spacecraft), train, tram, or motorcycle.

However, the techniques of the present disclosure are not limited to these use cases, but can be used for all devices in which machine learning systems for environmental control can be found (unless a corresponding aspect of the disclosure is dedicated only to at least semi-autonomous vehicles applicable).

In addition to at least partially autonomous vehicles, techniques of the present disclosure can be used, for example, to detect faults in a machine learning system for environment control in other means of transport (for passengers and/or freight).

In other examples, the techniques of the present disclosure may be used, for example, to detect failures of a machine learning system for environmental control in robotic devices (e.g., in industrial robots, scouting robots, household and/or tool robots).

In other cases, the techniques of the present disclosure may be used in devices for monitoring, for example, to detect failures of a machine learning system for environmental control. These monitoring devices can be used, for example, to monitor traffic situations, systems or machines (e.g. devices for inspection, process monitoring, and/or to provide security functions) and for security applications (e.g. to monitor objects or public or private spaces).

For example, the techniques of the present disclosure can be used, for example, to detect failures of environmental control machine learning systems in devices for monitoring and/or directing traffic flows (e.g., infrastructure components). This can be, for example, camera-based machine learning systems for environment control.

Claims (12)

Computer-based method for detecting a malfunction of a machine learning system for environmental control (200), comprising: receiving (202) data from the machine learning system for environment control and/or data from an environment model of the machine learning system for environment control; receiving (204) context information from one or more external data sources (700a,b), the context information describing an environment that the machine learning system monitors for environment control; Decide (206) whether there is a fault using the data from the machine learning system for environment control and/or the data from the environment model and the context information, and If the presence of a fault is detected, triggering (208) a response to the fault. Computer-based method according to claim 1 , wherein the decision includes matching the data from the machine learning system for environment control and/or the data from the environment model and the context information, and optionally assuming that there is a fault if the matching shows that the data from the machine learning system for Environmental control and/or the data from an environmental model and the context information do not match in relation to one or more parameters. Computer-based method according to claim 1 or 2 , wherein the decision involves using a further machine learning module to analyze the data from the machine learning system for environment control and/or the data from the environment model and/or a rule-based check of the data from the machine learning system for environment control and/or the data from the environment model. Computer-based method according to one of Claims 1 until 3 , wherein the context information includes map data of the environment that the machine learning system monitors for environment control. Computer-based method according to one of Claims 1 until 4 , wherein the context information includes information about known or suspected disturbances in the environment that the environmental control machine learning system is monitoring. Computer-based method according to one of Claims 1 until 4 , wherein the context information is received from external data sources outside of a device (100) in which the machine learning system is used for environment control. Computer-based method according to one of Claims 1 until 6 , wherein the data from the machine learning system for environment control include input data of the machine learning system for environment control, in particular image data, and/or data from processing stages of the machine learning system for environment control, in particular output data from processing units of different sensor modalities. Computer-implemented method according to one of Claims 1 until 7 , wherein the decision as to whether a fault is present comprises a criticality assessment of the fault and/or a confidence assessment of the probability of the fault, and in particular the reaction is selected taking into account the criticality assessment and/or confidence assessment. Computer-implemented method according to one of Claims 1 until 8th , wherein the reaction to the presence of a fault comprises carrying out a mitigation method to ward off the fault and/or generating a warning of the fault and/or checking the fault for plausibility using an external data source. System for detecting a malfunction of a machine learning system for environment control (600; 800), which is designed to implement the methods according to one of Claims 1 until 9 to execute. An apparatus comprising: a machine learning system for environment control (200); and the fault detection system according to FIG claim 10 , wherein the system for detecting a fault is designed to detect a fault in the machine learning system for environmental control. Machine-readable storage medium on which the computer program is stored, which is set up to carry out all steps of the method according to one of Claims 1 until 9 execute, or signal, which encodes a computer program, which is set up, all steps of the method according to one of Claims 1 until 9 to execute.

Images