EP4066072A1

EP4066072A1 - Module-prioritization method, module-prioritization module, motor vehicle

Info

Publication number: EP4066072A1
Application number: EP20808053.1A
Authority: EP
Inventors: Nico Maurice SCHMIDT; Peter Schlicht
Original assignee: Volkswagen AG
Current assignee: Volkswagen AG
Priority date: 2019-11-29
Filing date: 2020-11-13
Publication date: 2022-10-05
Also published as: WO2021104904A1; DE102019218614A1

Abstract

The present invention relates to a module-prioritization method (1; 10) for a redundant set of modules (21) of a hardware system (20), comprising: determining a reliability of at least one module of the redundant set of modules (21), the reliability being determined on the basis of a learning algorithm, in order to prioritize activation of a module of the redundant set of modules (21).

Description

description

"Module prioritization procedure, module prioritization module, motor vehicle"

The invention relates to a module prioritization method, a module prioritization module, and a motor vehicle.

Typically, safety systems (for example braking systems in a motor vehicle) can be redundant so that if one system fails, another is available which fulfills the same tasks.

However, it is possible that one of the redundant systems will function more reliably than the other. Such a reliability can depend, for example, on external circumstances, environmental influences and the like. It can therefore be advantageous to activate a first of the redundant systems in certain first environmental influences, while a second remains deactivated and is only available in an emergency or if the first system fails, with other environmental influences it being desirable for the second system to be activated while the first remains deactivated and is only available in the event of an emergency or failure of the second system.

The decision as to which system should be activated and which should be deactivated can be made by an artificial intelligence.

For example, US Pat. No. 9,852,475 B1 discloses a method which estimates an accident risk based on reliability. However, a redundant set of modules is not checked here. In addition, the reliability is also not determined based on a learning algorithm.

Systems and methods for object detection of a radar image are known from the patent application US 2016/0019458 A1. Object detection is based on network algorithms of a deep neural network. However, the reliability of a module is not determined here. A neural network which controls various autonomous devices is known from US Pat. No. 7,266,532 B2. Here, however, no reliability is determined based on a learning algorithm, but rather a correspondence of an object recognition of the autonomous devices is determined.

A vehicle is known from US Pat. No. 7,386,372 B2 which recognizes by means of object recognition whether a passenger is on a vehicle seat. However, the reliability of a module is not determined here.

The object of the present invention is to provide a module prioritization method, a module prioritization module, and a motor vehicle which at least partially overcomes the above-mentioned disadvantages.

This object is achieved by the module prioritization method according to the invention according to claim 1, by the module prioritization module according to the invention according to claim 12 and by the motor vehicle according to the invention according to claim 14.

According to a first aspect, an inventive module prioritization method for a redundant set of modules of a hardware system comprises: determining a reliability of at least one module of the redundant set of modules, the reliability being determined based on a learning algorithm for prioritizing an activation of a module of the redundant set of modules.

According to a second aspect, a module prioritization module according to the invention is set up to carry out a module prioritization method according to the first aspect.

According to a third aspect, a motor vehicle according to the invention has a module prioritization module according to the second aspect.

Further advantageous embodiments of the invention emerge from the subclaims and the following description of preferred exemplary embodiments of the present invention.

As already discussed, redundant modules, for example as a safety measure, can be provided, for example in the context of autonomous driving. In the case of autonomous driving of a motor vehicle, a decision can be made as to which of the redundant modules should be prioritized (or which should be trusted).

For this purpose, as is generally known, a redundancy resolution can be carried out for data-driven (intelligent) modules by means of a reliability assessment.

In this case, for example, an indicator for a reliability is determined (for example with a Monte Carlo dropout), but this has the disadvantage that such methods require a great deal of computing effort and are therefore typically time-consuming. Therefore, such methods typically cannot be carried out in real time (e.g. during an autonomous journey).

It is also known to perform a Monte Carlo dropout in a late layer of a neural network. However, not enough time is typically saved here, so that real-time implementation with such methods is also not possible.

Therefore, some exemplary embodiments relate to a module prioritization method for a redundant set of modules of a hardware system, comprising: determining a reliability of at least one module of the redundant set of modules, the reliability being determined based on a learning algorithm for prioritizing activation of a module of the redundant set of modules.

The hardware system can comprise any hardware system which is set up to generate and / or evaluate data.

For example, the hardware system can comprise several modules, for example a sensor, a control unit, an on-board computer, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a braking system, a safety system, and the like.

A hardware system according to the invention typically comprises at least two modules which form a set according to the invention, wherein the two modules can replace one another. This means that one module can be active while the other is inactive and vice versa without any basic functionality of the hardware system being restricted or lost. For example, two brake systems can be present in a motor vehicle, so that one brake system is active or activated in the event of failure of the other brake system, whereby driving safety can advantageously be ensured.

Such redundancy (multiple availability of a module) can typically be provided in safety modules (for example a motor vehicle), but the present invention is not limited to safety modules.

For example, a redundant set of distance sensors (for example lidar, radar), cameras (for example stereo cameras) and the like can also be provided in order to enable (partially) autonomous driving of a motor vehicle. In addition, comfort modules (for example infotainment modules, heating and air conditioning control modules), vehicle interior monitoring modules, and the like can also be redundant. In this respect, the present invention can always be used when there is or can be a redundant set of modules, in particular also in a technical environment that does not concern motor vehicles, such as medical technology, medical robotics, aviation, shipping, rail transport, space travel, and the like.

A module prioritization method according to the invention can include determining a reliability of at least one module of the redundant set of modules.

The reliability can be an indicator for an accuracy or for a reliability of the at least one module.

For example, it can be the case that the at least one module, depending on environmental influences (for example temperature, time of day, air pressure, and the like) produces a different data output than would be the case with a reference measurement. For example, a measured value (for example an oil pressure) can only be imprecise (ie an amount of a difference between an oil pressure measured by the at least one module and an actual oil pressure is above a threshold value or outside a tolerance range), while the measured value is at other values Environmental influences is reliable (ie an amount of a difference between an oil pressure measured by the at least one module and an actual oil pressure is below a threshold value or within a tolerance range). The reliability can be determined for at least one module of the redundant set of modules. However, the reliability can also be determined for more than one to all modules of the redundant set of modules.

In some exemplary embodiments, the reliability is determined based on a learning algorithm. Typically, the learning algorithm is implemented by means of an artificial intelligence, which can be provided on the at least one module, on another module of the redundant set of modules, or on a module prioritization module specially provided for this purpose.

The module prioritization module can have an artificial intelligence which determines a reliability of a perception module which, for example, carries out object recognition based on sensor data and the like. In this context, an artificial intelligence can also be implemented in the perception module, the reliability of which can be determined, for example, with a Monte Carlo droput (see below).

In this way, a reliability can advantageously be determined in a computationally efficient manner.

As already mentioned, the learning algorithm can be applied by an artificial intelligence (Kl) which, for example, uses methods based on machine learning, deep learning, explicit feature, and the like, such as pattern recognition, edge detection, a histogram-based method, pattern matching (Pattern Matching, Color Match, and the like.

This has the advantage that known methods for generating a KI can be used.

In some exemplary embodiments, the learning algorithm comprises machine learning.

In such exemplary embodiments, the learning algorithm can be based on at least one of a scale invariant feature transform (SIFT), a gray level co-occurrence matrix (GLCM), and the like.

The machine learning can also be based on a classification method, such as at least one of random forest, support vector machine, neural network, Bayesian network, and the like, such deep learning methods, for example at least one of autoencoder, generative adversarial network, weak supervised learning, boot strapping, and the like.

Furthermore, machine learning can also be based on data clustering methods, such as, for example, density-based spatial clustering of applications with noise (DBSCAN), and the like.

The supervised learning can also be based on a regression algorithm, a perceptron, a Bayesian classification, a naive Bayesian classification, a closest-neighbor classification, an artificial neural network, and the like.

In this way, known methods for machine learning can advantageously be used.

In some exemplary embodiments, the reliability can be determined in a training session, so that computing power can advantageously be saved when the module prioritization method according to the invention is used.

In addition, it is advantageous that sufficient time can be available in a training session to determine the reliability.

The certain reliability can be stored, for example, in a data memory, determined by the artificial intelligence, with advantageously no further training being necessary, or with an existing training being able to be continued, whereby the determination of the reliability can advantageously be improved.

The specific reliability can then be determined, that is to say, for example, can be retrieved from the data memory, from the artificial intelligence, and the like, so that, based on the reliability for the at least one module, a decision can be made about whether the module is activated or remains activated), or whether another module of the redundant set of modules is activated (or remains activated), wherein a reliability can also be determined for the other module. However, the present invention is not limited to such a case. For example, it can be predefined that a first module should in no case be activated when the reliability is below a predefined threshold value, so that a second module is automatically activated (or remains). In this respect, prioritization is to be understood as meaning that activation of a first module can be preferred to activation of a second module. As already discussed, the reliability can be determined again (for example in the event of a change in environmental influences), so that activation of the second module can then be preferred to activation of the first module.

In some exemplary embodiments, the learning algorithm is based on a Monte Carlo dropout.

Typically, a Monte Carlo dropout can be used for an artificial neural network during training.

A large number of iterations are carried out here, with impressions being presented to the neural network at each iteration. In the context of this invention, an impression can be understood as a measured value or an output date (or output data) of a sensor, an output of a control device, and the like. In addition, some neurons of the neural network are switched off pseudo-randomly with each iteration.

The neural network can, for example, determine a measured value of a sensor in each iteration based on the output data.

In general, the determined value can be different for each iteration. A mean value over all measured values can be formed, a mean deviation of the mean value (for example Gaussian error, standard deviation, half-width, and the like) being a measure of the reliability.

It is advantageous here that, given a sufficiently high number of iterations, a statistically significant or relevant value for the reliability can be determined.

Based on a reliability determined in such a way, a reliability label can be generated which, for example, can be used by a further artificial intelligence as an input to determine the reliability in a more efficient way than with a Monte Carlo dropout.

For example, when the hardware system is operated (outside of a training phase), a Monte Carlo dropout does not have to be carried out every time, but rather only the generated reliability labels are taken into account, whereby computing power can advantageously be saved.

This has the advantage that unannotated training data can be used to learn the reliability, since the reliability labels can be generated externally if the Monte Carlo dropout is outsourced (in another module).

In addition, there is the advantage that the Monte Carlo dropout can be carried out with a high intensity (in a statistically relevant number of repetitions), since a high latency can be accepted in a training session. This also advantageously results in a high sampling rate with which the Monte Carlo dropout can be carried out than when it is started up by a user (for example in road traffic).

Overall, a larger sensor impression can be presented to an artificial intelligence than with known methods, so that a more precise work of the artificial intelligence is possible.

In some embodiments, the redundant set of modules includes at least one of the controller, sensor, processor, and security system as described herein.

In some exemplary embodiments, the module prioritization method further comprises: selecting a module of the redundant set of modules based on the determined reliability; and activating the selected module.

As described herein, the module can be selected by the further artificial intelligence.

In addition, the selected module does not have to correspond to the at least one module in every case. As mentioned above, the reliability can be determined for at least one module, while, however, a further module is selected and activated without the reliability having been determined.

In some exemplary embodiments, the reliability is further determined based on a monitored training of the at least one module, as described herein. In some exemplary embodiments, the module prioritization method further comprises: using an input of the monitored training to determine the reliability of the at least one module, as described herein.

Here, a module prioritization module can use input from a perception module (as described herein).

An activation of at least one intermediate layer of the perception module for determining the reliability can also serve as an input in a module prioritization module.

A combination of an input and at least one intermediate layer of the perception module can also be considered.

In some exemplary embodiments, the module prioritization method further comprises: using an output of the monitored training to determine the reliability of the at least one module, as described herein (for example using multiple reliability labels).

In some exemplary embodiments, the learning algorithm is carried out with a neural network, as described herein.

In some exemplary embodiments, the neural network comprises a convolution network.

In this way, an efficient method of calculation can advantageously be achieved through parameter sharing.

In some embodiments, reliability is determined for at least one of object recognition, semantic segmentation, free space recognition, and depth estimation.

In the case of object recognition, a predetermined object, a pattern, a shape, and the like can be recognized based on the methods described above (for example, pattern matching).

With semantic segmentation, for example, a classification can take place for each pixel or for a group of pixels of a camera. In the case of free space detection, the absence of an object or a distance, an area, a volume, and the like, between a first object (for example a vehicle) and a second object can be detected.

Such a distance can also be determined in the case of a depth estimate.

In this way, (partially) autonomous driving can advantageously be made possible.

In some exemplary embodiments, the reliability further comprises a pixel-precise reliability and / or a bounding box value.

The pixel-precise reliability can comprise an output value of a pixel or a group of pixels for a semantic segmentation.

A bounding box value can include a threshold value which enables a statement to be made about the extent to which a recognized object deviates from a predetermined object (or pattern, and the like) for object recognition, free space recognition, and / or depth estimation.

Some exemplary embodiments relate to a module prioritization module which is set up to carry out a module prioritization method according to the invention.

The module prioritization module can be a control unit, a computer, a CPU, a GPU, an FPGA (field-programmable gate array), which has an artificial intelligence, as described herein, or which is set up to execute an algorithm or a method which or which is based on an artificial intelligence training.

In some exemplary embodiments, the module prioritization module is also set up to determine the reliability based on a reliability label of a perception module, as described herein.

The perception module can be a first artificial intelligence which determines a reliability based on a Monte Carlo dropout, as described herein. Some exemplary embodiments relate to a motor vehicle which has a module prioritization module according to the invention, as described herein.

The motor vehicle can refer to any vehicle operated by an engine (e.g. internal combustion engine, electric machine, etc.), such as an automobile, a motorcycle, a truck, a bus, agricultural or forestry tractors, and the like, where, As described above, the present invention is not intended to be limited to an automobile.

In some exemplary embodiments, the motor vehicle further comprises a perception module for generating a reliability label for the module prioritization module, as described herein.

Embodiments of the invention will now be described by way of example and with reference to the accompanying drawing, in which:

1 shows an exemplary embodiment of a module prioritization method according to the invention in a block diagram;

2 shows a further exemplary embodiment of a module prioritization method according to the invention in a block diagram; and

Fig. 3 shows an embodiment of a motor vehicle according to the invention in a block diagram.

An embodiment of a module prioritization method 1 according to the invention is shown in FIG. 1 in a block diagram.

2, a reliability of at least one module of a redundant set of modules of a hardware system is determined, the reliability being determined based on a learning algorithm for prioritizing activation of a module of the redundant set of modules, as described herein.

Another exemplary embodiment of a module prioritization method 10 according to the invention is shown in a block diagram in FIG. 11, a reliability of at least one module of a redundant set of modules of a hardware system is determined, the reliability being determined based on a learning algorithm for prioritizing activation of a module of the redundant set of modules, as described herein.

In Fig. 12, a module of the redundant set of modules is selected based on the determined reliability, as described herein.

In Figure 13 the selected module is activated as described herein.

3 shows an exemplary embodiment of a motor vehicle 20 according to the invention.

The motor vehicle 20 has a redundant set of emergency braking systems 21. In addition, the motor vehicle 20 has a perception module 22 which is set up to determine the reliability of the hazard braking systems of the set of hazard braking systems 21 with the aid of a Monte Carlo dropout. Furthermore, the perception module 22 is set up to generate a reliability label and to transmit this to a module prioritization module 23 comprised by the motor vehicle 20, which, on the basis of the label, learns to prioritize an emergency braking system.

The module prioritization module 23 is also set up to transmit an activation command to a controller 24, which then activates one of the emergency braking systems.

LIST OF REFERENCE NUMERALS Module prioritization method Determine reliability Module prioritization method Determine reliability Select module Activate module Motor vehicle Redundant set of hazard braking systems Perception module Module prioritization module Control

Claims

A module prioritization method (1; 10) for a redundant set of modules (21) of a hardware system (20), comprising:

Determining a reliability of at least one module of the redundant set of modules (21), the reliability being determined based on a learning algorithm for prioritizing activation of a module of the redundant set of modules (21).

2. module prioritization method (1; 10) according to claim 1, wherein the learning algorithm is based on a Monte Carlo dropout.

3. Module prioritization method (1; 10) according to one of the preceding claims, wherein the redundant set of modules (21) comprises at least one of the control device, sensor, processor and safety system.

4. Module prioritization method (1; 10) according to one of the preceding claims, further comprising:

Selecting one of the redundant set of modules (21) based on the determined reliability; and activating the selected module.

5. Module prioritization method (1; 10) according to one of the preceding claims, wherein the reliability is further determined based on a monitored training of the at least one module.

6. Module prioritization method (1; 10) according to claim 5, further comprising:

Using an input of the monitored training to determine the reliability of the at least one module.

7. module prioritization method (1; 10) according to any one of claims 5 and 6, further comprising:

Using an output of the monitored training to determine the reliability of the at least one module.

8. module prioritization method (1; 10) according to any one of the preceding claims, wherein the learning algorithm is carried out with a neural network.

9. module prioritization method (1; 10) according to claim 8, wherein the neural network comprises a convolution network.

10. Module prioritization method (1; 10) according to one of the preceding claims, wherein the reliability for at least one of object recognition, semantic segmentation, free space recognition, and depth estimation is determined.

11. Module prioritization method (1; 10) according to claim 10, wherein the reliability further comprises a pixel-precise reliability and / or a bounding box value.

12. Module prioritization module (23) which is set up to carry out a module prioritization method (1; 10) according to one of the preceding claims.

13. Module prioritization module (23) according to claim 12, which is further configured to determine the reliability based on a reliability label of a perception module

(22) to be determined.

14. Motor vehicle (20), which has a module prioritization module (23) according to one of claims 12 and 13.

15. Motor vehicle (20) according to claim 14, which further comprises a perception module (22) for generating a reliability label for the module prioritization module

(23).