DE102021107458A1 - Control device and method - Google Patents

Abstract

A method for controlling actuators (61, 62) of a vehicle (10) using a control device (30), which control device (30) has at least one input (31) for receiving vehicle data (53, 54), which control device (30) has at least one output (32) for outputting target values (63, 64) for actuators (61, 62), which control device (30) has an agent (20), a neural network (43) with network parameters (NP) for the neural network (43) and a reward generating device (65) and is designed to determine target values (63, 64) for the actuators (61, 62), has the following steps: A) The vehicle data (53, 54) are the at least one input (31) is detected, and a current vehicle status (41) is calculated;B) the agent (20) arranges the current vehicle status according to a strategy implemented with the neural network (43), depending on the network parameters (NP). (41) an action (42) to;C) In Depending on the action (42), updated target values (63, 64) for the actuators (61, 62) are determined and output via the at least one output (32); D) depending on the change in the vehicle data (53, 54) in As a result of the updated target values (63, 64), the reward generation device (65) uses a reward function to determine a reward value (66), which can be positive or negative;E) the network parameters (NP) are determined by reinforcing learning as a function of the reward value ( 66) in order to increase the expected reward values (66) when using the neural network (43).

Description

The invention relates to a control device and a method for controlling actuators of a vehicle.

the DE 10 2019 104 966 A1 FIG. 1 shows a self-learning controller configured to initialize a data set including data identifying one or more previously recorded input signals, provide a reward function, and train a neural network for the reward function.

the DE 10 2006 054 425 A1 shows a method for determining a value of a model parameter of a vehicle reference model, in which an estimated value of the model parameter is determined as a function of a driving state variable using an artificial neural network, which neural network is adapted by a learning method.

the DE 10 2017 007 136 A1 shows a device for training self-learning algorithms for an automated vehicle.

the DE 10 2020 106 936 A1 FIG. 1 shows a system for controlling the operation of a vehicle with a supervisory control module configured to receive error data.

the DE 11 2016 003 350 T5 shows a method for deep reinforcement learning and training of an actor neural network in which current values of parameters of a critical neural network are updated using errors between the target neural network outputs and the neural network outputs.

It is therefore an object of the invention to provide a new method for controlling actuators of a vehicle and a new control device.

This object is solved by the subject matter of claim 1 and claim 13.

1. A) Die Fahrzeugdaten werden über den mindestens einen Eingang erfasst, und es wird ein aktueller Fahrzeugzustand berechnet;
2. B) Der Agent ordnet gemäß einer mit dem neuronalen Netzwerk implementierten Strategie in Abhängigkeit von den Netzwerkparametern dem aktuellen Fahrzeugzustand eine Aktion zu;
3. C) In Abhängigkeit von der Aktion werden aktualisierte Sollwerte für die Aktoren bestimmt und über den mindestens einen Ausgang ausgegeben;
4. D) In Abhängigkeit von der Änderung der Fahrzeugdaten in Folge der aktualisierten Sollwerte wird durch die Belohnungserzeugungsvorrichtung mit Hilfe einer Belohnungsfunktion ein Belohnungswert ermittelt, welcher positiv oder negativ sein kann;
5. E) Die Netzwerkparameter werden durch bestärkendes Lernen in Abhängigkeit vom Belohnungswert geändert, um durch die Änderung die bei Anwendung des neuronalen Netzwerks zu erwartenden Belohnungswerte zu erhöhen.

A method for controlling actuators of a vehicle using a control device, which control device has at least one input for receiving vehicle data, which control device has at least one output for outputting target values for actuators, which control device has an agent, a neural network with network parameters for the neural network , and has a reward generation device and is designed to determine target values for the actuators, has the following steps:

1. A) The vehicle data are recorded via the at least one input, and a current vehicle status is calculated;
2. B) The agent assigns an action to the current vehicle state according to a strategy implemented with the neural network depending on the network parameters;
3. C) Depending on the action, updated target values for the actuators are determined and output via the at least one output;
4. D) Depending on the change in the vehicle data as a result of the updated target values, a reward value, which can be positive or negative, is determined by the reward generation device using a reward function;
5. E) The network parameters are changed by reinforcement learning depending on the reward value in order to increase the reward values to be expected when using the neural network.

The application of a neural network with reinforcement learning has led to an exceptionally good behavior of the agent in tests, the quality of which has at least partially surpassed the quality of previous control devices.

According to a preferred embodiment, the reinforcement learning is performed as model-free reinforcement learning. The model-free design reduces the risk of deterioration due to a faulty model, and the quality has proven to be particularly good.

According to a preferred embodiment, the agent is given first network parameters, which first network parameters are previously determined by reinforcement learning with the agent or with another agent with the same neural network structure. This enables a well-functioning agent from the start and reduces the time for further optimization.

According to a preferred embodiment, a simulation of a vehicle is created and the first network parameters are determined using the simulation. Using a simulation is significantly cheaper to start with than actual test drives, and special situations can also be used for learning.

According to a preferred embodiment, the agent for the simulation in a Vehicle real measured vehicle data supplied. This can be, for example, vehicle data from a wheel sensor or measured data on the condition of the road.

According to a preferred embodiment, the network parameters are also changed in the installed state of the control device in a vehicle by reinforcement learning depending on the reward value, the reward value being determined depending on the actual influencing of the vehicle data by the agent in order to change the network parameters Application of the neural network to increase expected reward values. The reinforcement learning can thus continue to take place at the end customer, and the neural network can be adapted to improve the driving characteristics of the end customer. Lifelong learning can be realized through the reinforcing learning in the built-in state.

According to a preferred embodiment, the network parameters are also changed while the vehicle is moving. This enables quick adaptation to current versions, for example to a new type of subsurface.

According to a preferred embodiment, the control device has a data interface, and second network parameters are sent to the control device when it is installed in a vehicle. Due to the presence of a data interface, the vehicle manufacturer can send improved network parameters to the vehicles and, if necessary, also evaluate network parameters of the vehicles and use them for optimization.

According to a preferred embodiment, the second network parameters are dependent on a plurality of other control devices from different other vehicles, and the control device is designed to change the network parameters as a function of the second network parameters. This allows the entire fleet to be optimized.

According to a preferred embodiment, the network parameters are changed in the neural network with the aim of increasing the probability of a specified action of the strategy given a specified vehicle state if the performance of this specified action in the specified vehicle state resulted in a positive reward value. This speeds up machine learning.

According to a preferred embodiment, the network parameters are changed in the neural network with the aim of reducing the probability of a specified action of the strategy given a specified vehicle state if the performance of this specified action in the specified vehicle state resulted in a negative reward value. Adverse actions are thereby suppressed over time.

According to a preferred embodiment, the network parameters are changed in the neural network by evaluating the influence of variations in the network parameters generated by adding disturbances with regard to the change in the reward value and changing the network parameters if the variation leads to a better reward value. This strategy is also called an evolutionary strategy, and artificially created mutations take place, which are reinforced in the case of positive traits.

A control device for controlling a vehicle with actuators has at least one input for receiving vehicle data, at least one output for outputting target values for actuators, an agent, a neural network with network parameters for the neural network, and a reward generation device and is designed to to carry out such a procedure.

A vehicle has such a control device and actuators. In a vehicle, the control device has proven to be unexpectedly advantageous.

• 1 in schematischer Darstellung einen Agenten und eine Umgebung,
• 2 ein Fahrzeugmodell für eine Simulation,
• 3 ein Diagramm, welches die Wahrnehmung des Menschen von Wankwinkelfrequenzen anzeigt,
• 4 in schematischer Darstellung bestärkendes Lernen,
• 5 bis 8 die Auslenkung der vier Räder eines Fahrzeugs bei unterschiedlichen Agenten,
• 9 den Wankwinkel bei unterschiedlichen Agenten,
• 10 die Wankwinkelgeschwindigkeit bei unterschiedlichen Agenten,
• 11 ein auftretendes Moment und ein durch ein trainiertes neuronales Netzwerk erzeugtes Moment,
• 12 ein auftretendes Moment und ein durch ein untrainiertes neuronales Netzwerk erzeugtes Moment,
• 13 ein Histogramm mit Aufteilung unterschiedlich trainierter neuronaler Netzwerke auf ein Maß der Unbequemlichkeit, und
• 14 in schematischer Darstellung ein Fahrzeug mit einem Steuergerät und Aktoren.

Further details and advantageous developments of the invention result from the exemplary embodiments described below and illustrated in the drawings, which are in no way to be understood as limiting the invention, and from the dependent claims. It goes without saying that the features mentioned above and those still to be explained below can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the present invention. It shows:

• 1 a schematic representation of an agent and an environment,
• 2 a vehicle model for a simulation,
• 3 a diagram showing human perception of roll angle frequencies,
• 4 in a schematic representation of reinforcement learning,
• 5 until 8th the deflection of the four wheels of a vehicle with different agents,
• 9 the sway angle of different agents,
• 10 the roll angle velocity for different agents,
• 11 an occurring moment and a moment generated by a trained neural network,
• 12 an occurring moment and a moment generated by an untrained neural network,
• 13 a histogram with partitioning of differently trained neural networks on a level of discomfort, and
• 14 a schematic representation of a vehicle with a control unit and actuators.

In the following, parts that are the same or have the same effect are provided with the same reference symbols and are usually only described once. The description builds on one another across figures in order to avoid unnecessary repetition.

1 shows basic elements for reinforcement learning. An agent 20 is provided and enables setpoint values to be output via a line 83 to an environment 81. The environment 81 reacts as a function of the setpoint values, and an observation is sent to the agent 20 via a line 82. In addition, a reward value is generated, which evaluates the reaction of the environment resulting from the control values as positive or negative. The agent 20 responds in response to the observation and issues updated setpoints. The agent 20 works with machine learning methods.

2 shows a simulation 110 of a vehicle 10, which uses a physical semi-vehicle model in the exemplary embodiment. The vehicle model 110 represents vertical vehicle dynamics for half a vehicle (front or rear). The body 112 has a center of gravity 113 and is connected to a left wheel 114 through a damper 131 and spring 132 and to a right wheel 116 through a damper 141 and spring 142 on the left side. The model operates on the Skyhook model and a damper 133 and spring 134 are provided between the left wheel 114 and the road 118 . A damper 143 and a spring 144 are provided between the right wheel 116 and the road 120 . The lateral distance between the center of gravity 113 and the left wheel 114 is plotted as I ^l and the lateral distance between the center of gravity 113 and the right wheel 116 as I ^r . The roll angle phi (cp) is located at the center of gravity 113 . Forces of a respective actuator are drawn in as u ^l and u ^r , such as can be generated, for example, by an active chassis.

3 shows a diagram with a curve 150, which indicates how unpleasant different roll angle frequencies are perceived. Roll angle frequencies of approx. 0.7 Hz or in the range of 0.4 Hz to 1.0 Hz are perceived as particularly unpleasant. The curve 150 has been determined empirically. The curve 150 is suitable for generating a reward value as a function of the currently occurring roll angle frequencies. For this purpose, for example, the rolling angle frequencies that occur can be multiplied by the associated value of curve 150 and then added together in order to generate a measure of the current comfort in body 112 . For example, if the resulting value is above a predetermined threshold, a negative reward value may be generated, and if it is below the predetermined threshold, a positive reward value may be generated. Naturally, different possibilities for generating the reward value are conceivable. For example, an evaluation can only be carried out in certain frequency ranges, and different limit values and normalizations can be used.

4 shows schematically the process of reinforcement learning. In a box 84 the environment 81 and the agent 20 are provided. The vehicle state and the setpoint values for the actuators are transferred to a target function 86 via an arrow 85 and the target function 86 calculates a target result which is transferred to an optimizer 88 via an arrow 87 . The target result can also be referred to as the reward value. The optimizer 88 determines changed network parameters for the neural network of the agent 20 and passes them on to a changed agent 90 via an arrow 89 . The changed agent 90 is fed to the agent 20 via the arrow 91 or replaces it.

When it comes to reinforcement learning algorithms, there are model-free algorithms and model-based algorithms. Experiments have shown that model-free algorithms for vehicle control deliver particularly advantageous results and lead to pleasant driving behavior. In the case of the model-free algorithms for reinforcement learning, there are in particular the method of strategy optimization (English: policy optimization) and the method of Q-learning (English: Q-learning). Research has shown that strategy optimization in machine learning with reinforcement learning for the selected area of application is particularly positive.

Two methods have proven to be particularly advantageous in strategy optimization. On the one hand there is the so-called REINFORCE method, and on the other hand the method of evolutionary strategy, which is also known as the ES method.

mit dem Strategie-Parameter θ (englisch: policy parameter) zum Zeitpunkt t bzw. t+1, einem Applikationsparameter α, der die Lernrate beeinflusst, der Belohnungsfunktion R(o,a) für die Beobachtung o (Fahrzeugzustand) und die durchgeführte Aktion a sowie dem Gradienten nach dem Strategie-Parameter θ (englisch: agent gradient) vom natürlichen Logarithmus der Wahrscheinlichkeit P im neuronalen Netzwerk für die Aktion A im Hinblick auf die Beobachtung o. Das Produkt der Belohnungsfunktion mit dem Gradienten wird als Strategiegradient (englisch: policy gradient) bezeichnet. Der Strategiegradient kann auch anders aufgebaut werden, beispielsweise mit dem Logarithmus zur Basis 10.In the REINFORCE process, behavior that yields a high reward or large positive reward value is reinforced and behavior that yields a low reward or negative reward is weakened. The following function is preferably used for this: $${\mathrm{\theta }}_{\text{t}+1}={\mathrm{\theta }}_{\text{t}}+\mathrm{a}\text{R}\left(\text{O}\text{,a}\right){\nabla }_{\mathrm{\theta }}\text{ln P}\left(\text{a}|\text{O}\right)$$

with the strategy parameter θ (policy parameter) at time t or t+1, an application parameter α that influences the learning rate, the reward function R(o,a) for the observation o (vehicle state) and the action a carried out and the gradient according to the policy parameter θ (English: agent gradient) from the natural logarithm of the probability P in the neural network for the action A with regard to the observation o. The product of the reward function with the gradient is called the policy gradient (English: policy gradient ) designated. The strategy gradient can also be structured differently, for example with the logarithm to base 10.

Applying this feature increases the probability of an action when the resulting reward value is positive. An action with a negative reward value, on the other hand, is reduced in probability. As a result, the network parameters are changed in such a way that the agent, with the strategy dependent on the network parameters, preferably carries out actions which result in a high reward.

mit dem Gradienten nach dem Strategie-Parameter θ von einer Kostenfunktion J, der in Abhängigkeit einer Störung ε des Strategie-Parameters θ nach dem Vorbild einer Mutation variiert wird, und Variationen mit hohen Belohnungswerten werden verstärkt. E_τ∼Aθ ist der Erwartungswert für eine gewählte Aktion τ aus den möglichen Aktionen A_θ. E_ε∼N ist der Erwartungswert für die Störung ε, die aus einer Normalverteilung N gewählt ist. Der hintere Teil stellt eine Abschätzung der finiten Differenz dar.The evolutionary strategy has an analogous goal to the REINFORCE method, but it works directly on the network parameter level. The network parameters are varied by adding disturbance values, and it is checked whether these variations can improve the expected reward values. The optimization is based on the following equation: $${\nabla }_{\mathrm{\theta }}\text{J}\left(\mathrm{\theta }\right)={\text{E}}_{\mathrm{\tau }\text{~A}\mathrm{\theta }}\left[{\text{E}}_{\mathrm{e}\text{~N}}\left[\left(\text{R}\left(\mathrm{\theta }+\mathrm{e}\right)-\text{R}\left(\mathrm{\theta }-\mathrm{e}\right)\right)/\mathrm{e}\right]\right]$$

with the gradient according to the strategy parameter θ of a cost function J varied depending on a perturbation ε of the strategy parameter θ along the lines of a mutation, and variations with high reward values are amplified. E _τ∼Aθ is the expected value for a chosen action τ from the possible actions A _θ . E _ε∼N is the expectation value for the disturbance ε, which is chosen from a normal distribution N. The rear part represents an estimate of the finite difference.

5 shows the wheel movement of the front right wheel, 6 the wheel movement of the front left wheel, 7 the wheel movement of the rear right wheel and 8th the wheel motion of the rear left wheel. The abscissa shows the time t in seconds and the ordinate shows the wheel movement in meters. The curves 201.1, 201.2, 201.3 and 201.4 show the respective wheel movement in the trained state of the neural network, and the curves 202.1, 202.2, 202.3 and 202.4 the wheel movement in the untrained state of the neural network. In the simulation, the underground has the form of a step, with one step up and one step down always following one another and the steps between the left side and the right side are offset to one another. The active chassis must correct accordingly in order to reduce the roll angle, at least in the relevant frequency ranges. The vertical movement of the wheels is generally greater when trained than when untrained, and this allows for better smoothing over bumps.

• - Sinusform
• - Sägezahnform (englisch: saw form)
• - zufällig erzeugte Form
• - tatsächliche Messung eines Straßenuntergrunds

In addition to the simulation with such a stepped background, simulations can be carried out with the following background variants:

• - Sinusoidal
• - saw tooth shape
• - randomly generated shape
• - actual measurement of a roadbed

Here, for example, the amplitude, the standard deviation, the period, the length, the left and right symmetry and the inclination can be varied.

9 shows the roll angle in radians over time t in seconds. Line 203 corresponds to the roll angle with a trained neural network, and line 204 with an untrained neural network. It can be seen that the amplitudes of the roll angle in the untrained state are four to six times larger than in the trained state and the trained state leads to a significantly smoother driving behavior of the vehicle.

10 shows the roll angular velocity plotted over time t in seconds. the Roll velocity is given as a rate of radians with units of rad/s. Curve 205 shows the roll angular velocity for the trained neural network, and curve 206 for the untrained neural network. The roll angular velocity of the trained neural network is significantly lower than that of the untrained neural network, and this leads to a smoother and more pleasant driving behavior of the vehicle.

11 shows the moment generated by the background as curve 207, and curve 208 shows the moment generated by the agent 20 with the aid of the trained neural network over time t in seconds. With the help of the agent 20 and the trained neural network, the external moment can be almost completely compensated, and this leads to a smooth and pleasant driving behavior and a high level of comfort.

12 shows as line 209 the moment generated by the background, and line 210 shows the moment generated by the agent 20 with the help of the untrained neural network over time t in seconds. With an untrained neural network, the agent 20 is not able to generate a corresponding counter-torque, and as a result a comparatively large rolling movement of the body occurs, which the driver perceives as unpleasant.

13 Fig. 12 shows a histogram in which the abscissa indicates an in-vehicle discomfort level and the ordinate divides the different results in the neural network reinforcement learning to the discomfort level. The level of discomfort increases from left to right. The bar at 221 corresponds to a passive controller and this is comparatively uncomfortable, measuring about 0.0114. The bar 222 corresponds to a preferred result of a trained neural network, which with a value of 0.00487 offers a high level of comfort or has a low degree of inconvenience. It has been shown that excellent parameters for the neural network can be determined through reinforcement learning, which can be determined with the simulation of 2 led to a good result. In the present case, the confirmatory learning was carried out according to the method of evolutionary strategies.

• - Hinterradlenkvorrichtung;
• - Wankmomentverteilungsvorrichtung;
• - Antriebskraftverteilungsvorrichtung bei einem Fahrzeug mit Zweiachsenantrieb;
• - Fahrdynamikregelung.

Significantly more complex agents with additional actuators can also be implemented with such a neural network, and the framework with the neural network can be used for different control devices or regulating devices. Examples of other actors are

• - rear wheel steering device;
• - rolling moment distribution device;
• - Driving force distribution device in a vehicle with two-axle drive;
• - Driving dynamics control.

14 shows a schematic representation of a vehicle 10 with the control device 30, actuators 61, 62 and sensors 67, 68 for generating vehicle data 53, 54. The control device 30 has an input 31 for receiving the vehicle data 53, 54 and an output 32 for outputting Target values 63, 64 for the actuators 61, 62.

The control device 30 has the agent 20, which controls the actuators 61, 62 with the target values 63, 64. The agent 20 has the neural network 43 with network parameters NP, and the vehicle data 53, 54 are supplied to it.

In a step A), the vehicle data 53, 54 are recorded via the at least one input 31, and a current vehicle state 41 is calculated.

The vehicle status can consist of a series of vehicle data 53, 54, but further status data are preferably calculated from vehicle data 53, 54.

In a step B), the agent 20 assigns an action 42 to the current vehicle state 41 according to a strategy implemented with the neural network 43 as a function of the network parameters NP. The neural network 43 can also be referred to as a neural network 43 .

Depending on action 42 , updated target values 63 , 64 for actuators 61 , 62 of vehicle 10 are determined in a step C) and are output via the at least one output 32 .

In a step D), depending on the change in the vehicle data 53, 54 as a result of the updated target values 63, 64, the reward generation device 65 uses a reward function to determine a reward value, which can be positive or negative. Positive indicates that the strategy was beneficial and negative indicates that it was not beneficial.

The reward value 66 is fed to a step E), and in this step the network parameters NP are changed (NP′) by reinforcement learning depending on the reward value 66 in order to increase the reward values 66 to be expected when using the neural network 43 through the change. The reward values NP can be changed immediately, or it can take place, for example, after the vehicle 10 has stopped.

The control device 30 preferably has a data interface 69, for example wireless or wired, and the control device 30 can be sent second network parameters NP'' when it is installed in a vehicle 10. This enables the neural network to be improved from the outside, even when the control device 30 is installed. The data interface 69 is preferably bidirectional.

The second network parameters NP'' are preferably dependent on a plurality of other control devices 30 (not shown) from different other vehicles 10, and the control device 30 is designed to change the network parameters NP depending on the second network parameters NP''. In this way, different network parameters of a vehicle type can be evaluated, and particularly advantageous network parameters can be sent to vehicles of the same vehicle type. This is advantageous, for example, when a specific vehicle state occurs only very rarely and is therefore not optimized, or only slightly so. This can also improve security in situations that rarely occur.

A wide range of variations and modifications are of course possible within the scope of the present invention.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102019104966 A1 [0002]
• DE 102006054425 A1 [0003]
• DE 102017007136 A1 [0004]
• DE 102020106936 A1 [0005]
• DE 112016003350 T5 [0006]

Claims (14)

Method for controlling actuators (61, 62) of a vehicle (10) by means of a control device (30), which control device (30) has at least one input (31) for receiving vehicle data (53, 54), which control device (30) has at least one output (32) for outputting desired values (63, 64) for actuators (61, 62), which control device (30) has an agent (20), a neural network (43) with network parameters (NP) for the neural network (43), and a reward generation device (65) and is designed to set target values (63, 64) for the to determine actuators (61, 62), which procedure has the following steps: A) The vehicle data (53, 54) are recorded via the at least one input (31), and a current vehicle status (41) is calculated; B) The agent (20) assigns an action (42) to the current vehicle state (41) according to a strategy implemented with the neural network (43) as a function of the network parameters (NP); C) Depending on the action (42), updated target values (63, 64) for the actuators (61, 62) are determined and output via the at least one output (32); D) Depending on the change in the vehicle data (53, 54) as a result of the updated target values (63, 64), a reward value (66), which can be positive or negative, is determined by the reward generation device (65) using a reward function; E) The network parameters (NP) are changed by reinforcement learning depending on the reward value (66) in order to increase the reward values (66) to be expected when using the neural network (43). procedure after claim 1 , in which the reinforcement learning is performed as model-free reinforcement learning. procedure after claim 1 or 2 , in which the agent (20) first network parameters (NP) are specified, which first network parameters (NP) previously determined by reinforcement learning with the agent (20) or with another agent (20) with the same neuronal network (43) structure will. procedure after claim 3 , in which a simulation (110) of a vehicle (10) is created, and in which the first network parameters (NP) are determined using the simulation (110). procedure after claim 3 or 4 , in which the agent (20) for the simulation (110) in a vehicle (10) actually measured vehicle data (53, 54) are supplied. Method according to one of the preceding claims, in which the network parameters (NP) are also changed when the control device (30) is installed in a vehicle (10) by reinforcement learning as a function of the reward value (66), the reward value (66) being dependent is determined from the actual influencing of the vehicle data (53, 54) by the agent (20) in order to increase the reward values (66) to be expected when using the neural network (43) by changing the network parameters (NP). procedure after claim 6 , in which the change in the network parameters (NP) is also carried out while the vehicle (10) is moving. Method according to one of the preceding claims, in which the control device (30) has a data interface (69) and in which the control device (30) is sent second network parameters (NP'') when installed in a vehicle (10). procedure after claim 8 , in which the second network parameters (NP'') are dependent on a plurality of other control devices (30) from different other vehicles (10), and in which the control device (30) is designed to change the network parameters (NP) as a function of to change the second network parameter (NP''). Method according to one of the preceding claims, in which the network parameters (NP) are changed in the neural network (43) with the aim of increasing the probability of a specified action (42) of the strategy given a specified vehicle state (41) if the Carrying out this specified action (42) in the specified vehicle state (41) has resulted in a positive reward value. Method according to one of the preceding claims, in which the network parameters (NP) are changed in the neural network (43) with the aim of reducing the probability of a specified action (42) of the strategy given a specified vehicle state (41) if the Carrying out this specified action (42) in the specified vehicle state (41) has resulted in a negative reward value. Method according to one of the preceding claims, in which the network parameters (NP) are changed in the neural network (43) by evaluating the influence of variations in the network parameters (NP) caused by adding disturbances with regard to the change in the reward value (66). and changing the network parameters (NP) if the variation results in a better reward value (66). Control device for controlling a vehicle (10) with actuators (61, 62), which control device (30) has at least one input (31) for receiving vehicle data (53, 54), which control device (30) has at least one output (32) for Output of target values (63, 64) for actuators (61, 62), which control device (30) has an agent (20), a neural network (43) with network parameters (NP) for the neural network, and a reward generation device (65) has and is designed to carry out a method according to any one of the preceding claims. Vehicle (10), which according to a control device Claim 13 and actuators (61, 62).

Images