Classifications

• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B13/00—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion
  • G05B13/02—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
  • G05B13/0205—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric not using a model or a simulator of the controlled system
  • G05B13/021—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric not using a model or a simulator of the controlled system in which a variable is automatically adjusted to optimise the performance
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B13/00—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion
  • G05B13/02—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
  • G05B13/04—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators
  • G05B13/041—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators in which a variable is automatically adjusted to optimise the performance
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
  • G06F17/10—Complex mathematical operations
  • G06F17/11—Complex mathematical operations for solving equations, e.g. nonlinear equations, general mathematical optimization problems

Definitions

• the invention relates to a method for operating an actuator control system, a learning system, the actuator control system, a computer program for executing the method and a machine-readable storage medium on the
• a new value of the at least one parameter is selected depending on a long-term cost function, wherein this long-term cost function is determined as a function of a predicted temporal evolution of a probability distribution of the control variable of the actuator and the parameter is then set to this new value.
• the invention relates to a method for operating an actuator control system, which is set up for controlling a control variable of an actuator to a predefinable target size, wherein the actuator control system is set up, depending on a size characteristic of a control strategy, in particular also dependent on the setpoint and / or the control variable, to generate a manipulated variable and to control the actuator depending on this manipulated variable,
• control strategy can be determined in such a way that for each control variable the action from which the manipulated variable is derived is determined, which maximizes the value function.
• the value function is determined iteratively by stepwise approximating the value function by means of the Bellmann equation by successive iterations of an iterated value function, wherein by means of the Bellmann equation an iterated value function of a previous iteration, an iterated value function of a subsequent iteration is determined,
• this makes it possible to ensure that the iteratively determined value function maximizes a predefinable reward, especially in the long run and under consideration of the system dynamics.
• the projections it is possible to solve the Bellman equation, which is only analytically solvable pointwise on account of a maximum value formation contained in it, particularly easy to approximate.
• This iterative procedure makes it possible to limit a numerical error of the method to a predefinable maximum value in a particularly efficient manner, and thus to operate the actuator control system particularly reliably.
• the at least one further basic function is selected as a function of a maximum point of the control variable at which the residual becomes maximum. This makes the method particularly efficient, since a numerical error that can be reduced particularly quickly by the projection onto the function space spanned by the set of basis functions. The efficiency is particularly high if in this case the at least one further base function assumes its maximum value at maximum point.
• the at least one further basic function depends on a variable characterizing a curvature of the residuum at the maximum point, in particular the
• Hesse matrix of the residuum at the maximum point is chosen.
• the at least one further basic function is selected such that its Hesse matrix at the maximum location equals the Hesse matrix of the residual.
• a conditional probability on which the Bellman equation is dependent is determined by means of a model of the actuator.
• the method becomes particularly efficient, since a real behavior of the actuator does not have to be determined again.
• the model is a Gaussian process.
• the basic functions are given by Gaussian functions, since the integrals that occur can then be solved analytically as integrals via products of Gaussian functions, which enables a particularly efficient implementation.
• the invention relates to a learning system for automatically setting a variable characterizing a control strategy of an actuator control system, which is set up for controlling a control variable of an actuator to a predefinable desired value, wherein the learning system is set up to execute one of the aforementioned methods.
• the invention relates to a method in which the variable characterizing the control strategy is determined according to one of the aforementioned methods and then the manipulated variable is generated as a function of the variable characterizing the control strategy and the actuator is controlled as a function of this manipulated variable.
• the invention in another aspect, relates to an actuator control system which is adapted to drive an actuator with this method.
• the invention relates to a computer program configured to perform any of the foregoing methods. That the computer program includes instructions that, when executed on a computer, cause that computer to perform the procedure.
• the invention relates to a machine-readable storage medium on which this computer program is stored.
• FIG. 1 shows schematically an interaction between the learning system and the actuator
• FIG. 2 schematically shows an interaction between actuator control system and actuator
• FIG. 3 shows in a flow chart an embodiment of the method for
• FIG. 4 shows in a flow chart an embodiment of a method for determining iterated value functions
• Figure 5 is a flowchart showing an embodiment of a method for determining a set of basis functions
• FIG. 6 shows flow diagrams of embodiments of methods for determining the manipulated variable.
• Figure 1 shows the actuator 10 in its environment 20 in interaction with the learning system 40.
• Actuator 10 and environment 20 are collectively referred to below as the actuator system.
• a state of the actuator system is detected by a sensor 30, which may also be provided by a plurality of sensors.
• An output signal S of the sensor 30 is transmitted to the learning system 40.
• the learning system 40 determines therefrom a drive signal A, which the actuator 10 receives.
• the actuator 10 can be, for example, a (partially) autonomous robot, for example a (partially) autonomous motor vehicle, a (partially) autonomous lawnmower. It may also be an actuation of an actuator of a motor vehicle, for example, a throttle valve or a bypass actuator for idle control. It may also be a heating system or a part of the heating system, such as a valve actuator.
• the actuator 10 may in particular also be larger systems, such as an internal combustion engine or a (possibly hybridized) drive train of a motor vehicle or even a brake system.
• the sensor 30 may be, for example, one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more position sensors (for example GPS). Other sensors are conceivable, for example, a temperature sensor.
• the actuator 10 may be a manufacturing robot
• the sensor 30 may then be, for example, an optical sensor that detects characteristics of manufacturing products of the manufacturing robot.
• the learning system 40 receives the output signal S of the sensor 30 in an optional receiving unit 50, which converts the output signal S into a control variable x (alternatively, the output signal S can also be taken over directly as the control variable x).
• the control variable x may be, for example, a section or a further processing of the output signal S.
• the control variable x is supplied to a controller 60. In the controller either a control strategy ⁇ can be implemented, or a value function V * .
• parameters ⁇ are stored, which are supplied to the controller 60.
• the parameters ⁇ parameterize the control strategy ⁇ or the value function V * .
• the parameters ⁇ may be a singular or a plurality of parameters.
• a block 90 supplies the regulator 60 with the predefinable setpoint value xd. It can be provided that the block 90 generates the predefinable desired value xd, for example, depending on a sensor signal that is given to the block 90. It is also possible for block 90 to read the setpoint variable xd from a dedicated memory area in which it is stored.
• the controller 60 Depending on the control strategy ⁇ or the value function V * of the setpoint xd and the control variable x, the controller 60 generates a manipulated variable u. This can be determined, for example, as a function of a difference x-xd between the control variable x and the target variable xd.
• the controller 60 transmits the manipulated variable u to an output unit 80, which determines the drive signal A therefrom. For example, it is possible that the output unit first checks whether the manipulated variable u lies in a predefinable value range. If this is the case, the control signal A is determined as a function of the manipulated variable u, for example by an associated control signal A being read from a characteristic field as a function of the manipulated variable u. This is the normal case. If, on the other hand, it is determined that the manipulated variable u is not within the predefinable value range, it may be provided that the actuation signal A is designed such that it causes the actuator A to be converted into a safe mode.
• Receiving unit 50 transmits the control variable x to a block 100. Also, controller 60 transmits the corresponding manipulated variable u to the block 100.
• Block 100 stores the time series of the control variable x received at a sequence of times and the respectively corresponding manipulated variable u.
• Block 100 can then adapt model parameters ⁇ , ⁇ ⁇ , ⁇ f of model g on the basis of these time series.
• the model parameters ⁇ , ⁇ ⁇ , ⁇ f are fed to a block 1 10, which stores them, for example, at a dedicated memory location. This will be described in more detail below in FIG. 4, step 1010.
• the learning system 40 in one embodiment, includes a computer 41 having a machine-readable storage medium 42 on which is stored a computer program that, when executed by the computer 41, causes it to perform the described functionality of the learning system 40.
• the computer 41 comprises in the exemplary embodiment a GPU 43.
• the model g can be used to determine the value function V * . This will be explained below.
• FIG. 2 illustrates the interaction of the actuator control system 45 with the actuator 10.
• the structure of the actuator control system 45 and its interaction with actuator 10 and sensor 30 are largely similar to the design of the learning system 40, so only the differences are described here.
• the actuator control system 45 has no block 100 and also no block 1 10 on. The transmission of quantities to the block 100 is therefore omitted.
• Parameters ⁇ are stored in the parameter memory 70 of the actuator control system 45 and have been determined using the method according to the invention, for example as illustrated in FIG.
• FIG. 3 illustrates an embodiment of the method according to the invention.
• First (1000) an initial value x 0 of the control variable x is selected from a predefinable initial probability distribution p (x 0 ).
• random variables u 0 , u 1 , u T-1 are randomly selected up to a predeterminable time horizon T with which the actuator 10 is driven as described in FIG.
• the actuator 10 interacts via the environment 20 with the sensor 30, whose sensor signal S is received as control variable Xi, x T-1 , x T indirectly or directly from the controller 60.
• D is the dimensionality of the control variable x and F is the dimensionality of the manipulated variable u, ie
• a covariance function k of the Gaussian process g is given by, for example
• a covariance matrix K is defined by
• the Gaussian process g is then characterized by two functions: By an average ⁇ and a variance Var given by
• the parameters ⁇ , ⁇ ⁇ , ⁇ ⁇ are then fitted to the pairs ( ⁇ ', ⁇ ') in a known manner by maximizing a logarithmic marginal likelihood function.
• step 1080 follows.
• an optimal control strategy assigned to the episode index e is defined by
• the initial value x 0 of the control variable x is again selected from the initial probability distribution p (x 0 ).
• step 1030 If it has been decided in step 1030 that the iteration over episodes has resulted in a convergence of the iterated value functions associated with episode index e, the value function V * is set equal to that of the iterated value functions V e * associated with the episode index e . This ends this aspect of the process.
• FIG. 4 illustrates an embodiment of the method for determining the iterated value functions Aus associated with episode index e
• episode index e is omitted below.
• the superscript index will be referred to below with the letter t.
• the method always calculates a subsequent iterated value function
• This previous iterated value function ⁇ * is a linear combination with basic radio and coefficients. These coefficients fa).
• a set B of basis functions is determined (1510).
• the operator A is defined as
• r is a reward function, which assigns a reward value to a value of the control value x.
• the reward function r is chosen such that it assumes the greater the smaller the deviation of the control variable x from the target value xd.
• the conditional probability p (x 'Ix, u) of the control variable x' given the previous control variable x and the manipulated variable u can be determined in formula (8) by means of the Gauss process g.
• the abort criterion can be met, for example, if the iterated value function is converged, for example, when a difference to the previous iterated value function V * becomes smaller than a second threshold ⁇ 2 , ie
• the abort criterion can also be considered fulfilled if the index t has reached the predeterminable time horizon T.
• the index t is increased by one (1570). If the abort criterion is fulfilled, however, the value function V * is set equal to the iterated value function of the last iteration.
• Figure 5 illustrates an embodiment of the method for determining the set B of basis functions for the actual iterated value function V * of the Bellman equation.
• telt eg with a gradient ascending method and a Hesse matrix of the residual is determined at the maximum point x * .
• Value function is sufficiently converged, for example, by checking whether an associated standard (eg Norm) of the deviation falls below a third predetermined limit ⁇ 3 , ie
• index I is incremented by one and the method branches back to step 1610.
• FIG. 6 illustrates embodiments of the method for determining the manipulated variable u.
• FIG. 7a illustrates an embodiment for the case that the parameters ⁇ stored in the parameter memory 70 parameterize the control strategy ⁇ .
• a set of test points x t is defined, for example as a Sobol plan (English: “Sobol design plan").
• a data-based model is then learned (1720), for example a Gaussian process g ⁇ , so that the data-based model efficiently determines an associated optimal manipulated variable u for a control variable x.
• the parameters ⁇ characterizing the Gaussian process g ⁇ are stored in the parameter memory 70.
• the steps (1700) to (1720) preferably run in the learning system 40.
• the actuator control system 45 determines (1730) by means of the Gaussian process g ⁇ the given manipulated variable u for a given control variable x.
• FIG. 7b illustrates an embodiment for the case in which the parameters ⁇ stored in the parameter memory 70 parameterize the value function V * .
• the corresponding equation is given by equation

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Mathematical Physics (AREA)
• Software Systems (AREA)
• Pure & Applied Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Computational Mathematics (AREA)
• Mathematical Analysis (AREA)
• Mathematical Optimization (AREA)
• Theoretical Computer Science (AREA)
• Evolutionary Computation (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Automation & Control Theory (AREA)
• Artificial Intelligence (AREA)
• Medical Informatics (AREA)
• Health & Medical Sciences (AREA)
• Operations Research (AREA)
• Databases & Information Systems (AREA)
• General Engineering & Computer Science (AREA)
• Algebra (AREA)
• Feedback Control In General (AREA)

Priority Applications (6)

Applications Claiming Priority (2)

Related Child Applications (2)

Publications (1)

Family

ID=63244585

Family Applications (1)

Country Status (7)

Cited By (1)

Families Citing this family (3)

Citations (3)

Family Cites Families (12)

• 2017
  • 2017-10-20 DE DE102017218811.1A patent/DE102017218811A1/de active Pending
• 2018
  • 2018-08-10 CN CN201880067677.3A patent/CN111406237B/zh active Active
  • 2018-08-10 EP EP18755774.9A patent/EP3698223B1/de active Active
  • 2018-08-10 WO PCT/EP2018/071753 patent/WO2019076512A1/de not_active Ceased
  • 2018-08-10 KR KR1020207014310A patent/KR102326733B1/ko active Active
  • 2018-08-10 JP JP2020542498A patent/JP7191965B2/ja active Active
  • 2018-08-10 US US16/756,953 patent/US20210003976A1/en not_active Abandoned
• 2021
  • 2021-09-15 US US17/475,911 patent/US20220075332A1/en not_active Abandoned

Patent Citations (3)

Also Published As

Similar Documents

Legal Events