DE10021929A1 - Computerized determination of control strategy for technical system involves using reinforcement learning to determine control strategy for each state and learn optimal actions - Google Patents

Abstract

The method involves describing the system using a continuous state space and an action space. A state change assessment is performed and a model of the technical system is determined using training data describing the system by forming fuzzy association functions. The functions are fed to a reinforcement learning method with which a control strategy is determined for each state in state space and the optimal actions in action space are learned. The method involves describing the system using a continuous state space and an action space, whereby the state space has states that the technical system can adopt and the action space has actions that are carried out to produce a state change from a previous state to a subsequent state in state space. An assessment of the state change is performed and a model of the technical system is determined using training data describing the system by forming fuzzy association functions with which at least the state space state space is described. The fuzzy association functions are fed to a reinforcement learning method with which a control strategy is determined for each state in state space and the optimal actions in action space are learned. Independent claims are also included for the following: a fuzzy controller for determining a control strategy for a technical system and a computer-readable storage medium.

Description

The invention relates to a method and a fuzzy Control device for determining a control strategy for a technical system and a computer readable Storage medium and a computer program element.

Such a method and such a fuzzy control device are known from [1] and [3].

In the method known from [3], a descriptive and controllable technical system which originally with a continuous state space and a continuous area of action is described, discretized.

On the basis of the discretized state space and the discretized action space, the reinforcement Learning method according to the principle of the so-called "Prioritized Sweeping ".

This known procedure has the particular disadvantage that that either a very fine partitioning of the continuous space is required, resulting in a great complexity of the discrete problem to be solved with the resulting very large computing time requirements and associated with considerable space requirements in the Under the control of a technical system results.

However, if the partitioning is coarser, the Approximation of the technical system to be controlled very much inaccurate. This leads to a suboptimal, that is to say one  relatively poor tax strategy, according to the Reinforcement learning is determined.

To improve the achievable approximation accuracy, it is known from [4] to use an interpolation strategy use what is basically the use of a so-called, described in [1] Takagi-Sugeno system with constant Consequences in the rules corresponds.

In the method known from [4], however, training becomes the values in the centers of the interpolation scheme a hard one Partitioning the state space and the action space carried out, which is why the above again disadvantages shown.

It is also known from [2] to use fuzzy partitions of a fuzzy C mean clustering method is known.

The invention is therefore based on the problem of a Control strategy for a technical system under Use a reinforcement learning process to specify at compared to the method known from [3] improved control strategy is determined.

The problem is exacerbated by the process, the fuzzy Control device for computer-aided determination of a Control strategy for a technical system through which Computer readable storage medium and through a Computer program element with the features according to the independent claims solved.

Determine a procedure for the computer-aided procedure This becomes a control strategy for a technical system technical system with a continuous state space and a continuous or discretized area of action described. The state space has states that the technical system can basically adopt. Further points  the action space on actions that are carried out to a State transition from a previous state of the state space to generate in a successor state of the state space. With Training data that describe the technical system a model of the technical system is determined and in fuzzy Partitions grouped by fuzzy membership functions to the fuzzy partitions with which at least the state space is described.

The state transition is evaluated in such a way that an optimization with regard to the ratings to a leads optimal control strategy.

Using the fuzzy membership functions, a Reinforcement learning process at least for the state space carried out, whereby a control strategy, d. H. one action per state, determined for each State of the state space and an evaluation of all state Action pairs is performed. The technical system is taking into account the control strategy by means of Control variables that depend, for example, on the Control strategy are selected or formed, controlled.

The individual partitions are also called clusters designated.

The invention in particular ensures that the Approximation accuracy and thus the determination of the Control variables accelerated considerably, that is with reduced computing time is feasible.

Furthermore, the determined control strategy with regard to of the profit as an optimization variable within the Reinforcement learning process significantly compared to the state technology improved.  

Also the number of the approximation of the technical Systems required partitions, in particular those for Description of the partitions used one centers Partition significantly reduced.

Due to the reduction in the number required Partition centers will have a faster calculation of the Q- Function in the context of the reinforcement learning process allows higher accuracy.

Preferred developments of the invention result from the dependent claims.

For each state of the state space and the corresponding ones Actions of the action area can each have a Q value as Control strategy, d. H. the evaluation of the execution of the Action is determined in the state.

Furthermore, in the conclusions of the fuzzy rules of the fuzzy Systems, which according to the reinforcement learning process is formed, which approximates the Q function, linear Terms are used.

In this way, it becomes a quick, yet accurate Determine the control strategy that is possible for the model reproduces very precisely.

According to a further embodiment of the invention Reinforcement learning procedures performed by in particular such actions, d. H. such experiments are carried out that meet a given criterion.

In this way, an optimized selection of actions, d. H. Experiments possible, reducing the number of Experiments are minimized and learning is accelerated becomes.  

The criterion can be an expected information gain about the conditional transition probabilities be within the reinforcement learning process.

From a gain of information a future one can Profit can be estimated. In particular, it can only or essentially only the actions selected and carried out those regarding the directly or indirectly expected information gain is better than one predeterminable minimum information gain.

The invention can be advantageously used in general to control a traffic system as a technical system, especially for control, d. H. to choose one Framework signal plan for controlling traffic lights within a Transport network. Thus, for example, due to the Control strategy a framework signal plan can be selected and based on the selected frame signal plan corresponding control signals at traffic lights of a traffic network are transmitted, the traffic lights according to the selected Activate the frame signal plan.

For each fuzzy partition in the state space and in that Action space can be determined an information gain that from previous explanations of fuzzy Actions belonging to the partition in the corresponding states resulted.

According to a further embodiment of the invention are counters provided with which the number of executions of Actions in a state of the technical system and the Number of state transitions from an initial state, i.e. H. a previous state into a successor state due to the Action until the iteration is specified. The counters assigned values are determined when a new one is determined State transition depending on the degree of belonging to the  States or the state transitions to the respective fuzzy Clusters updated.

The state transition probabilities can be within the reinforcement learning process depends on the counters be determined.

According to a further development, the Invention at the beginning of the process fuzzy partitions formed by starting from. in an iterative process a given set of output partition subsets these are divided into several fuzzy partition Subsets or merged from multiple fuzzy Partition subsets into a fuzzy partition subset, depending on the determined training data.

Alternatively, at the beginning of the process, the fuzzy Partitions according to the fuzzy C means clustering method be formed.

The invention can clearly be seen in that for Control of a technical system the system description by means of fuzzy partitions and accordingly with fuzzy Membership functions are discretized and in the discretized model using reinforcement Learning a control strategy to control the technical Systems is determined.

A fuzzy control device has a processor that is set up such that those described above Process steps are feasible.

There is a program in a computer readable storage medium saved, the process steps when it is executed of the method described above. Furthermore, a Computer program element when executed by a  Processor also does the steps above described method on.

The invention can be used both as a computer program, ie in Software, as well as by means of a special electronic Circuit, i.e. in hardware, can be realized.

Embodiments of the invention are in the figures shown and are explained in more detail below.

Show it:

Fig. 1 is a flow diagram in which the individual process steps of the method are illustrated according to an embodiment of the invention;

Fig. 2 is a sketch of a traffic network, with reference to which an embodiment of the invention is represented;

Fig. 3 is a sketch of a central control computer, which is coupled to individual sensors in the transport network;

Figures 4a to 4d, a plurality of signal images in accordance with different frame signal plans for different intersections of the traffic network of Fig. 2.

Fig. 5 is a sketch of a frame signal;

Fig. 6 is an illustration of fuzzy partitions and their membership functions;

Fig. 7a and 7b representations of different clusters.

FIG. 2 shows a traffic network 200 , on the basis of which the training and the selection of a traffic-dependent selection of a frame signal plan from a plurality of stored frame signal plans is explained below.

The traffic network 200 has a first road 201 which leads from a residential area 202 to a commercial area 203 . The residential area 202 is located in the west of a city 204 , and the commercial area 203 is in the east of the city 204 .

A second street 205 leads from a first shopping area 206 located in the north of the city 204 to a second shopping area 207 with a leisure center, which is located in the south of the city 204 .

The first street 201 and the second street 205 cross each other at a first intersection 208 .

Furthermore, the traffic network 200 has a third road 209 , which extends from the first road 201 from a second intersection 210 to a third intersection 211 , which is located on the second road 205 . Clearly, the third street 209 thus represents a diagonal connection from the first street 201 to the second street 205 , the second intersection 210 being west of the first intersection 208 , that is to say the second intersection 210 is closer to the residential area 202 than to the commercial area 203 .

Furthermore, a fourth street 212 leads from the third intersection 211 to a fourth intersection 213 , the fourth intersection 213 on the first street 201 lying east of the first intersection 208 , that is to say closer to the commercial area 203 than to the residential area 202 .

At each intersection, traffic lights are provided for each direction that a vehicle can travel on the road, which control the flow of traffic at the respective intersection 208 , 210 , 211 , 213 .

The traffic lights are described by one below central control unit controlled.

Furthermore, sensors 215 are provided on the roads, with which the number of vehicles passing by the sensor or traveling over the sensor can be detected.

Such a sensor 215 can be, for example, a conductor loop that is introduced into the respective road or a light barrier or an ultrasound sensor, with which a vehicle can pass the respective sensor in a predetermined direction for which the sensor 215 is intended .

Every time a vehicle passes sensor 215 , sensor 215 transmits a detection signal to a central computer 301 described below.

Alternatively, a sensor can also be provided in the sensor 215 , the counter being incremented for a predetermined period of time for each vehicle passing through the sensor 215 , and after the predetermined period of time has elapsed, the counter reading is transmitted to the central control computer 301 and the counter is then changed to a preset counter reading reset.

In the city 204 there are different requirements for the switching, ie the control of the traffic lights 214 , at different times of the day, since different types of traffic flows and different main loads occur at different times of the day within the traffic network 200 .

In the morning of one day, i.e. essentially between 6 a.m. and 9.30 a.m., there is primarily rush-hour traffic, which leads from residential area 202 to commercial area 203 , first shopping area 206 and second shopping area 207 .

In the mornings, i.e. essentially between 9:30 a.m. and 12:00 p.m. one day, a main traffic direction occurs from the residential area 202 to the first shopping area 206 and the second shopping area 207 , the flow of traffic corresponding to a shopping traffic of the residents of the city 204 .

In the afternoons, i.e. essentially between 12 p.m. and 4 p.m., in addition to shopping traffic, there is also rush hour traffic, mainly directed from commercial area 203 to residential area 202 .

In the evening, that is essentially between 4:00 p.m. and 9:00 p.m., the main traffic between the residential area 202 and the leisure center is recorded in the second shopping area 207 .

According to this exemplary embodiment, sensors 215 detect sensor occupancy B, which is defined as the time in which sensor 215 is occupied in relation to the time period during which occupancy is detected. The sensor assignment B can be determined, for example, by means of an induction loop as the sensor 215 . As an alternative, for example when recording a traffic parameter using a visual sensor, the traffic density ρ can be measured. The occupancy B, which is usually similar to the traffic density ρ, is thus obtained on a sensor 215 in accordance with the following regulation:

being with

• - tb is the time during which the sensor is occupied, d. H. during which a vehicle is over the Sensor is located, and
• - t is the length of time during which the number m of the vehicles is determined.

According to this exemplary embodiment, the average occupancy B of the sensor 215 is determined on each sensor 215 for a time period t of 15 minutes, and then the average occupancy B determined according to regulation (1) is transmitted to the central control computer 301 described below.

FIG. 3 shows the central control computer 301 , which is coupled to the sensors 215, for example via a radio connection or a wired connection 302 .

The control computer 301 has an input / output interface 303 and a central processor unit 304 and a memory 305 , which are each coupled to one another via a computer bus 306 .

Furthermore, via the input / output interface 303 via a first connection 307 , e.g. B. coupled via a cable or an infrared radio link, a computer mouse 308 with the control computer 301 .

A screen 310 is coupled to the input / output interface 303 via a second connection 309 .

Furthermore, a keyboard 312 is coupled to the input / output interface 303 via a third connection 311 .

According to this exemplary embodiment, a plurality of frame signal plans 313 are stored in the memory 305 of the control computer 301 .

The plurality of frame signal plans 313 is shown in the following table, with A1, A2, B1, B2, B3, C1, C2, D1, D2, D3, signal images for the first intersection 208 (B1, B2, B3), the second Intersection 210 (A1, A2), the third intersection 211 (D1, D2, D3) and the fourth intersection 213 (C1, C2), as shown in FIG. 4, are designated.

According to the exemplary embodiment, three frame signal plans RSP1, RSP2, RSP3 are stored in the memory 305 , as shown in the following table:

A frame signal plan has a set of so-called frame signals, each of which determines a traffic flow, in which time restrictions which states of the light signals acting on this traffic flow are permitted at the traffic lights 214 .

An example frame signal is shown in FIG. 5. A period of a light signal 501 of the frame signal 500 has a request area 502 and an extension area 503 .

Within this time frame, a local Optimization with regard to the goals mentioned below, in particular an optimization of the traffic flow, be carried out, for example by expanding Green phases or preferential treatment for the public Local transport.  

In the request area 502 , in particular in the case of pending traffic, that is to say in the case of vehicles standing at the traffic light 214 or approaching a respective traffic light 214 , green phases of the traffic light 214 can be initiated, which must be ended within the extension area 504 .

In FIGS. 4a through FIG. 4d are those of the validity of the respective signal image permissible traveling directions of the vehicles represented in the respective crossing during the period, that is by the arrows, respectively.

The figures in the above table to a respective signal image as shown in Fig. 4a through FIG. 4d, the duration of validity corresponding to the respective image signal per period of the respective frame signal plan.

For example, the first frame signal plan RSP1 indicates that a first signal image 401 shown in FIG. 4a has twice as long a validity period due to the number 60 compared to the second signal image 402 (assigned value number 30).

According to the second frame signal plan RSP2 and the third frame signal plan RSP3, the first signal image 401 and the second signal image 402 each have the same validity period (the same value number 45 is assigned to both signal images 401 , 402 ).

This clearly means that the traffic lights 214 are switched at the second intersection 205 due to the traffic light switching in such a way that the traffic flow shown in the first signal image 401 and the second signal image 402 is possible with the same weighting.

The first frame signal plan RSP1 specifies for the first intersection 208 in a third signal image 403 , fourth signal image 404 and fifth signal image 405 shown in FIG. 4b that the third signal image 403 is twice as long per period as the fourth signal image 404 and that fifth signal image 405 is not formed at all because of the traffic light switching of traffic light 214 at first intersection 208 (value number third signal image 403 : 60, value number fourth signal image 404 : 30, value number fifth signal image 405 : 0).

According to the second frame signal plan RSP2, the third signal image 403 and the fourth signal image 404 are weighted equally, and the fifth signal image 405 is not formed due to the traffic light control (value number third signal image 403 : 45, value number fourth signal image 404 : 45, value number fifth signal image 405 : 0 ).

According to the third frame signal plan RSP3, the fifth signal image 405 is weighted considerably more than the third signal image 403 and the fourth signal image 404 (value number third signal image 403 : 20, value number fourth signal image 404 : 20, value number) due to the traffic lights 214 at the first intersection 208 fifth signal pattern 405 : 50).

At the third intersection 211 , according to the first frame signal plan RSP1, the traffic lights 214 are switched on in such a way that the sixth signal image 406 shown in FIG. 4c is weighted half as much, that is to say only half the validity period compared to the eighth signal image 408 . The seventh signal image 407 is not generated at all in accordance with the first frame signal plan RSP1 (value number sixth signal image 406 : 30, value number seventh signal image 407 : 0, value number eighth signal image 408 : 60).

According to the second frame signal plan RSP2, the sixth signal picture 406 and the eighth signal picture 408 are weighted equally (value number sixth signal picture 406 : 45, value number seventh signal picture 407 : 0, value number eighth signal picture 408 : 45) and according to the third frame signal plan RSP3 is the seventh signal picture 407 weighted considerably more than the sixth signal image 406 and the eighth signal image 408 (number of values sixth signal image 406 : 15, number of values seventh signal image 407 : 65, number of values eighth signal image 408 : 10).

At the fourth intersection 213 , according to the first frame signal plan RSP1, the ninth signal image 409 shown in FIG. 4d is weighted twice as much, that is to say it has a validity period twice as long as the tenth signal image 410 (number of ninth signal images 409 : 60, Number of values tenth signal pattern 410 : 30).

According to the second frame signal plan RSP2 and the third frame signal plan RSP3, the two signal images 409 , 410 each have the same period of validity per period (number of values ninth signal image 409 : 45, number of values tenth signal image 410 : 45).

As can be seen from the table shown above, the first frame signal plan RSP1 represents a traffic light switching of the traffic lights 214 in the traffic network 200 which is optimized with respect to rush hour traffic.

The second frame signal plan RSP2 weights all connections in the traffic network largely uniformly, so that a good connection, that is to say a good traffic flow, is also possible between the first shopping area and the second shopping area 207 with regard to the respective requirements.

The third frame signal plan RSP3 is optimized with regard to the traffic between the residential area 202 and the second shopping area 207 located to the south, that is to say it preferably favors the traffic flow between the residential area 202 and the second shopping area 207 .

From the central control computer 301 , according to the reinforcement learning method described below, using fuzzy membership functions and fuzzy partitions, an optimized selection of the frame signal plans is provided to ensure a maximum gain, which according to this exemplary embodiment is the sum of the squared average relative traffic densities per route 1. For example, before an intersection, that is, the gain g of the reinforcement learning method described below for determining the optimized control strategy, that is to say control strategy, which is formed by the corresponding selection of the traffic densities ρ determined for the average occupancies B approximated, in connection with the reinforcement learning process, optimized selection of the frame signal plan RSP1, RSP2, RSP3 according to the following regulation:

being with

• - ρ _{l, max} the maximum possible traffic density and with
• - ρ _{l is} the average traffic density on route l at the end of a period of 15 minutes

referred to as.

The control computer 301 thus clearly has to learn a strategy that minimizes the sum of the profits g.

The basic idea of regulation (2) can be seen in the fact that the average traffic density in the traffic network 200 is to be minimized by the selection of the frame signal plans, the quadrature of the terms with respect to the individual routes l resulting in a homogeneous network state with average traffic densities on all routes l is rated better than a condition with very low traffic density on some routes l with congestion on other routes l.

In the exemplary embodiments described below, for all learning methods, the relative vehicle densities averaged over a period of 90 seconds in each case, which are formed according to the following regulation, are determined at the points in the traffic network where sensors 215 are present:

In FIG. 2 this is symbolically represented by representations of individual traffic density profiles 216 , 217 , 218 .

The relative traffic densities become non-linear in accordance with the following regulation:

distorted, so that in the area of low traffic density basically gives a higher resolution than in the area high traffic density.

A model description of the traffic network 200 and its control as a technical system in a general form as a finite state machine with a number of continuous states and continuous actions, on the basis of which a state transition from a previous state to a subsequent state is triggered, is described below.

The action space can be both continuous and discreet be.

In general, the technical system to be controlled is described according to the invention using the following components:
The technical system has a continuous state space ℵ of dimension d ^ℵ .

Furthermore, the technical system has a continuous action space A of dimension d ^A or a discrete space U.

Conditional probability density functions p ( y , x , a ) describe the probability of a transition from a state x to a state y when action a is carried out .

With a gain g ( x , a , y ) in the sense of reinforcement learning, a gain g ( x , a , y ) is described when an action a is carried out in the previous state x if the technical system changes into a successor state y due to the control due to action a .

The state space ℵ is in fuzzy partitions with fuzzy membership functions

grouped, for which the following applies:

The fuzzy partitions are with

denotes and each have a fuzzy center, which with

referred to as.

Furthermore, action space A is also in fuzzy partitions with membership functions

grouped, for which the following applies:

The fuzzy partitions of action space A are included

denotes and each have fuzzy centers

on.

According to the invention there are different options for Forming the fuzzy partitions of the state space ℵ provided.

It becomes fuzzy partitions

educated.

According to an alternative, to form the fuzzy Partitions of the state space ℵ a fuzzy C means  Clustering as described in [2] will.

According to a further alternative, it is provided to form the fuzzy partitions in a manner as shown in FIG. 6.

The relative traffic density is grouped in an interval from "0" to "1" into four partitions 601 , 602 , 603 , 604 , to which membership functions 605 , 606 , 607 , 608 are assigned over a predetermined interval.

A first fuzzy membership function 605 describes a very low traffic density "very small", a second fuzzy membership function 606 describes a low traffic density "small", a third fuzzy membership function 607 describes a high traffic density "high" and a fourth fuzzy membership function 608 a very high traffic density "very high".

The fuzzy centers and limits of the individual fuzzy membership functions and fuzzy partitions shown in FIG. 6 can alternatively be determined according to the following procedure.

State transitions of the technical system shown above ( x _k , u _k , x _{k + 1} , _k ) can be described by vectors ( x _k , x _{k + 1} , _k ) in a state transition space T: = ℵ '× ℵ "× , where ℵ 'and ℵ "denote the same state space ℵ.

The fuzzy cluster is also clustered performed in the state transition space T due to the observed state transitions during a learning phase Use of training data from a technical System can be determined, for example by measurement or also by simulating the technical system according to this  Embodiment with the determined traffic densities as Training data.

For every action u ∈ U separate clusters, that is fuzzy partitions used.

Clustering in state space ℵ performed using the observed conditions during the learning phase described above.

It should be noted that according to that described below Process the clustering of states and state transitions is performed incrementally so that none State transitions must be saved explicitly, such as this according to fuzzy-C-means clustering, but without be carried out according to another alternative can, would be required.

Result of fuzzy clustering, that is, the formation of the Fuzzy partitions with the associated fuzzy Membership functions are directly the fuzzy Partitions of the state space ℵ, which in the further described reinforcement learning process and the resulting control strategy can be used.

The clusters in the state transition room serve as compact Description of the observed state transitions from which the Model, that is, the conditional State transition probabilities as above have been described, and the profits g as below described, can be determined.

The clusters are also in the state transition space used to determine those described below optional intended splitting and merging of clusters during the formation of the fuzzy partitions under the incremental procedure.  

The splitting or combining of a fuzzy cluster is described with reference to Fig. 7a and Fig. 7b.

According to the situation described in FIG. 7a, it is assumed that a state transition from one state

in a state

and further from a state

in a state

with an identical profit of

is observed.

The middle cluster 701 of the three clusters 701 , 702 , 703 shown in FIG. 7a would, when split up, make it possible to distinguish between these two classes of state transitions in the discretized model in the course of learning.

In the example shown in FIG. 7b, in which all state transitions begin in a region of the middle cluster 701 and in a similar final state

end up, however, with two different classes of winnings

and

observed in the training phase, splitting the middle cluster 701 would enable an improved differentiation of these classes in the state transitions.

It can thus be seen that in the cases illustrated in FIG. 7a and in FIG. 7b, splitting the middle cluster 701 each time would improve the learning process and the fuzzy set of fuzzy partitions formed by the learning process.

A corresponding procedure can be carried out according to an optional Extending the approach by uniting individuals Fuzzy partitions, i.e. clusters, can be reached where when merging basically an analog The approach chosen is compared to dividing the Partitions.

The following sections of the procedure to form the fuzzy partitions, that is, clustering of the state space ℵ and in the state transition space T that  Increase the accuracy of the clusters in the state space ℵ because of the clusters in T and finally deriving the discretized model from the clustered State transitions described.

The clustering of state space ℵ in fuzzy partitions becomes used to describe a Q- described below Function in connection with a reinforcement Learning process.

The clusters are created in an incremental manner.

Each cluster c ℵ | i is characterized for the respective iteration k by the respective cluster center ℵ | i, k, a counter value M ℵ | i, k for counting the number of states that the cluster c ℵ | i based on the previous one Process steps, that is to say iterations, have been assigned and a diagonal matrix A ℵ | i, k, which is also referred to below as a scaling matrix, by means of which the size of the respective cluster is determined.

Furthermore, the entirety of all clusters in the State space ℵ for an iteration k denoted by C ℵ | k.

A distance dist ℵ | k ( x , c ℵ | i) of a state x ∈ ℵ from a cluster c ℵ | i is given by the following rule:

Because the diagonal shape of the scaling matrix A ℵ | i, k, which is generally not required in accordance with this exemplary embodiment, all clusters are symmetrical in all dimensions. However, the scaling of the dimensions can be varied.

At the beginning of the procedure, all clusters are initialized with the same scaling ^matrix  ^ℵ .

As will be explained in more detail below, is due to dividing a cluster into two clusters with respect a dimension d a reduction in the size of each Clusters achieved in the respective dimension d.

If a new state x _{k is} determined during the learning phase, the distance between the newly determined state x _k and all existing clusters is determined.

If there is no cluster c ℵ | i to which the distance dist ℵ | k ( x , c ℵ | i) of the new state x _{k is} smaller than a predetermined maximum distance d ℵ | max, then a new cluster c ℵ | i, with a new center

and a new counter initialized to the value "0"

and a new scaling matrix

generated.

The maximum distance d ℵ | max can be specified by the user and usually depends on the initialization diagonal matrix A ℵ | i, k and the desired size of the initialized clusters.

The cluster c ℵ | i₀ ∈ C ℵ | k, to which the new state x _{k has} the smallest distance, is shifted in a further step in the direction of the newly determined state x _k within the state space ℵ.

The step size of the respective shift step is determined by the fuzzy membership function according to the following rule:

of the new state x _k in the cluster c ℵ | i₀ and the number of states that were previously assigned to the cluster c ℵ | i₀ are denoted by M ℵ | i₀, k, which results in a new counter value M ℵ | i₀, k +1 and a new, updated center ℵ | i₀, k + 1 of the respectively selected cluster c ℵ | i₀ result according to the following regulations:

This alternative approach can be illustrated as one incremental variant of the fuzzy-C- described in [2] Means clustering process can be considered.

According to this embodiment, a fuzzification value m used in regulation (24) with the value 2.

In an alternative procedure, it is possible to shift the centers of all clusters in the direction of the newly determined state x _k instead of just the selected center ℵ | i₀, k + 1.

The aim of the clustering described below State transition space T is a compact description of the observed state transitions during the learning phase produce.

As will be described hereinafter, this description will be used to make sensible division of clusters in the State space ℵ and to estimate the average State transition probabilities described above have been estimated, as well as to estimate the profits g used.

A cluster c T, u | j in the state transition space T is characterized by its cluster centers T, u | j, k, which result from the following rule:

M T, u | j, k denotes a counter with which the number of State transitions are given that this particular Clusters are assigned.  

With a scaling matrix A T, u | j, k, and with a subscript u for the action that created the respective state transition, which state transition is assigned to the respective cluster.

The entirety of the clusters of state transitions to one Action u ∈ U is denoted by C T, u | k.

The scaling matrix A T, u | j, k comprises three independent diagonal matrices B T, u | j, k, C T, u | j, k, b T, u | j, k, whereby a first diagonal matrix B T, u | j, k describe the respective previous state, a second diagonal matrix C T, u | j, k a successor state and a third diagonal matrix b T, u | j, k describe the gain that is generated by the state transition.

This results in the scaling matrix A T, u | j, k the following rule:

To determine if a cluster is split into two Cluster along a dimension d in the state space ℵ is sensible, the resolution of the clustering in the State transition space T depending on the resolution of the Clustering in the state space ℵ can be selected.  

It is assumed that c ℵ | i'₀ is a cluster in the state space ℵ is which cluster of component T, u | j, k of the cluster Center of the cluster c T, u | j is closest and with c ℵ | i "₀ the cluster is designated which of the component T, u | j, k am next lies.

According to the heuristic approach in this Embodiment has proven to be advantageous exposed the size of the cluster c T, u | j in the direction Halb 'to be half the size of the cluster c ℵ | i'₀ and to choose the size of the cluster c T, u | j in the direction ℵ "so large like the size of the cluster c ℵ | i'₀.

In this way, the first diagonal matrix B T, u | j, k and the second diagonal matrix C T, u | j, k of the cluster c T, u | j result according to the following regulations:

where A ℵ | i'₀k and A ℵ | i "₀, k denote the scaling matrices of the cluster c ℵ | i'₀ and c ℵ | i" ₀, respectively.

The scaling matrices A ℵ | i'₀k, A ℵ | i "₀k depend on the number of divisions of the clusters c ℵ | i'₀ and c ℵ | i" ₀ up to the iteration k.

The third diagonal matrix b T, u | j, k is chosen to be constant, for example according to the following rule:

when winnings are differentiated should.

On the basis of the scaling matrices A T, u | j, k shown above, a distance measure dist T | k ( z , c T, u | j) is determined in accordance with the following rule:

If a new state transition ( x _k , u _k , x _{k + 1} , _k ) is determined, it is checked whether at least one cluster

to which the vector exists

has a distance that is smaller than a predetermined one maximum state transition distance d T | max.

If this is not the case, then a new cluster c with a cluster center

a new counter initialized with the value "0"

and a new scaling matrix A in the entirety of all clusters.

The maximum state transition distance d T | max can, must however, do not have the same value as the maximum Distance d ℵ | max with respect to the state space ℵ.

The smaller the maximum state transition distance d T | max is selected, the finer the state transition space T clustered.

For

each state transition in state transition space T is explicitly stored in the memory of control computer 301 .

In a further step, all clusters c ∈ C in the direction of the vector z _k according to their respective affiliation, which results according to the following rule:

shifted and the counter of the respective cluster is increased, so that updated values of the counter M and of the respective cluster center result in accordance with the following regulations:

The division of a cluster c ℵ | i ∈ C ℵ | k in is clear Dimension d in the state space ℵ makes sense if there is a more detailed modeling of the state transition Probabilities or the profits enabled.

This is the case if two clusters c T, u | j and c T, u | l in the State transition space T exist, both of which are high Membership value v u | d, j, l, k (c ℵ | i) to the cluster that split should have, and their centers a clear Have a distance from each other with respect to the direction ℵ "×.

Thus, a cluster c ℵ | i ∈ C ℵ | k is divided in the direction of dimension d during an iteration k if the value vu | d, j, l, k (c ℵ | i), which is formed according to the following rule:

exceeds a predeterminable threshold value v ^min for at least one pair of clusters c T, u | j, c T, u | l ∈ C ^{T, u} and an action u ∈ U, that is to say:

In the regulation (41) shows the sigmoid function

indicates whether (T, u | l, k) _{d is} greater than (ℵ | i, k) _d .

The regulation shows accordingly

indicates whether (T, u | j, k) _{d is} smaller than (ℵ | i, k) _d .

With the function

it is indicated whether the clusters c T, u | j and c T, u | l are clearly separated from one another in the direction of the dimension ℵ "×, the distance dist (c T, u | j, c T, u | l ) is given according to the following regulation:

With the distance dist ℵ "| k (c T, u | j, c T, u | l)

in the state space and the distance dist (c T, u | j, c T, u | l)

in the space of profits through the state transitions to be generated.

According to the exemplary embodiment, it has proven to be advantageous to select the individual parameters in accordance with the following regulations:

If the criterion according to (41) is met by the cluster c ℵ | i₀ and the dimension d ₀ , the cluster c ℵ | i₀ is replaced by two new clusters c ℵ | i 'and c ℵ | i ".

The dimension d _{0 of} the cluster centers of the clusters c ℵ | i 'and c ℵ | i "are in opposite directions with respect to the dimension d ₀ by half the radius

of the cluster c ℵ | i₀ shifted, the other dimensions of the original cluster c ℵ | i₀ also for the new clusters c ℵ | i 'and c ℵ | i "remain unchanged.

The following update rules thus result for the new clusters c ℵ | i 'and c ℵ | i ":

The size of the new clusters c ℵ | i 'and c ℵ | i "in the direction of dimension d ₀ is halved, that is, the size, that is, the scaling matrix of the new clusters c ℵ | i' and c ℵ | i "the following update rules:

The counters of the new clusters c ℵ | i 'and c ℵ | i "are changed to the set the same value as the counter of the original Clusters c ℵ | i₀.

This results in the following update rules for the counters of the new clusters c ℵ | i 'and c ℵ | i ":

so that the new clusters adapt to newly determined states x _k at the same speed as the original cluster c ℵ | i₀ would have done.

Because of the size adjustment of each cluster in the State transition space T to the size of the neighboring one Cluster in the state space ℵ divides the Clusters in the state space ℵ also to a higher one Dissolution of the clustering in the state transition space T.

This can lead to further division of the clusters.

Thus the fuzzy partitioning of the state space ℵ can be chosen exactly as desired, if any Splitting a cluster into a more accurate internal one Model description leads.

However, creating clusters can be done in two ways be restricted.

For one thing, there can be a maximum number of divisions based on a cluster may be used.

Furthermore, the threshold value v ^min , with which the division of the clusters is controlled, can correspond to the number of existing clusters | C ^ℵ | increase.

As will be explained in more detail below, the Basis of the determined clusters c T, u | l ∈ C T, u | k and the counter M T, u | l, k assigned to the respective cluster c T, u | l, with the the number of state transitions that this particular Clusters are assigned to be determined.

With

can be estimated how often action a is carried out has been in the state c ℵ | i and how often the  State transition has been observed through the cluster c T, u | l is described.

Thus, the quotient q _{i, l, k} (u), which is formed according to the following rule:

estimated the likelihood that the Action u in the state c ℵ | i in a state transition that is described by the cluster c T, u | l.

Therefore, the average probability p _{i, j, k} (u) of a state transition from a previous state c ℵ | i to a successor state c ℵ | j can be formed by an approximate probability _{i, j, k} (u), according to the following rule:

can be estimated.

Accordingly, the average profit for executing the action u in the state c ℵ | i and a state transition to the state c ℵ | j can be approximated according to the following rule:

In this context, it should be noted that the above described methods for forming fuzzy clusters as well regardless of the reinforcement described below Learning process related to the selection of Framework signal plans, generally related to the Control of a technical system that can be used.

The procedure described above can be clearly seen in this that a cluster of a state space or a State transition space in at least two or more clusters is divided if from the clustered state transitions it can be seen that by dividing different Groups of state transitions, for example different successor states and / or different Profits are generated that are differentiated from one another can.

This procedure can clearly be seen as a kind Midway between an explicit storage of all State transitions and the mere counting of state transitions between given partitions of the state space will.

This way, the benefits of being explicit Storage, namely a very good partitioning of the State space and the counting of state transitions, the is called a very compact representation of a model of the technical system, according to that described above Approach to be united.

It should be noted that the on the above Partitioning determined in this way as well as one  alternatively possible, fixed, d. H. manual Partitioning the fuzzy partitions the reinforcement Learn considerably as described below accelerates.

Using determined training data as well as the on the fuzzy partitions determined as described above, that is, the fuzzy clusters become one in the further described reinforcement learning process.

To facilitate understanding, a Brief overview of the basics of reinforcement learning given.

The basic idea of model-based reinforcement learning is a maximum likelihood at the beginning of the learning process Estimate the model of the system to be controlled and the optimized control strategy, that is optimized taxes through selection of tax parameters (indirect) based on the previously determined model description work out.

These two phases can overlap each other, that is, before trained strategies can be determined from the one initially Model description can be derived based on observed state transitions during a learning phase, and the information for a future derivation of the Tax strategy, that is, the selection of tax parameters, can by means of these control strategies.

In a discrete indirect reinforcement learning process there is a maximum likelihood estimate of the model of the technical system based on discrete counters, with which the number of actions carried out and the resulting state transitions and on the basis of Variables for the observed gains.  

The counters and variables are explained in more detail below.

With N 0 | i, u, k and M 0 | i, u, j, k, i = 1,. . ., N ^ℵ , u = 1 ^,. . ., N ^A, j = 1,. . ., N ^ℵ , k ∈ N, are counters with which the number of fuzzy actions A _u carried out in a fuzzy state X _i and the number of state transitions from a state X _i to a successor state X _j due to the action A _u referred to an iteration k.

If a state transition ( x _k , a _k , x _{k + 1} , g _k ) is observed, x _k ∈ ℵ, x _{k + 1} ∈ ℵ, a _k ∈ A, g _k ∈, the counters N 0 | i, u , k and M 0 | i, u, j, k increased according to the degree of belonging to the corresponding cluster centers according to the following regulations:

The counters N 0 | i, u, k and M 0 | i, u, j, k are then used to calculate the average conditional probabilities based on this

for a state transition from a state X _i to a subsequent state X _j estimated on the basis of the action A _u according to the following rule:

Furthermore, r 0 | iuj denotes the average profit that is obtained if, in the predecessor state X _i, due to the execution of the action A _u, the successor state X _{j is} assumed in the state space ℵ.

The profit r 0 | iuj thus results according to the following rule:

An estimate of the respective profit r 0 | iuj, that is to say an estimated profit 0 | iuj, is determined in accordance with the following update rule:

With

i = 1,. . ., N ^ℵ , u = 1 ^,. . ., N ^A, j = 1,. . ., N ^ℵ (74)

when observing a state transition ( x _k , a _k , x _{k + 1} , g _k ), x _k ∈ ℵ, x _{k + 1} ∈ ℵ, a _k ∈ A, g _k ∈.

For this discrete model

can an optimal control strategy according to the Reinforcement learning processes can be determined.  

Q ( x , a ) denotes the true, continuous Q value in the context of the reinforcement learning process, which is formed according to the following rule:

On the basis of the true, continuous Q value Q ( x , a ), an estimated Q value 0 | iu of the average Q values is obtained according to the following rule:

which results from the fixed point solution of the following system of equations:

According to this exemplary embodiment, the continuous Q values Q ( x , a ) are approximated by a so-called Takagi-Sugeno-fuzzy system, as described in [3], with linear terms in the consequences of the fuzzy rules in accordance with the following regulation:

if x is X _i and a is A _u

then

with i = 1,. . . N ^ℵ , u = 1 ^,. . ., N ^A , where:

Due to the orthogonality of the fuzzy membership functions, regulation (79) can be written as the following regulation:

The terms Q 0 | iu can be determined by determining the fixed point solution of the Systems of equations (78) with the estimates 0 | ij of average conditional State transition probabilities according to regulation (70) and estimates 0 | iuj of average profits according to Regulation (72) can be determined.

For the discrete case there is a special one in [3] Implementation of the procedure for recursive solution of the so-called Bellmann equation (78) described.  

The basic idea of the approach known from [3] is that recursively update the Q values according to the change prioritize the Q values as they come from the update result.

Because of this approach, the speed of the Convergence of the fixed point solution increased significantly compared to an update according to a fixed order.

Since the interpretation of the variables 0 | ij (u) and  0 | iuj, k + 1 of the Bellmann equation (78) in the discrete case is the same, this can be advantageous Update mechanism also for that according to this Embodiment of the invention provided approach under Use of fuzzy partitions as part of the reinforcement Learning method can be used.

The constant terms Q 0 | iu are obtained by solving the Bellmann Equation (78) determined.

The associated partial derivatives Q and Q can by averaging and partial Derivatives of the profit function and the conditional State transition probabilities can be determined.

The partial derivatives Q are formed according to the following rule:

with the abbreviations:

which were used in the previous step.

The replacement of the integral by the sum of local integrals according to the regulations (86) and (87) and the Averages (88), (89) is consistent in the sense  that with increasing the accuracy of partitioning the State space these are getting better and better.

In an analogous way it can be shown that:

The average local gain r 0 | iuj and the average local derivatives r and r of the gain function g can be estimated by adjusting the parameters 0 | iuj,, and the following linear function, depending on the gains in the vicinity of the cluster centers ( _i , _u , _j ), according to the following regulation:

This adaptation can take place by means of a known gradient descent, taking into account an error function E, which results according to the following rule:

when observing a state transition ( x _k , a _k , x _{k + 1} , g _k ).

According to this exemplary embodiment, the following update rules thus result:

a possible choice for the step size η _{iuj, k} can be given within the update according to the following rule:

so that the step size η _{iuj, k is} selected depending on the degree of belonging of an observed state transition to a cluster center and is reduced over time.

The average conditional probabilities p 0 | ij (u) can be estimated according to regulation (71).

The average partial derivatives p and p can be approximated according to the following rules:

where e d | l denotes a vector of dimension d with vector components ed | l, i = δ _il .

N denotes a counter with which the number of executions of an action A _{u is} counted in a fuzzy state which arises when state X _{i is} shifted along dimension l by a predeterminable value ε.

M denotes another counter with which the number of state transitions from the state X _i shifted by ε along the dimension l to a subsequent state X _j is counted on the basis of the action A _u .

In addition, N denotes a counter with which the number of actions A _u carried out is given in the state which arises by shifting from state X _i along dimension l by a negative value −ε, and M denotes another counter, with which the number of state transitions to state X _j from this state is specified on the basis of action A _u .

When determining a state transition ( x _k , a _k , x _{k + 1} , g _k ), the individual counters N, M, N, M are updated in accordance with the following update regulations:

Accordingly, counters N, M, N and M for the action space are updated in accordance with the following update regulations:

The local partial derivatives p and p are then determined in accordance with the following regulations:

With the estimated probabilities 0 | iuj, k + 1, and for the probabilities p 0 | iuj, p and p and the estimates 0 | iuj, k + 1, and for the gains r 0 | iuj, r and r can now be the respective local partial derivative Q and Q according to Regulations (85) and (90) can be determined.

In summary, the reinforcement learning process can be in the form of a pseudo code can be described as follows:

1. Initialization

for i = 1,. . ., N ^ℵ , u = 1 ^,. . ., N ^A do

(a) N 0 | iu ← 0
(b) N ← 0, N ← 0, l = 1,. . ., d ^ℵ
(c) N ← 0, N ← 0, l = 1,. . ., d ^A
(d) M 0 | iuj ← 0, j = 1,. . ., N ^ℵ
(e) M ← 0, M ← 0, j = 1,. . ., N ^ℵ , l = 1 ^,. . ., d ^ℵ
(f) M ← 0, M ← 0, j = 1,. . ., N ^ℵ , l = 1 ^,. . ., d ^A
(g) r 0 | iuj ← 0, j = 1,. . ., N ^ℵ
(h) r ← 0, r ← 0, j = 1,. . ., N ^ℵ , l = 1 ^,. . ., d ^ℵ
(i) r ← 0, r ← 0, j = 1,. . ., N ^ℵ , l = 1 ^,. . ., d ^A
(j) PQueue ← empty
(k) Observe initial state x ₀
or

2. Main program

for k = 0, 1, 2,. . . do

• a) Select fuzzy action u _k in the current state x _k according to the exploration strategy (e.g. Boltzmann exploration / F-ISE exploration). Choose continuous action a _k from the set of states belonging to A _{u k} ≠ 0.
• b) Perform action a _k and observe the successor state x _{k + 1} and g _k = g ( x _k , a _k , x _{k + 1} )
• c) for i = 1,. . ., N ^ℵ , j = 1 ^,. . ., N ^ℵ do
  • a) Counting the state transitions
    
  • b) Estimating the state transition probabilities
    
  • c) Estimating the partial derivatives of the state transition probabilities
    
  • d) Calculate the deviation from the expected local profit
    
  • e) Update the average profit and average deviation estimates
    
• d) for i = 1,. . ., N ^ℵ do
  • a) Calculate the priority of saving for (i, u _k ):
    
  • b) if P <Φ _k then add (i, u _k ) to PQueue with priority P fi or
• e) while PQueue ≠ empty do
  • a) (i, u) ← first (PQueue)
    
  • b) for all predecessors (l, w) of i, ie all pairs (l, w) with M 0 | lwi <0 do
    
  • c) if P <Φ _k then add (l, w) to PQueue with priority P fi or
• f) Estimating the derivatives of the Q values
  

The optimal control strategy, that is, the optimal selection of a frame signal plan based on the determined, measured relative traffic density at the respective sensors 215 , generally formulated as an optimal control strategy µ: ℵ → A, is achieved in that the action a is selected in the respective state x , that is to say, for example, that frame signal plan is selected according to the exemplary embodiment that promises a profit according to regulation (79) that is maximum, that is to say the following applies:

The method described above can continue according to the in further described embodiment of the invention be improved.  

To the number of training steps required within the To reduce reinforcement learning process it is useful targeted the expected profit in the sense of a Information content of the training data on the technical To use the system, in other words, in everyone State to perform that action by either a great immediate, that is, immediate profit Information can be expected or through which an area is achieved in the state space in which high profits Information can be expected.

According to this embodiment, a model-based Exploration strategy planned.

The procedure described below is based on A values A _iu , i = 1 _,. . ., N ^ℵ , u = 1 ^,. . ., N ^A , with which the "attractiveness" of executing the respective fuzzy action A _u in the state X _i is designated.

Carrying out an action in a state of the state space ℵ then results in a high probability of a high information gain if a large immediate gain of information can be expected due to the execution of the action A _u , or if the technical system to be controlled is based on the Action passes into states in which a large amount of information can be expected.

Thus the relation between the A values A _{iu is} very similar to that of the Q values in connection with the Q learning process.

In the following, â _iu denotes the immediate information _gain which results from a single execution of the action A _u in the state X _i .

An estimated A value  _{iu is then} derived, which denotes the expected immediate information gain that results from future executions of the action A _u in the state X _i .

Finally, an overall attractiveness A _{iu is} determined recursively based on the  _iu .

The instant information can be gained by the amount of Knowledge that the learning system is measured about State transition probabilities between the fuzzy Partitions due to observation of a Receives state transition.

A maximum change

in the state transition probabilities of a state X _i and an action A _u , which result from an observed state transition ( x _k , a _k , x _{k + 1} , g _k ), is given by the membership of ( x _k , a _k ) to the individual fuzzy partitions, designated by:

In this way, the change in the probabilities is calibrated with an upper limit, which is formed according to µ ℵ | i ( x _k ) µ A | u ( a _k ), in order to make the measure of the immediate information gain independent of the position of ( x _k , a _k ) within the respective fuzzy partition.

Thus, for the update of the immediate information gain from one iteration k to the next iteration k + 1:

From the immediate information gains determined in accordance with regulation (115) due to the execution of the action A _u in the state X _i , it is possible to draw conclusions regarding expected future information gains.

It has proven to be advantageous to use a weighted one Sum of all previous immediately determined Calculate information gains.

The influence of an information gain for a state X _i and an action A _u on the immediate "attractiveness" should be limited by the affiliation of the corresponding state transition into the respective fuzzy partitions.

This can be achieved by weighting previous information gains according to the sum of the degrees of affiliation of the following observations:

In the following algorithm, the immediate attractiveness is described as a quotient of the weighted sum of the immediate information gains and the sum of the weights, that is, the immediate attractiveness results according to the following rule:

A total attractiveness _{iu of} a state-action pair (X _i , A _u ) is determined recursively according to the following rule:

with the spatial damping factor λ ∈ [0; 1] and the attractiveness _{j of} the partition subset X _j , given according to the following rule:

In summary, the exploration strategy can following procedure shown in a pseudo code to be discribed:

1. Initialization

• a) N 0 | iu ← 0, i = 1,. . ., N ^ℵ , u = 1 ^,. . ., N ^A
• b) M 0 | iuj ← 0, i = 1,. . ., N ^ℵ , u = 1 ^,. . ., N ^A, j = 1,. . ., N ^ℵ
• c) Initialize the components of the immediate attractiveness as if the maximum immediate information gain with the maximum degree of membership had been achieved in each previous iteration:
  Each state-action pair (X _i , A _u ) is thus initialized with the maximum immediate attractiveness  _iu = 1.
• d) Initialize total attractiveness
  
• e) Determine initial state x ₀

2. Main program

for k = 0, 1, 2,. . . do

• a) Let A _{u k be} the partition subset (fuzzy action) of the action space, in which the attractiveness _u ( x _k ) is maximized in the current state x _k , the attractiveness _u ( x _k ) being given by
  if x is X _i then _u ( x ) = _iu
  That is, the following applies:
  Randomly selecting an action a _k , from
  
• b) Execute action a _k and observe the successor state x _{k + 1} and the profit g ( x _k , a _k , x _{k + 1} )
• c) Executing an iteration of any reinforcement Learning process, for example the F- PS learning process or the F-Q learning process
• d) for i = 1,. . ., N ^ℵ do
  • a) Counting the state transitions:
    
  • b) Calculating the immediate information gain resulting from the state transition:
    
  • c) Recalculating the immediate attractiveness for (X _i , A _{u k} ):
    
  • d) Re-estimate the state transition probabilities:
    or
• e) for i = 1,. . ., N ^ℵ do
  • a) Calculate the priority of saving for (X _i , A _{u k} ):
    
  • b) if P <Φ _k then add (i, u _k ) to PQueue with priority P fi
• f) while PQueue ≠ empty do
  • a) (i, u) ← first (PQueue)
    
  • b) for all predecessors (l, w) of i, ie all (l, w) with M 0 | lwi <0 do
    

In summary, the method described above is explained again with reference to FIG. 1.

In a first step, data about the technical system, in the case of a traffic network 200, the respective traffic density at a sensor point is determined by means of a sensor (step 101 ).

In a further step, fuzzy partitions of the state space and / or the action space are determined (step 102 ).

In a further step, a reinforcement learning process is carried out using the determined data on the technical system and using the determined fuzzy partitions (step 103 ).

In a further step (step 104 ), an optimal control strategy is determined in the manner described above in accordance with the reinforcement learning method, that is to say an optimal starting value is determined, with which it is specified which frame signal value is to be selected for the respective iteration (step 104 ).

As is further illustrated in FIG. 1, in a further step (step 105 ) the optimal frame signal plan determined according to the reinforcement learning method is selected, read out and, depending on the frame signal plan, the traffic lights 214 at the respective intersections, i.e. generally the technical one System to be controlled, taking into account the selected optimized control strategy and the selected frame signal plan, controlled (step 106 ).

It should be noted that the above Invention not on controlling traffic lights in one Traffic network is limited, but that the fuzzy Partitioning a continuous state space and / or a continuous action space for any technical system that works with a continuous State space and / or continuous action space is described and by means of a reinforcement Learning process should be controlled.  

The following publications are cited in this document:

[1] H. Takagi and M. Sugeno, Fuzzy Identification of Systems and its Application to Modeling and Control, IEEE Transactions on Systems, Man and Cybernetics, Vol. 15, pp. 116-132, 1985
[2] JC Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms, Plenum Press, New York, ISBN 0-306- 40671-3, 1981
[3] A. Moore and C. Atkeson, Efficient Memory Based Reinforcement-Learning: Efficient Computation with Prioritized Sweaping, Information Processing, Vol. 5, pp. 263-270, 1992
[4] S. Davies, Multi Dimensional Triangulation and Interpolation for Reinforcement-Learning, Advances in Neural Information Processing Systems, NIPS'9, pp. 1005-1011, 1996

Claims (15)

1. Method for computer-aided determination of a control strategy for a technical system,
in which the technical system is described with a continuous state space and an action space,
in which the state space has states that the technical system can assume,
in which the action space has actions which are carried out in order to produce a state transition from a previous state of the state space to a subsequent state of the state space,
in which an assessment of the state transition takes place,
in which a model of the technical system is determined using training data that describe the technical system by forming fuzzy membership functions with which at least the state space is described and
in which a reinforcement learning procedure is carried out using the fuzzy membership functions,
whereby a control strategy is determined for each state of the state space,
whereby the optimal actions of the action space are learned. 2. The method according to claim 1, where for each state of the state space and the corresponding actions of the action room a Q-value is determined as a control strategy. 3. The method according to claim 1 or 2, where in the conclusions of the fuzzy rules of the fuzzy Systems, which according to the reinforcement learning process linear terms are used. 4. The method according to any one of claims 1 to 3,  in which the reinforcement learning process is carried out by selecting such actions during training that meet a specified criterion. 5. The method according to claim 4, where the criterion is an information gain about the conditional transition probabilities within of the reinforcement learning process. 6. The method according to any one of claims 1 to 5,
in which a frame signal plan is selected on the basis of the control strategy, and
in which control signals are transmitted to traffic lights of a traffic network on the basis of the selected frame signal plan. 7. The method according to any one of claims 1 to 6, where for each fuzzy partition in the state space and Action space an information gain is determined from earlier versions of fuzzy Actions belonging to the partition in the corresponding states has resulted. 8. The method according to claim 7, where from a gain of information a future one Profit is estimated. 9. The method according to any one of claims 1 to 8,
the counter is provided, with which the number of executions of actions in a state of the technical system and the number of state transitions from an initial state to a subsequent state is specified on the basis of the action up to the current iteration,
in which the values assigned to the counters are updated when a new state transition is determined as a function of the degree of belonging of the states or of the state transitions to the respective fuzzy clusters. 10. The method according to claim 9, where the state transition probabilities are within the reinforcement learning process depends on the counters be determined. 11. The method according to any one of claims 1 to 10, in which fuzzy partitions were formed at the beginning of the process by using an iterative process based on a given set of output partition subsets these are divided into several fuzzy partition Subsets or merged from multiple fuzzy Partition subsets into a fuzzy partition, depending on the determined training data. 12. The method according to any one of claims 1 to 10, where the fuzzy partitions according to the fuzzy C-means clustering process. 13. Fuzzy control device for determining a control strategy for a technical system, with a processor that is set up in such a way that the following steps can be carried out:

• - the technical system is described with a continuous state space and an action space,
• the state space has states that the technical system can assume,
• the action space has actions which are carried out in order to generate a state transition from a previous state of the state space to a subsequent state of the state space,
• - there is an assessment of the state transition,
• - With training data that describe the technical system, a model of the technical system is determined by forming fuzzy membership functions with which at least the state space is described and
• - Using the fuzzy membership functions, a reinforcement learning process is carried out, as a result of which a control strategy is determined for each state of the state space, whereby the respectively optimal actions of the action space are learned.

14. Computer-readable storage medium in which a computer program for determining a control strategy for a technical system, which, when executed by a processor, has the following method steps:

• - the technical system is described with a continuous state space and an action space,
• the state space has states that the technical system can assume,
• the action space has actions that are carried out in order to generate a state transition from a previous state of the state space to a subsequent state of the state space,
• - there is an assessment of the state transition,
• - With training data that describe the technical system, a model of the technical system is determined by forming fuzzy membership functions with which at least the state space is described and
• - Using the fuzzy membership functions, a reinforcement learning process is carried out, as a result of which a control strategy is ascertained for each state of the state space, whereby the respectively optimal actions of the action space are learned.

15. Computer program element for determining a control strategy for a technical system, which, when executed by a processor, has the following method steps:

• - the technical system is described with a continuous state space and an action space,
• the state space has states that the technical system can assume,
• the action space has actions which are carried out in order to generate a state transition from a previous state of the state space to a subsequent state of the state space,
• - there is an assessment of the state transition,
• - With training data that describe the technical system, a model of the technical system is determined by forming fuzzy membership functions with which at least the state space is described and
• - Using the fuzzy membership functions, a reinforcement learning process is carried out, as a result of which a control strategy is ascertained for each state of the state space, whereby the respectively optimal actions of the action space are learned.