JPH0850548A - Method and device for learning path - Google Patents

Abstract

PURPOSE:To apply this method/device to more general-purpose problems as well by continuously handling an input/output parameter in a fixed state while using only the limited pieces of action. CONSTITUTION:This method is composed of a 1st step 101 for arranging membership functions in respective discretely distributed states, 2nd step 102 for selecting one of adjacently arranged states at the time of transiting an object, 3rd step 103 for storing the state, through which the object passes, as the history of a state number, 4th step 104 for erasing state number history when the state number of the 2nd step 102 is matched with the state number of the 3rd step 103, 5th step 105 for deciding and executing the action of the object by performing fuzzy arithmetic processing, 6th step 106 for avoiding an obstacle when transiting to the next state, 7th step 107 for evaluating the result of the action of the object until arriving at a goal point based on the traveling distance from a start point to the goal point, and 8th step 108 for distributing the reward corresponding to the action evaluated result to the parameters of the membership functions, and by repeating these operations through these steps, learning is converged so that the object can adopts the optimum action (9th step 109).

Description

Detailed Description of the Invention

[0001]

INDUSTRIAL APPLICABILITY The present invention can be used in fields relating to applications such as combinatorial optimization problems and shortest path search. In particular, it relates to a mobile device having a learning function.

[0002]

2. Description of the Related Art Recently, a reinforcement learning method, which is one field of artificial intelligence, has been attracting attention. This reinforcement learning is a method that tries to adapt the object to the environment by using a special input called reward as a clue. Machine Learnig, Vol.3, pp225-245 (1988) is a paper on conventional reinforcement learning methods.

The problems targeted for reinforcement learning are classified into classes according to the nature of state transitions and the types of input / output variables. The nature of the state transition in this paper is that the state is fixed and the number of actions that can be taken in each state is limited to several.
Also, the types of input / output variables are discrete. Looking at the reward distribution method of this reinforcement method, reward is an experiential reinforcement learning method in which actions are distributed as weights in each state.

As another typical conventional learning method, US Pat. No. 5,113,482 proposes a reinforcement learning method incorporating a neural network capable of continuously treating input / output variables.

[0005]

The above-mentioned conventional experiential reinforcement learning method can handle only limited problems and has a small degree of freedom. Further, the reinforcement learning method incorporating a neural network has a drawback that the number of neurons increases and the memory becomes huge. The present invention adopts fuzzy reasoning as a reinforcement learning method, and can handle input / output variables continuously in a fixed state with a few limited actions, and can be applied to more general-purpose problems. The object is to provide a learning device.

[0006]

The route learning method includes a first step of initializing a membership function in each state that is discretely distributed, and the target from the current state to the next state. Upon transition, a second step of selecting a state from among the states arranged in the vicinity thereof, a third step of storing the state passed by the object as a history of state numbers, and the second step When it is determined that the state number selected by the step number and the state number stored in the third step match, a fourth step of deleting the looped state number history as a route loop, and Fuzzy arithmetic processing from the membership function and the current position of the target to determine and execute the action of the target. And a sixth step of avoiding an obstacle when the target transits to the next state, and a behavior result of the target until the target reaches the goal point from a start point to a goal point. A seventh step of behavior evaluation evaluated based on the traveling distance, and a reward distribution eighth step of distributing the reward according to the behavior evaluation result of the seventh step to the parameters of all membership functions arranged in each passing state By sequentially repeating the steps and the second to eighth steps,
And a ninth step of converging learning so that the object takes an optimal action.

Further, the route learning apparatus of the present invention comprises an initialization processing means for initializing a membership function in each of the discretely distributed states, the membership function and the present position of the object. A route learning processing unit for reinforced learning of a route by deciding an action while performing fuzzy arithmetic processing and distributing rewards according to the action evaluation result, and the target according to the route reinforced based on the result of the route learning processing unit. And a route movement processing unit for moving the vehicle.

[0008]

The present invention employs fuzzy reasoning in the empirical reinforcement learning method so that problems that can only be treated discretely can be treated continuously. A route learning device according to a second aspect of the present invention incorporates the route learning method according to the first aspect.

The first state selecting means of claim 2 of the route learning apparatus of the present invention is used as a route learning processing section, and selects a state with a weighted probability. However, the probability is not used in the second state selecting means in the route movement processing unit after the route learning process is completed. In the action determining means of the route learning processing unit of the present invention, the moving speed and the traveling direction of the moving body are determined by the fuzzy arithmetic processing and executed. In addition, by changing the parameters of the membership function according to the reward, the route along which the moving body moves is strengthened. As a result, the movement of the moving body is smooth in the movement of the route after the route learning.

A typical example of the route learning method and apparatus of the present invention is an autonomous mobile robot, but it can also be applied to optimal problems such as a scheduling problem. The object of the present invention refers to an object to be operated, and is a moving body in this embodiment. Further, the moving body in the present embodiment means a moving body controlled by a programmable device such as a CPU, and the CPU or the like may be inside or outside the moving body. . In this embodiment, the moving body can move in all directions.

In this embodiment, the membership function has a conical shape, and its maximum height is always set to 1 with the same height (standard value). Although the present invention deals with a route learning device on a two-dimensional plane, the membership function is spherical, and the fitness is inversely proportional to the distance between the center of the sphere and the moving body (eg, 1 at the center, the surface of the sphere). Can be extended to the problem of route learning including three-dimensional obstacles such as underwater and outer space.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A route learning method for a moving body according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a basic processing flow chart of the route learning method of the present invention.
The route learning method of the present invention is the first initialization in which a membership function is arranged in each of the discretely distributed states.
Step 101, a second step 102 of selecting one of the states arranged in the vicinity when the target transits from the current state to the next state, and a state number of the state passed by the target. When it is determined that the third step 103 to be stored as history and the state number selected in the second step and the state number stored in the third step match, the state number history looped as a route loop And a fifth step 105 of determining and executing the action of the target by performing fuzzy arithmetic processing from the membership function arranged in each state and the current position of the target. A sixth step 106 of avoiding obstacles when the target transits to the next state
According to the action evaluation result of the seventh step 107 for evaluating the action of the target until the target reaches the goal point, the action is evaluated by the distance traveled from the start point to the goal point, and the action evaluation result of the seventh step. By repeating the eighth step 108 of distributing the reward to the parameters of all membership functions arranged in each passed state and the second to eighth steps in order, the target is optimized. It is composed of a ninth step 109 for converging learning so as to take action.

FIG. 2 is a block diagram of a route learning apparatus using the route learning method according to the first aspect of the present invention.
The device configuration has initialization processing means 301 for allocating a membership function to each discretely distributed state, and performs fuzzy arithmetic processing from the membership function and the current position of the target. Make a decision,
A route learning processing unit 201 that reinforces and learns a route by distributing rewards according to the action evaluation result, and a route movement processing unit 202 that moves the target according to the route reinforced based on the result of the route learning processing unit. It consists of

In the initialization processing means 301 of the route learning processing unit of the apparatus of the present invention shown in FIG. 2, a conical membership function is arranged in each state evenly distributed on the plane, and the parameters of the membership function are Perform initialization. Further, the starting point and the goal point are arranged in the state by this means. The route from the placed start point to the goal point is called an episode (a selection sequence of rules from reward to reward). In addition, a subgoal point may be provided on the way from the start point to the goal point. At that time, the route will be divided by the subgoal points. This divided route is also called an episode.

Starting point-subgoal 1, subgoal 1
~ Subgoal 2, ... Subgoal n ~ Each section of goal is learned in this order. That is, episode 1
Is a process of learning episode 2 after finishing learning. Also, in the route moving process after the route learning, the sub-goals are used to divide the episodes, as in the route learning processing unit. Furthermore, when the routes divided by the subgoals overlap, a separate membership function is used for each overlapping state. The route learning process must be performed once before the route moving process, but if the result of the route learning is stored in the storage medium, the re-learning process is not necessary unless the environment changes.

FIG. 3 is a block diagram of the route learning processing unit in each episode of the device of the present invention. The configuration is such that the initialization processing means 301 for arranging a membership function in each state and weighting one of the states arranged in the vicinity when the moving body transits from the current state to the next state. A first state selecting means 302 for selecting with probability, and a state history storing means 303 for storing the state passed by the target as a history of state numbers.
When it is determined that the state number selected by the first state selection means and the state number stored in the state history storage means match, a path for deleting the looped state number history as a path loop The action of the target is determined by performing fuzzy arithmetic processing from the loop deleting means 304, the membership function arranged in each state, and the current position of the target,
Action determining means 305 to be executed and obstacle avoiding means 30 for avoiding an obstacle when the target transits to the next state
6, the action evaluation means 307 that evaluates the action result of the target until the target reaches the goal point by the traveling distance from the start point to the goal point, and the reward according to the action evaluation result. Reward distribution means 308 that distributes to the parameters of all membership functions arranged in the state, and learning convergence means 309 that converges the learning so that the target takes an optimal action by sequentially and repeatedly processing the respective means. It consists of and.

FIG. 4 is a block diagram of the route movement processing section in each episode of the device of the present invention. The configuration is such that when the mobile moves from the current state to the next state,
Second state selecting means 401 for selecting one from the states arranged in the vicinity thereof, and a conical membership function arranged in each state of the plurality of states in which the moving body is currently located, A behavior determining unit 402 that determines and executes the behavior of the mobile body using fuzzy arithmetic processing in order to calculate the fitness and the traveling direction of the mobile body from
It comprises an obstacle avoiding means 403 for avoiding an obstacle when the target transits to the next state.

Next, the operation of the object (moving body) using the conical membership function arranged in each discretely distributed state will be described. FIG. 5 is a plane model diagram modeling a route search environment for learning a start point, a goal point, and a subgoal point on a plane where an obstacle exists. Although each component in the embodiments is mainly realized by software means in the computer system, it is not limited to this and may be hardware means.

The two-dimensional plane is cut into XY meshes of X × Y, and the mesh points at the intersections are used as the state points for reinforcement learning. In this embodiment, X = 10 and Y = 12. The initialization processing means arranges conical membership functions at equal intervals in each of the 120 states, and performs initialization processing for the membership functions. In addition, a start point, a goal point, and a subgoal point are selected from the 120 states. The state of the target start point is set in the lower left of the figure, and the state of the goal point of the target point is set slightly above the center of FIG. In addition, the point that requires a stop between the start and the goal is the subgoal point. Here, one subgoal is placed in the center right. In this way, the start point, goal point, and subgoal point can be freely selected from 120 state points. The route is divided by selecting a subgoal point.

FIG. 6 is a diagram for explaining the relationship between a moving body placed in a certain state on the plane and a state adjacent to the moving body. The moving body placed at a certain state point is adjacent to the eight state points around it. The movement of the moving body is performed by walking across the states while selecting one of the eight neighboring states adjacent to the currently staying state. In a specific autonomous mobile robot, the moving body has an obstacle sensor that can reach only the range of the eight surrounding states, and the moving body is autonomously moved by the sensor.

FIG. 7 is a diagram showing the conical shape of the membership function used in the present invention. The shape of the membership function arranged in each state has a three-dimensional shape in which the height is always represented by a conical shape having a prescribed value (1 in FIG. 7). The radius of the conical bottom surface (h in FIG. 7) is made variable, and the operation of the moving body is determined by controlling this value.
The diameter 2h is called "hedge width". This membership function is arranged in all the states (120 in this embodiment), and the hedge width 2h is represented by the minimum width in the initial state, and does not fall below this width even during learning. This membership function has two functions.

The first role is to define the shape of the membership function as a "weight" when one of a plurality of adjacent states is determined by the weighted probability when moving from one state to the next state during route learning. It is to use the parameter bottom surface diameter 2h. 8 (a) and 8 (b)
It is the figure which showed the state used as a moving object according to a moving direction, when a moving body selects a state. Actually, in order to speed up the learning convergence, the backward state selection is prohibited. Therefore, of the eight adjacent states, the five states indicated by the dotted lines in the front and left and right directions in the moving direction are to be selected. In other words, it can be considered that the moving body runs while swinging the modified dice in which each face of the five-sided dice is deformed by weight and the face becomes easier to appear as the weight is larger.

The second role is to give the moving body a velocity in the moving direction. The moving speed of the moving body located inside the conical membership function is determined by the length (z in FIG. 7) from the position to the intersection with the conical surface extending vertically upward. This length z is called the fitness. At the time of entering the next state from the current position of the moving body, the state to go to next is determined by rolling the modified 5-sided dice. The force to go to the next state will be represented by this goodness of fit z. When the conical surfaces of the conical membership functions arranged in the adjacent states overlap, fuzzy arithmetic processing is performed with the values of the respective fitness z to determine the moving direction. To put it simply, the membership function of the present invention has two symmetrical roles: "attractive force" by which the moving body determines the next state, and "repulsive force" by which the moving body that enters the state is pushed out. I will have. In addition, by performing the fuzzy calculation processing, the moving body automatically moves while drawing a smooth curvature.

In the present invention, the reward for reinforcement learning is reflected in the shape parameter of the membership function. That is,
When the moving body reaches the goal point (or sub-goal point, in the following description, the goal and sub-goal are collectively referred to as the goal point), the learning at that time is evaluated by the size of the "mileage", and the evaluation is performed. The reward according to is distributed to all the states of the route that went this time. When the action evaluation is high, the reward added to the shape parameter of the membership function of the state (here, the cone-shaped hedge width 2h) is increased, and when the action evaluation is low, it is decreased. Consider “negative reward” when behavioral assessment is extremely low. A moving body that transits between states with a weighted probability is likely to be attracted to a state with a large hedge width 2h, and thus a route with a high action evaluation gradually passes easily.

By this learning, the state hedge width 2h
By uniformly reducing the initial state in the initial state, a random walk is performed in the beginning, all routes are tried, and after some high-estimated routes are generated, the human thought to evaluate them at any time. A model close to is formed. Next, the operation of the route learning processing unit in each episode will be described. FIG. 9 is a process flow diagram for explaining the operation.

Step S1 The learning converging means determines the end of the route learning. In the determination of the end in the present embodiment, it is determined that the time when all the hedge widths of the conical membership function arranged in the state that constitutes the shortest path in the episode reaches the maximum specified value. The maximum prescribed value is the maximum diameter in a circle passing through the centers of eight adjacent states. However, the maximum specified value when the circle hits an obstacle is the diameter of the largest circle that does not hit the obstacle.

If it is determined that learning has ended, step S
If there is the next episode, the route learning process is repeated again, and if it is the last episode, the learning is ended. In step S3, it is determined whether or not the moving body has reached the goal point. If the determination in step S3 is affirmative, the process proceeds to subroutine SR1. Subroutine SR
In No. 1, when the moving body reaches the goal point, the behavior of the route that has passed is evaluated, and then the shape parameter of the membership function arranged in each state is rewarded.

FIG. 10 is a processing flow chart in the action evaluation means and the reward distribution means. In step S101, the reward to be given to the state is calculated according to the traveling distance of the route that has passed this time. The calculation uses, for example, the following calculation formula. Reward = (1.5-mileage this time / shortest mileage so far) x constant In this formula, the constant is a value determined by experiment. In general, the larger the value, the faster the convergence but the harder it is to find the shortest path,
The smaller the value, the opposite tendency.

In step S102, it is determined whether or not the current mileage is the smallest in this episode. If the determination is affirmative, the process proceeds to step S103, and the reward is distributed to some extent larger than the normal reward described above in order to strengthen the minimum path. In step S104, the numbers of all the states that have passed through as the shortest route are stored in a part of the state history storage means together with the total traveling distance.

Step S104 is completed, and step S1
It moves directly to step S106 without passing 05. In step S105, whether the reward is positive or negative is determined. If the reward is positive, the process proceeds to step S106, the reward is added to the hedge width of the membership function arranged in all the states passed this time, and this subroutine is exited. If the reward is negative, the process proceeds to step S107. Here, step S104
The hedge width is reduced by the reward amount for all states excluding the states that compose the shortest path stored in.

After the end of this subroutine, the process returns to step S1 in FIG. In step S4 of FIG. 9, it is determined whether or not the moving body is located in a new state range that is not stored in the state history storage means. The state range is the inside of the circle at the bottom of the conical membership function.
If the determination in step S4 is positive, the subroutine SR
Move to 2.

The subroutine SR2 in FIG. 11 is the operation of the first state selecting means. This subroutine SR
In step 2, a process for determining the next state to go to is performed. In the action selection of the moving body, one of the five states in the traveling direction is selected with a weighted probability. Further, not only the hedging widths of the five states are used as weights, but also the direction in which the goal is added may be added to the weights, so that the goal point may be discovered earlier. However, experiments have shown that it is better not to add weights when searching for the shortest route. Therefore,
It is necessary to properly handle the weights toward the goal point depending on whether to focus on the early detection of the route or to find the shortest route even if it takes time. Similarly, a method of adding the inertia of the moving body to the weight can be considered. In other words, this is a method of increasing the weight of the direction that is currently proceeding, but in an actual moving body, since there is always inertia, it is necessary to add an evaluation term of minimum energy to the shortest path. Therefore, it is rational to use the inertia of the moving body as a parameter for determining the next state.

In step S201, the current position of the moving body is confirmed. In step S202, the direction of the goal point is confirmed in order to add the direction of the goal point to the probability weight, as described above. In step S203, states near the front, diagonally forward, and left and right in five directions from the traveling direction of the moving body are selected as candidates for the next state.

In step S204, it is determined whether the five selected states are inside the obstacle. If the determination is positive, the process proceeds to step S205, and the state inside the obstacle is removed from the candidates. Step S2
At 06, a weighted probability calculation is performed to determine the next state. Here, an example of the weighted probability calculation will be described.

FIG. 12 is an explanatory diagram in which the mobile unit is located at the state number n in the figure and selects one from the five states. In the figure above, the hedge widths of the membership functions of the five states are 10, 30, 100, 40 and 20, respectively.
The case is shown. Further, the following figure is an explanatory diagram of weighted probability calculation. The RND function (the function that generates a real number from 0 to 1 as a random number is RN)
It can be seen from the figure that when the value obtained by multiplying by (D function) is 120, the state in which the mobile body should proceed is determined to be c. After the completion of this subroutine, the process proceeds to subroutine SR3 in FIG.

In the subroutine SR3, the loop determination of the route is performed. FIG. 13 is a process flow chart of the route loop deleting means. 14 (a) and 14 (b) are explanatory diagrams of the operation. FIG. 14A is a loop route diagram when it is determined that the state number next selected by the mobile unit matches the state number already learned and stored in the state history memory. FIG. 14B is a route diagram when the looped route is deleted.

In a system in which a route including a subgoal is learned from a start point to a goal point,
A mobile may pass through the same state again near the subgoal. However, in this embodiment, the learning of the next episode is not performed until the learning of each episode has converged, and therefore the state that has already passed is not returned. Therefore, deleting the looped route is effective.

In step S301, subroutine SR2
It is determined whether or not the next state determined in step 3 is a state that has already passed. When the determination is negative, this subroutine is ended. If the determination is affirmative, in step S302, the record of the state from the state up to the present state is deleted from the state history list stored in the state history storage means. After the end of this subroutine, the process proceeds to step S5 in FIG.

In step S5, the states determined for use in the evaluation after reaching the goal point and the determination of the deletion loop are added to the history list. After that, the process returns to step S3 in FIG. When the determination in step S4 of FIG. 9 is negative, the moving state of the moving body is normal, and the process proceeds to the subroutine SR4 action determining means for determining the direction of the moving body.

In step S401 of FIG. 15, it is determined whether or not the position of the moving body is currently inside some state (hedge). If the determination is negative, the process proceeds to step S402, and the direction of the moving body is the subroutine SR
Take the direction of the next state determined in 2. When the reward is not added to the parameter of the membership function arranged in the state of the route at the early stage of learning, the hedge width of the state is small and it becomes easy to shift to this step.

In step S403, the distance between the center of all the states in which the moving body is located and the moving body is calculated as a parameter used in the subsequent fuzzy calculation processing. This distance is the value of n in FIG. In step S404, the degree of compatibility with each state is calculated from the position of the moving body. In FIG. 7, it is the value of z. In step S405, the direction of the moving body is determined by fuzzy calculation processing.

FIG. 16 is a diagram for explaining the operation when the moving body makes an action decision by the action decision means. When the moving body enters the area of the state A at the point ae, the moving body selects the state B by the subroutine SR2 and moves toward the center of the state B. The fitness af toward B is calculated by the membership function of A and the position of the moving body up to the point be in the state B area. Further, when the moving body enters the region of state B at point be, it selects the next state C by the subroutine SR2 and moves toward the center of state C. Mobile is state C
Until entering the region of A, the membership function of A, the fitness af toward B according to the position of the moving body, and the fitness bf toward C from the membership function of B are calculated at any time. In this way, the behavior of the mobile body is determined by performing fuzzy arithmetic processing from the position of the mobile body and the membership function arranged in each state.

FIG. 17 is a diagram showing the traveling direction of the moving body as a vector component. In this way, the moving direction of the moving body is
Both goodness of fits are calculated using the normalized vector components. The weighted average is calculated by the following formula.

[0044]

[Equation 1]

This equation is the same as that generally used in fuzzy control. As described above, in the present embodiment, the state to which the moving body moves next is determined at the moment when the moving body enters the range of a certain state, and goes to the next state without passing through the center of the state. Therefore, the limitation of having only finite actions, which is a drawback of the conventional discrete reinforcement learning method, that is, the limitation that you must go to any one of the centers of discrete states, is eliminated, and the movement of the moving body A smooth curve will be drawn automatically without smooth control.

After the direction of the moving body is determined, this subroutine ends and the process proceeds to the next obstacle avoidance subroutine. During normal movement, the moving body keeps monitoring obstacles at all times. Subroutine SR5 in FIG. 9 is a process flow chart of the obstacle avoiding means. In step S501 of FIG. 18, it is determined whether or not the obstacle sensor of the moving body has detected an obstacle in the traveling direction. If the determination is negative, this subroutine ends. If the determination is positive, the obstacle avoidance algorithm is activated. A step S502 measures a rough distance between the left and right ends of the obstacle by the sensor of the moving body. 19 (a) and 19 (b) are operation explanatory views of the obstacle avoiding means of the moving body. In the figure, the obstacles are indicated by shaded areas.

In FIG. 19A, a circle centering on the moving body is a range that can be measured by the sensor, Lr is a distance between the right end of the obstacle and the moving body, and Ll is a distance between the left end of the obstacle and the moving body. .
In step S503, Lr and Ll are compared and the direction of the moving body is changed to the closer one. Further, in FIG. 19B, the left end of the obstacle is in the range measurable by the sensor, but the right end is out of the range, so the left end is selected as the direction. If both ends are out of the range, the end closest to the state to be advanced is selected.

Thus, the obstacle avoidance algorithm is finished,
Move to the next step. In step S6 of FIG. 9, a predetermined distance is moved based on the direction determined by the subroutine SR4. The setting of the distance should be decided by experimentation. After that, the process returns to step S4. As described above, the mobile object receives a reward (may be negative) each time it reaches the goal point while randomly walking, and strengthens the shortest path.

Next, the route movement processing unit in which the moving body moves after the route learning is completed will be described. FIG. 4 is a block diagram of the route movement processing unit after the route learning is completed. In addition, FIG. 20 is a process flow diagram of the route movement in each episode in which the moving body moves in the route movement processing unit. In this block diagram 4, many steps and subroutines are common to the operation of the route learning processing unit. For example, the action determining unit 402 and the obstacle avoiding unit 403 have the same operation as the action determining unit 305, the obstacle avoiding unit 306, and the like of the route learning processing unit. Therefore, only the different points will be described here.

In step S7 of FIG. 20, it is determined whether or not the current position of the moving body is in the goal point state.
When the determination is affirmative, the process proceeds to step S2. The second state selecting means of step S8 is different from the route learning subroutine SR2 first state selecting means.
Probability is not used to determine the next selected state. FIG.
This state selection will be described with reference to (a). The target states are the five states shown within the dotted line. This 5
Of the two states, the state with the largest hedge width of the conical membership function is selected. Therefore, n + 1 states are selected in the figure.

FIG. 21 is a simulation diagram while the moving body is learning the route in the first episode on the plane model in which the obstacle exists. FIG. 22 is a simulation diagram showing a route in which the moving body is learning-reinforced from the start point to the goal point as a result of the learning convergence. Although not shown in the figure, the states may overlap between different episodes in the vicinity of the subgoal. In such a case, the movement of the moving body is hindered, and it is necessary to give different membership functions to the same state for each episode.

[0052]

As described in detail above, according to the invention of claim 1, the membership function is arranged in the state in the reinforcement learning, and the reward in the reinforcement learning is reflected in the size of the membership function to perform the learning. Can be converged. In addition, by performing fuzzy processing using the membership function placed in the state, the state position is fixed, the possible actions remain finite, and learning close to continuous input / output variables can be realized, making it suitable for practical use. A compact system can be assembled.

Further, in a moving body that moves on a plane where an obstacle exists, the user can learn the route by himself by unconditionally arranging finite states at equal intervals on the plane. Became. Further, in the conventional movement of the moving body in the reinforcement learning, the moving body is a so-called zigzag traveling that walks across discrete and fixed states, whereas in the present invention, the state expands the range and adjacent states and fuzzy Smooth running was achieved by performing arithmetic processing.

Further, since the size of the membership function itself is set as the range of states, the result after learning convergence is only the position information of the states and the range information of the states, so the storage area of the device can be small. There is an effect.

[Brief description of drawings]

FIG. 1 is a processing flow chart of a route learning method of the present invention.

FIG. 2 is a block diagram of a route learning device of the present invention.

FIG. 3 is a block diagram of each episode in the route learning processing unit of the present invention.

FIG. 4 is a block diagram of each episode in the route processing unit of the present invention.

FIG. 5 is a plane model diagram for learning a route in which an obstacle exists.

FIG. 6 is a diagram showing a range in which an obstacle sensor of a moving body can detect.

FIG. 7 is an explanatory diagram of a conical membership function.

FIG. 8 is a diagram showing a target state when a moving body selects a state.

FIG. 9 is a processing flowchart of each episode in the route learning processing unit of the present invention.

FIG. 10 is a process flow chart of the behavior evaluation unit and the reward distribution unit of the route learning processing unit of the present invention.

FIG. 11 is a processing flow chart of the first state selecting means of the route learning processing unit of the present invention.

FIG. 12 shows an example of a weighted probability calculation by the first state selection means.

FIG. 13 is a processing flow chart of the route loop deleting means of the route learning processing unit of the present invention.

FIG. 14 is an explanatory diagram of the operation of the route loop deleting means.

FIG. 15 is a processing flow chart of the action determining means of the route learning processing unit of the present invention.

FIG. 16 is an operation explanatory view of the action determining means.

FIG. 17 is an explanatory diagram of a moving direction vector of a moving body in the action determining means.

FIG. 18 is a processing flowchart of the obstacle avoiding means of the route learning processing unit of the present invention.

FIG. 19 is an operation explanatory view of the obstacle avoiding means.

FIG. 20 is a processing flowchart of a route movement processing unit of the route learning device of the present invention.

FIG. 21 is a simulation diagram in the middle of a route learning of a moving body using the route learning device of the present invention.

FIG. 22 is a simulation diagram showing a result of route learning performed by a moving body and route movement using the route learning device of the present invention.

Claims (9)

[Claims] 1. A path learning method in which an object moves from a start point to a goal point, a first step of allocating a membership function to each discretely distributed state, and the object is a current state. A second step of selecting one of the states arranged in the vicinity when transitioning from the next state to the next state, a third step of storing the state passed by the target as a history of state numbers, When it is determined that the state number selected in the two steps and the state number stored in the third step match, the state number history looped as a route loop is deleted, and the state number is arranged in each state. By performing fuzzy arithmetic processing from the membership function determined and the current position of the target, the behavior of the target A fifth step of determining and executing, a sixth step of avoiding an obstacle when the target transits to the next state, and a behavior result of the target until the target reaches a goal point from a start point A seventh step of behavior evaluation evaluated by the distance traveled to the goal point, and a reward of distributing the reward according to the behavior evaluation result of the seventh step to the parameters of all membership functions arranged in each passing state An eighth step of distributing and a ninth step of converging learning so that the object takes an optimal action by sequentially repeating the second step to the eighth step are provided and the operation is performed. A route learning method characterized by the above. 2. In a route learning device in which an object moves from a start point to a goal point, initialization processing means for initializing a membership function in each of the discretely distributed states, and the membership function. A route learning processing unit that determines an action while performing fuzzy arithmetic processing from the current position of the target and distributes rewards according to the action evaluation result, and a route learning processing unit that performs reinforcement learning of a route, and is strengthened based on the result of the route learning processing unit. And a route movement processing unit for moving the target according to the route. 3. The route learning processing unit, when the target transits from a current state to a next state, first state selecting means for selecting one from among states arranged in the vicinity thereof, The state history storage means for storing the state passed by the object as a history of state numbers, the state number selected by the first state selection means, and the state number stored in the state history storage means match. When it is determined that the object is a path loop, a route loop deleting means for deleting the looped state number history, a membership function arranged in each of the states and a current position of the object are subjected to fuzzy arithmetic processing to obtain the object. Behavior decision means for deciding and executing the action, and obstacles for avoiding obstacles when the target transits to the next state. Passing a harmful substance avoidance means, an action evaluation means for evaluating the action result of the object until the object reaches the goal point by the traveling distance from the start point to the goal point, and a reward according to the action evaluation result, Reward distribution means for distributing the parameters of all membership functions arranged in the respective states, and learning convergence means for converging the learning so that the object takes an optimal action by sequentially and repeatedly processing the respective means. The route learning device according to claim 2, further comprising: 4. The route movement processing unit, when the target transits from a current state to a next state, second state selecting means for selecting one from among states arranged in the vicinity thereof, Action determining means for determining and executing the action of the target by performing fuzzy arithmetic processing from the membership function arranged in each state and the current position of the target, and when the target transits to the next state 3. The route learning device according to claim 2, further comprising: obstacle avoiding means for avoiding an obstacle. 5. The route learning device according to claim 2, wherein the shape of the membership function in the initialization processing means is a conical shape. 6. The route learning device according to claim 2, wherein the first state selecting means selects a state to move to next using a weighted probability. 7. A weighted probability with respect to a parameter of a membership function placed in each state of the goal point or the moving direction of the object in the one state selected by the first state selecting means. 7. The route learning device according to claim 6, wherein the weights are added. 8. The route learning device according to claim 2, wherein the obstacle avoiding means uses the closer one of the left and right ends of the obstacle as the direction of the target for avoiding the obstacle. 9. The route learning device according to claim 2, wherein the learning convergence unit converges the learning when all the passed states reach the maximum specified value.