JP2013205890A - Machine learning system and machine learning method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To adaptively select a parametrically-expressed state space and a non-parametrically-expressed state space.SOLUTION: A machine learning system (1) comprises: knowledge acquisition means (12) for generating a parametrically-expressed class set through reinforcement learning on the basis of a reward or penalty given to input and output to the input; knowledge reconstitution means (14) for generating a non-parametrically-expressed class set, on the basis of the learned input used for the generation of the parametrically-expressed class set; and knowledge utilization means (16) for performing class determination of determining to which of the non-parametrically-expressed classes unknown input belongs, and performing output corresponding to a result of the determination. The knowledge reconstitution means (14) generates the non-parametrically-expressed class set when the number of learned inputs is greater than a predetermined number and a variance of each of the parametrically-expressed classes is less than a predetermined value.

Description

  The present invention relates to machine learning, and more particularly to improving the robustness of knowledge acquisition by reinforcement learning.

  When controlling a system, a top-down approach based on modeling is generally taken. However, it may be difficult to control due to factors such as an increase in the scale of the system. On the other hand, in the bottom-up approach, it is possible to acquire purposeful input / output relationships as a whole system by making the system components intelligent. One of them is the reinforcement learning method. Reinforcement learning methods are expected to be applied to various systems because of the ease of implementation in which an input / output sequence leading to a reinforcement state can be established autonomously simply by giving a target state.

  The conventional framework of reinforcement learning is aimed at constructing mapping relationships in discrete state / action spaces. Here, the learning performance is greatly influenced by the discretization of the state / action space, but there is no design guideline for that purpose at present. This problem is a serious problem in many real systems operating in continuous space. The inventors of the present application have studied and developed Bayesian-discrimination-function-based Reinforcement Learning (BRL), which is a function expansion of reinforcement learning, as a method for designing the state / action space. BRL has a function of autonomously dividing a continuous state / action space. Furthermore, conventional reinforcement learning only guarantees learning convergence in a Markov environment, but BRL can be updated in a dynamic environment because it can adaptively update the degree of division in the learning process. have. Up to now, the present inventors have taken up a robot as a real system, in particular, a multi-robot system (MRS) composed of a plurality of robots, and in a cooperative problem by an autonomous mobile robot group or an arm type robot group, BRL Has been shown to be effective.

  However, in subsequent additional experiments, it was observed that the robustness of the BRL may be gradually lost if further learning is continued after acquiring the action. This is because rules that do not contribute to task achievement are deleted, and only the contributing rules are strengthened and remain in the rule set. That is, in the BRL, a rule set specialized for the environment results in an overlearning state, and the system becomes unstable. Therefore, in recent years, the inventors of the present application have focused on the high discrimination performance of Support Vector Machine (SVM), which is one of the pattern recognition methods, and clarified that rule discrimination by SVM is effective in suppressing overlearning of BRL. (For example, refer nonpatent literature 1).

J. Sakanoue, T. Yasuda, and K. Ohkura, "Preservation and Application of Acquired Knowledge Using Instance-Based Reinforcement Learning," Joint 5th International Conference on Soft Computing and Intelligent Systems and 10th International Symposium on advanced Intelligent Systems, 2010, pp .576-581

  Although it has been proved that SVM is effective in suppressing BRL overlearning, it has not yet been proposed how to use SVM specifically for BRL rule discrimination. In view of such a problem, an object of the present invention is to provide a method for adaptively selecting a state space expressed in a parametric manner and a state space expressed in a non-parametric manner in a machine learning system.

  The machine learning system according to one aspect of the present invention performs class discrimination as to which class in the state space the input belongs to, outputs according to the discrimination result, and repeats input / output to acquire knowledge adapted to the environment. A machine learning system for acquiring knowledge acquisition means for performing reinforcement learning based on an input and a reward or punishment given to an output for the input to generate a set of parametrically expressed classes, and the parametrically expressed Based on the learned input used to generate the class set, knowledge reconstructing means for generating a non-parametrically expressed class set, and class discrimination for which class the unknown input belongs to Knowledge utilization means for performing and outputting according to the determination result. The knowledge reconstructing means generates the non-parametrically represented class set when the number of learned inputs is larger than a predetermined number and the variance of each class represented by the parametric expression is smaller than a predetermined value. .

  According to this, when the reinforcement learning by the knowledge acquisition means has sufficiently progressed, the class set expressed by the parametric expression generated by the knowledge acquisition means is reconstructed into the class set expressed by the parametric expression by the knowledge reconstruction means, and A class determination of unknown input is performed by a knowledge using means using a class set expressed in parametric form.

  Specifically, each class represented by the parametric expression is a multivariate normal probability distribution, and the knowledge acquisition means performs class discrimination according to a Bayes discrimination method to which class the input belongs to the parametric expression.

  Further, specifically, the knowledge reconstructing means linearly separates the learned input using SVM to generate the non-parametric expressed class set.

  According to the present invention, in a machine learning system, a state space expressed in a parametric manner and a state space expressed in a non-parametric manner are adaptively selected, and the robustness of the machine learning system is improved. As a result, the machine learning system can flexibly cope with environmental changes.

Functional block diagram of a machine learning system according to an embodiment of the present invention Flowchart of knowledge acquisition and use by the machine learning system of FIG. Schematic diagram showing the experimental environment for computer experiments Graph showing task achievement rate by each SVM Graph showing the number of episodes required to acquire knowledge

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In addition, this invention is not limited to the following embodiment.

(Embodiment of machine learning system)
FIG. 1 is a functional block diagram of a machine learning system according to an embodiment of the present invention. The machine learning system 1 according to the present embodiment performs class discrimination as to which class in the state space the input belongs to, outputs according to the discrimination result, and acquires knowledge adapted to the environment 100 by repeating input / output To do. The machine learning system 1 can be applied to each robot constituting the MRS, for example.

  The machine learning system 1 includes a knowledge acquisition unit 12, a knowledge reconstruction unit 14, and a knowledge use unit 16. Each of these means may be realized as hardware such as an electronic device, or may be realized as a software module executed by a computer. Hereinafter, each means will be described in detail.

(Detailed explanation of the knowledge acquisition means 12)
The knowledge acquisition unit 12 performs reinforcement learning based on a reward or punishment given to an input from the environment 100 and an output corresponding to the input, and generates a class set expressed in a parametric manner. For example, the knowledge acquisition unit 12 performs reinforcement learning by BRL. In other words, each parametric-expressed class is a multivariate normal probability distribution, and the knowledge acquisition means 12 performs class discrimination according to the Bayes discrimination method to which class the input belongs to the parametric representation.

≪BRL≫
In BRL, a Bayes discriminant that statistically classifies patterns is used to identify whether the input x is classified into k-th class C _k (where k is an integer from 1 to N). In the Bayes discriminant method, the input x is observed when the class C to be identified is C = {C _k } ^K _{k = 1,} and the prior probability P (C _k ) and probability distribution p (x | C _k ) of each class are known. In this method, the posterior probability P (C _k | x) of each class C _k is obtained from the Bayes formula, and the input is identified to the class having the maximum posterior probability (see Expression (1)). In BRL, the environment probability model is updated from observation data in real time by (1) addition and deletion of classes, and (2) parameter update of the probability distribution model, and the state space is divided.

≪Rule structure≫
Each class is represented by a Gaussian distribution, and parameters representing the probability distribution of each class and the output at that time are stored in the knowledge acquisition means 12 as rules described in an if-then format. Hereinafter, class and rule are treated as synonymous. The rule set R is composed of rules rlεR, and each rule is described by the following expression.

Each rule rl includes a feature vector v = {v ₁ ,..., V _nd } ^T , a covariance matrix Σ, a class prior probability f, a validity u representing class reliability, and a set of sensor inputs observed in each class Φ = {φ ₁ ,..., Φ _ns } ^T and operation a = {a ₁ ,..., A _na } ^T. Here, n _d is the number of dimensions in the input space, n _a is the number of dimensions in the output space, and n _s is sample data stored in each class. In the initial stage of learning, no class exists in the state space. A class is added to the state space based on the input / output actually observed by the machine learning system 1, and the state space is covered with a Gaussian distribution.

≪Operation selection≫
The outline of BRL action selection is shown below.

  -Determine which class the perceived input belongs to by the Bayes discrimination method.

  ・ If it does not belong to an existing rule, it outputs a random action and creates a new rule if there is no penalty.

  -If it belongs to an existing rule, the rule action is output.

The posterior probability of each rule for the input is obtained from the Bayes formula, and the output described in the rule with the maximum posterior probability is executed. Here, first, the negative logarithm of the posterior probability is taken, and the rule that minimizes the probability g _{i of} erroneous identification is defined as the winner rule rl _w .

At this time, it is not appropriate to select a rule having a very small posterior probability, and a threshold value _Pth is set for the posterior probability. Then, it is determined whether or not to execute the operation of rl _w based on the threshold value g _th = −log {f ₀ · P _th } calculated based on the threshold value. _{Specifically,} in the case of g w _{<g th,} it performs the operations _{A w} of rl _w. When g _w ≧ g _th , the operation is executed at random. Note that f ₀ and P _th are constants.

≪Update of effectiveness≫
Profit Sharing and Bucket Brigade strategies are used to propagate rewards retroactively. In addition, there is dissipation that acts on all the rules when the task is completed in order to reduce the cost imposed on the rule selected to prevent the loop action and the rule that does not contribute to the reward acquisition to reduce the memory consumption.

≪Parameter update≫
Each rule updates the probability distribution parameters online based on the input. Updating in real time can be expected to respond quickly to environmental and system changes. On the other hand, some measures are necessary because it is easily affected by noise and temporary bias of input. In BRL, this problem is solved by updating parameters using the interval estimation method. The interval estimation method is a method for guaranteeing that the probability of the probability distribution parameter entering a certain interval is equal to or higher than the set probability, and the estimation accuracy increases as the sample data increases. Therefore, more reliable parameter estimation can be expected as the observation data increases.

Where v is the average of rl _w , σ ² is the variance of rl _w , α and β are constants in the interval [0, 1], j is each dimension in the input vector, x bar is the average of the sample input, and s ² is Variance of sample input, P is reward.

≪Adaptive search of action space based on parameters of existing rules≫
From the viewpoint that it is inefficient to always search the behavior space at random, in the situation where knowledge has been acquired to some extent, search the neighborhood of existing rules and adjust the behavior rather than searching the behavior space widely Is considered effective. Therefore, in addition to the threshold value P _th for performing a random search, a new threshold value P ′ _th is set (P ′ _th <P _th ). If g _th ≦ g _w <g ′ _th , the parameters of the rules in the meantime To determine new rule parameters. That is, the action selection is changed as follows.

In the case of · _g w _{<g th,} to perform the operations _{A w} of rl _w.

When g _th ≦ g _w <g ′ _th , an action is generated based on the rules in between.

When g _w ≧ g ′ _th , the operation is executed at random.

≪g _th ≦ g _w <g ' _th action≫
This range may include a plurality of rules other than rl _w . Although these rules do not have a large difference in selection probability in the situation, their effectiveness differs according to the degree of contribution to task achievement in the learning process so far. Therefore, the action A ′ of the new rule is obtained by a weighted average based on the effectiveness of the rules included in this range.

n _r is the number of rules included in this range, and N (0, σ) is noise using a normal distribution with mean 0 and standard deviation σ. By adding noise, even if there is no rule other than rl _w , the vicinity of A _w can be searched.

(Detailed description of the knowledge reconstructing means 14 and the knowledge using means 16)
The knowledge reconstructing unit 14 generates a non-parametrically represented class set based on the learned input used by the knowledge acquiring unit 12 to generate the parametrically represented class set. For example, the knowledge reconstructing unit 14 linearly separates the learned input in the knowledge acquiring unit 12 using SVM, and generates a class set expressed in a non-parametric manner. The knowledge utilization means 16 performs class discrimination as to which class the unknown input from the environment 100 belongs to in a non-parametric expression, and outputs according to the discrimination result.

≪Knowledge reconstruction and use by SVM≫
SVM has been shown in various fields to have high discrimination performance. Therefore, it can be expected that more accurate action determination can be achieved by identifying BRL acquired knowledge data by SVM.

  The inventors of the present application have shown in the past that it is not effective to use knowledge by SVM in a state where sample rules are insufficient because a rule effective for accomplishing a task cannot be obtained by BRL (see Non-Patent Document 1). . Therefore, an index is provided to reconstruct the acquired knowledge by operating the knowledge reconstructing means 14 at the timing when the learning in the knowledge acquiring means 12 has sufficiently progressed and the behavior of the machine learning system 1 has started to stabilize.

  First, as a premise for using knowledge by the SVM, the sample data needs to be sufficiently large, that is, the number of learned inputs used for generating the class set expressed parametrically in the knowledge acquisition means 12 needs to be sufficiently large. Therefore, the fact that the number of learned inputs is greater than a predetermined number is set as an index.

Further, even if the number of sample data is sufficiently large, if there is a class having a large variance among the classes represented by parametric expression, there is a possibility that sufficient identification accuracy by SVM cannot be obtained. Therefore, the indicator that the variance of each class (rule) expressed parametrically is smaller than a predetermined value is also set as an index. For example, this index is set by a covariance matrix Σ that is a component of each rule. The value of Σ represents the range of the rule in the state space. When the behavior starts to converge, the value of Σ converges. Therefore, for example, the knowledge reconstructing unit 14 calculates the average (Σ _eps ) of | Σ | of each rule for each episode, and the difference from the average (Σ _eps−1 ) of | Σ | In the case of (4), knowledge reconstruction by SVM is performed to generate a class set expressed in a non-parametric manner. Then, the knowledge using means 14 performs class discrimination, that is, rule selection by SVM, using the non-parametrically expressed class set.

In addition, a new threshold value P _s (P _s > P _th ) is set in order to make the range in which the rule can be determined wider than in the case of BRL. Widening the threshold is expected to suppress the creation of new rules and prevent behavioral instability. Below, the outline | summary of the action selection of the machine learning system 1 is shown. FIG. 2 shows an outline of knowledge acquisition and use by the machine learning system 1.

Based on the perceived input, whether to perform knowledge search by BRL or to use knowledge by SVM is determined by comparing the value of Σ ' _eps with a threshold value.

  -When performing a knowledge search by BRL, a random action is output and a new rule is created if there is no penalty.

  When knowledge is used by SVM, the state space is subdivided by SVM, and the action of the rule determined by the subdivided state space is output.

Note that Equation (9) is an example of an index, and the present invention is not limited to this. For example, a moving average or a weighted average of Σ ′ _eps may be used.

≪Multi-class classification SVMs≫
SVM is basically formulated for two classes of identification problems. However, multi-class classification is possible by combining two classes of discrimination models. Two types of combinations, One-versus-All and One-versus-One, are taken up. One-versus-All is a method of creating an identification plane that divides into one class and other classes for all classes, and outputting the class that returns the highest discrimination value among these identification planes. is there. For n-class problems, the number of identification planes is n. On the other hand, One-versus-One is a method of creating a paired identification plane for each class and determining the output by majority vote. The number of identification planes is n (n-1) / 2. These two types of multi-class classification methods are introduced into the SVM used for the knowledge utilization means 14.

≪Kernel function used for SVM≫
The SVM used in the knowledge reconstruction unit 14 uses a kernel trick that enables nonlinear separation by mapping to a high-dimensional space. As the kernel, the following linear kernel K _line , polynomial kernel K _poly , RBF kernel K _RBF , and sigmoid kernel K _sig are used. u represents a support vector, and v represents a feature vector to be identified.

(Computer experiment)
As a task of knowledge acquisition by BRL, we take up the problem of carrying a piano by two robots and conduct computer experiments. The piano carrying problem is a problem in which a robot cooperates to convey a long object that cannot be conveyed alone to the goal. In order to pass through a narrow passage, two robots must cooperate to form a formation.

  The experimental environment is shown in FIG. The field is surrounded by walls on all sides, and the task is accomplished when moving from the initial position to the goal line. The robot uses a differential drive type and has two drive wheels. Each robot perceives the state of other robots with an omnidirectional camera and does not communicate between the robots. One decision of the robot is taken as one step, and when the task is completed or 400 steps have passed, the episode is updated and the robot is returned to the initial position. The definition of learning success is when the task is achieved for 20 consecutive episodes. One trial is made until 500 episodes have elapsed, and 100 trials are repeated. The simulation environment is created by an open source three-dimensional physics engine ODE (Open Dynamic Engine).

≪Experiment 1≫
Learning is performed using only BRL. For trials in which learning has succeeded, the environment changes as shown in FIG. 3B are performed, and action selection is performed only by the SVM in order to verify the effect of knowledge utilization by the SVM. The environmental change is that the width of the passage is narrowed, and the difficulty level of the task increases. Therefore, it is required to accurately determine the acquired knowledge according to the state. The multi-class classification SVMs used here uses eight patterns of a combination of two types of multi-class classification methods and four types of kernel functions. Each combination is represented by symbols A to H shown in Table 1. During successful learning trials, observe the success rate of trials that were able to accomplish the task of environmental change.

≪Experiment 2≫
The machine learning system 1 is used to learn the combination patterns that have a high success rate in Experiment 1. The transition of the number of episodes required for learning is examined, and the effectiveness of the machine learning system 1 for action acquisition is verified. By parameter tuning, the threshold of Σ ' _eps using SVM was defined as 0.02.

≪Setting of machine learning system 1≫
The inputs are eleven dimensions of I = {r ₀ , cos θ ₀ , sin θ ₀ , r ₁ , cos θ ₁ , sin θ ₁ , r ₂ , cos θ ₂ , sin θ ₂ , cos θ ₃ , sin θ ₃ }. r and θ indicate the distance to the object and its angle, subscripts 0, 1 and 2 indicate the goal line, the nearest wall, the second adjacent wall, and subscript 3 indicate the adjacent robot. The output of the BRL is two-dimensional with the left and right motor rotation speeds O = {m ₀ , m ₁ }. The SVM was set using an improved library LIBSVM (see http://www.csie.ntu.edu.tw/~cjlin/libsvm/index.html), and each parameter was defined by tuning.

≪Experimental result 1≫
Learning was successful in 76 trials out of all 100 trials. FIG. 4 shows the success rate of trials in which the SVM from A to H was selected for each of the 76 trials and the tasks during the 76 trials were achieved. The BRL in FIG. 4 is a case where the behavior is determined by the Bayes discriminant method without knowledge of random behavior and knowledge only when learning is successful. In the cases of C, D, E, and F, the task success rates were high, which were 78%, 71%, 78%, and 71%, respectively. Since the task achievement rate of the environmental change is 51% with only BRL, it can be seen that the discrimination performance is high when the kernel function uses K _poly and K _RBF . Compared to BRL, it can be said that the discrimination performance of SVM is relatively high because the task success rate is high in all cases A to H using SVM. Moreover, when comparing One-versus-All and One-versus-One in the case of K _poly and _KRBF , One-versus-All has a higher success rate. It is generally said that One-versus-All is effective when the number of classes is as large as 10,000, but there is no difference in effectiveness in a BRL having a maximum number of classes (number of rules) of 100.

≪Experiment result 2≫
From the results of Experiment 1, in Experiment 2, learning was performed using the machine learning system 1 for the case of four patterns of C, D, E, and F. As a result of 100 trials, C, D, E, and F succeeded in learning in 78, 79, 75, and 79 trials, respectively. It seems that there is no big difference compared with 76 trials with BRL only. FIG. 5 shows the average number of episodes required for successful learning and the standard deviation. When this result was subjected to a T test, it was shown that there was a difference at a significance level of 1% in the case of D and F as compared with BRL alone. That is, in the case of D or F, it was shown that the convergence speed required for learning is fast.

  Judging comprehensively from the results of Experiments 1 and 2, it can be said that the use of the SVM of the D or F pattern is effective for action acquisition and the identification accuracy is high.

  Since the machine learning system and the machine learning method according to the present invention are excellent in robustness and can cope with environmental changes, they are useful for multi-robot systems and the like. Moreover, it is useful not only for robots but also for reinforcement learning in pattern recognition.

DESCRIPTION OF SYMBOLS 1 Machine learning system 12 Knowledge acquisition means 14 Knowledge reconstruction means 16 Knowledge utilization means 100 Environment

Claims (6)

It is a machine learning system that performs class discrimination as to which class in the state space belongs, outputs according to the discrimination result, and acquires knowledge adapted to the environment by repeating input and output,
A knowledge acquisition means for performing reinforcement learning based on an input and a reward or punishment given to an output for the input to generate a class set represented by parametric expression;
Knowledge reconstructing means for generating a non-parametrically represented class set based on the learned input used to generate the parametrically represented class set;
A knowledge using means for performing class discrimination as to which class the unknown input belongs to which is represented in the non-parametric expression and outputting according to the discrimination result;
The knowledge reconstructing means generates the non-parametrically represented class set when the number of learned inputs is larger than a predetermined number and the variance of each class represented by the parametric expression is smaller than a predetermined value. A machine learning system characterized by that. The machine learning system according to claim 1,
Each parametric expressed class is a multivariate normal probability distribution;
The machine learning system according to claim 1, wherein the knowledge acquisition unit performs class discrimination according to a Bayes discrimination method to which class the input belongs to the parametric expression. The machine learning system according to any one of claims 1 and 2,
The knowledge reconstructing means linearly separates the learned input using a support vector machine to generate the class set represented by the non-parametric expression. A machine learning method for classifying which class an input belongs to in a state space, performing output according to the result of the determination, and acquiring knowledge adapted to the environment by repeating input and output,
A first step of performing reinforcement learning based on an input and a reward or punishment given to an output for the input to generate a parametric expressed class set;
Generating a non-parametrically represented class set based on the learned input used to generate the parametrically represented class set;
A third step of performing a class determination as to which class the unknown input belongs to in the non-parametric representation and outputting according to the determination result;
In the second step, when the number of learned inputs is greater than a predetermined number and the variance of each class represented by the parametric expression is smaller than a predetermined value, the class set represented by the non-parametric expression is generated. A machine learning method characterized by that. The machine learning method according to claim 4,
The parametric expressed class is a multivariate normal probability distribution;
In the first step, a class discrimination is performed as to which class the input belongs to in the parametric expression according to a Bayes discrimination method. The machine learning method according to any one of claims 4 and 5,
In the second step, the learned input is linearly separated using a support vector machine, and the non-parametrically expressed class set is generated.