JP2007041723A - Sensor designing device, sensor designing method, sensor designing program and robot - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a sensor designing device which can automatically design form of a sensor being advantageous to a robot's action learning. <P>SOLUTION: An initial generation creation part 2 creates a plurality of genotypes for specifying the form of the sensor and virtually creates the plurality of robots having the form of the sensor specified according to each genotype. An action learning part 3 makes the plurality of robots to virtually learn and calculates adaptability of each robot based on the learning result. A selection part 4 selects the plurality of robots which serves as parent individuals based on the adaptability of each robot. A next-generation creation part 5 creates a next-generation genotype based on a genetic algorithm from the genotype of selected robot and virtually creates the plurality of robots having the form of the sensor specified by each next-generation genotype. The action learning part 3 makes the plurality of next-generation robots to virtually relearn. Processes by the selection part 4, the next-generation creation part 5, and the action learning part 3 are repeated by predetermined generations and the form of the sensor is determined. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a sensor design device, a sensor design method and a sensor design program for designing the form of a sensor used for robot behavior learning, and a robot having a sensor designed by the sensor design device.

In order to construct a robot that is adaptive to changes in the environment, it is necessary to consider the balance of the robot's form, control system, and environment. In recent years, attention has been focused on research for acquiring adaptive behavior in robots by combining forms and control systems. For example, in Non-Patent Document 1, a form obtained on software is realized as hardware by combining a method of evolving an agent form and a control system on software and a rapid molding technique.
Lipson H. et al., “Automatic design and manufacture of robotic lifeforams”, Nature, 2000, Vol. 406, no. 6799, p. 974-p. 978

  However, with the above-described method, it is not possible to automatically design a sensor configuration that is advantageous for adaptation in ontogenic spans, that is, for learning.

  An object of the present invention is to provide a sensor design device, a sensor design method, a sensor design program, and a robot capable of automatically designing a sensor form advantageous for robot behavior learning.

  The sensor design device according to the present invention is a sensor design device for designing the form of a sensor used for robot behavior learning, and creates a plurality of genotypes for specifying the form of the sensor. An initial generation creation means for virtually creating a plurality of robots having a specified sensor form, and a plurality of robots created by the initial generation creation means are virtually trained, and each robot based on the learning result The learning means for calculating the fitness of each robot, and a plurality of parent robots are selected based on the fitness of each robot calculated by the learning means. Next-generation creation means to create a genotype of the next generation and virtually create a plurality of robots having a sensor form specified by each next-generation genotype, and learning The stage allows the robot created by the next generation creation means to virtually perform learning again, calculates the fitness of each robot based on the learning result, and repeats the processing by the next generation creation means and the learning means a predetermined number of times. This determines the form of the sensor.

  In the sensor design device according to the present invention, a plurality of genotypes for specifying the sensor form are created, and a plurality of robots having the sensor form specified by each genotype are virtually created and created. Learning is performed by a plurality of robots, and the fitness of each robot is calculated based on the learning result of each learned robot.

  Next, based on the calculated fitness of each robot, a plurality of parent robots are selected, and a next generation genotype is created based on the genetic algorithm from the selected genotypes of the plurality of robots. A plurality of robots having the form of sensors specified by the next-generation genotype are virtually created, and the created robots are made to learn again, and the fitness of each robot is calculated based on the learning results. The form of the sensor is determined by repeating the next generation creation process and the learning process thereof a predetermined number of times.

  Therefore, based on the learning result, a robot with high fitness, that is, a robot with high learning performance, is selected as a parent individual robot without dying, and the genotype that identifies the form of the sensor can be evolved, It is possible to automatically design a sensor form that is advantageous for robot behavior learning.

  The learning means preferably causes a plurality of robots to perform learning by Q learning. In this case, a robot having various genotypes, that is, various sensor forms can be efficiently learned, and a sensor having a suitable form can be designed at high speed.

  Based on the fitness of each robot calculated by the learning means, the next generation creation means selects a predetermined number of robots with high learning performance as a parent individual from a plurality of robots, and selects tournaments from the remaining robots. A selection unit that selects the same number of robots as a parent individual, and a next generation genotype based on the genetic algorithm from the genotype of the parent individual selected by the selection unit, and a sensor that is identified by each next generation genotype Creating means for virtually creating a plurality of robots having the following forms as next-generation robots, and the learning means virtually causes the next-generation robot created by the creating means to perform learning again, and based on the learning results. It is preferable to calculate the fitness of each robot.

  In this case, the robot with high fitness is selected as the parent individual as it is by the elite strategy, so that it is possible to prevent the robot with high learning performance from being killed without being selected by chance, and the adaptation selected by the tournament selection Since a next-generation robot can be created from a highly skilled robot, a robot with high learning performance can be prolific. As a result, robots with high learning performance can be sequentially learned without being killed, so that an optimal sensor configuration can be designed based on the learning results.

  The genotype preferably specifies at least one of the position, number, resolution, and sensing interval of the sensor. In this case, forms such as the position, number, resolution, and sensing interval of the sensors can be automatically designed.

  The next-generation creation means creates a next-generation genotype using at least one of crossover and mutation from the genotype of a robot that becomes a parent individual, and includes a plurality of robots having a sensor form specified by each genotype. It is preferable to create it virtually. In this case, an optimal genotype can be searched from a wide search space, and an optimal sensor can be efficiently designed.

  A sensor design method according to the present invention is a sensor design method for designing a form of a sensor used for robot behavior learning using a sensor design device including an initial generation creation means, a learning means, and a next generation creation means. A first step in which the initial generation creating means creates a plurality of genotypes for specifying a sensor form and virtually creates a plurality of robots having a sensor form specified by each genotype; The learning means causes the plurality of robots created by the initial generation creating means to virtually perform learning, and the next generation creating means performs learning by calculating the fitness of each robot based on the learning result. Based on the fitness of each robot calculated by the means, a plurality of robots as parent individuals are selected, and the next generation remains based on the genetic algorithm from the genotypes of the selected robots. A third step of creating a sub-type and virtually creating a plurality of robots having a sensor form specified by each next-generation genotype; and a learning means for the robot created in the third step. A fourth step of virtually re-learning and calculating the fitness of each robot based on the learning result, and determining the form of the sensor by repeating the processes in the third and fourth steps a predetermined number of times To do.

  A sensor design program according to the present invention is a sensor design program for designing a sensor form used for robot behavior learning, and creates a plurality of genotypes for specifying a sensor form, An initial generation creation unit that virtually creates a plurality of robots having a sensor form specified by a mold, and a plurality of robots created by the initial generation creation unit virtually perform learning, and based on the learning result A learning means for calculating the fitness of each robot, and a plurality of robots that become parent individuals are selected based on the fitness of each robot calculated by the learning means, and the genetic algorithm is selected from the genotypes of the selected robots. Next generation creation that creates next generation genotypes and virtually creates multiple robots with sensor forms identified by each next generation genotype The computer functions as a stage, and the learning means virtually re-learns the robot created by the next generation creating means, calculates the fitness of each robot based on the learning result, and the next generation creating means and learning The form of the sensor is determined by repeating the processing by the means a predetermined number of times.

  The robot according to the present invention has a sensor designed by any one of the sensor design apparatuses described above.

  According to the present invention, based on the learning result, a robot with high fitness, that is, a robot with high learning performance, is selected as a parent individual robot without dying, and evolves a genotype that identifies the form of the sensor. Therefore, it is possible to automatically design a sensor form that is advantageous for learning behavior of the robot.

  Hereinafter, a sensor design device according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a sensor design apparatus according to an embodiment of the present invention.

  The sensor design apparatus shown in FIG. 1 includes an input unit 1, an initial generation creation unit 2, a behavior learning unit 3, a selection unit 4, a next generation creation unit 5, and an output unit 6. The sensor design apparatus uses a normal computer including a ROM (Read Only Memory), a CPU (Central Processing Unit), a RAM (Random Access Memory), an external storage device, a recording medium driving device, an input device, a display device, and the like. The above functions can be realized by executing a sensor design program for executing a sensor design process, which will be described later, on a CPU or the like.

  In the present embodiment, the computer and Webbots manufactured by Cyberbotics are used as a simulator, the characteristics of sensors and actuators are defined, and a robot described later is virtually created to simulate the sensor configuration. The configuration of the sensor design apparatus is not particularly limited to the above example, and various modifications such as realizing a part or all of the above functions by a dedicated hardware circuit are possible.

  The input unit 1 is used by a user to input model data and the like for defining sensors, actuators, controllers (agents) and the like constituting the robot.

  The initial generation creation unit 2 creates a plurality of genotypes using the model data input from the input unit 1 so that the genotypes for specifying the sensor form are random, and identifies each genotype A plurality of initial generation robots having the form of a sensor to be created is virtually created, and data of the created initial generation robots is output to the behavior learning unit 3. The actuators, controllers, and the like, which are components other than the sensors of the initial generation robot, are common to all robots, and the same applies to the next generation robot described later.

  Here, the form of the sensor includes not only the physical form but also the characteristics and use state of the sensor, and the genotype used in the present embodiment includes the position, number, resolution, sensing interval, and sensing interval of the sensor. This is information that specifies a code or the like that specifies a parameter such as a coupling strength that represents the weight of the sensor value for the control system, and is expressed using “1” or “0”. For example, when one parameter is decoded in 4 bits and a genotype is expressed from 4 types of parameters, the length of the genotype is 256 bits.

The behavior learning unit 3 causes the initial generation robot to virtually perform reinforcement learning using the data of the initial generation robot, calculates the fitness based on the learning result, and outputs it to the selection unit 4 together with the data of each robot. . For example, the behavior learning unit 3 causes the robot to perform learning using Q learning, and causes the controller of the robot to virtually execute the following processing.
1. An environmental state _st is observed using a sensor.
2. To perform the action a _t in accordance with the action selection strategy.
3. It receives a reward r _t depending on the state.
4). The state s _{t + 1} after the state transition is observed using the sensor.
5. The Q value is updated according to the following formula (1).

Q (s _t , a _t ) <-Q (s _t , a _t ) + α [r _t + γmax _{at + 1} Q (s _{t + 1} , a _{t + 1} ) −Q (s _t , a _t )] (1)
Here, α is a learning rate (0 <α <1), and γ is a discount rate (0 <γ <1).
6). Advance time step t to t + 1 and return to procedure 1.

  The selection unit 4 selects a predetermined number of upper robots as parent individuals based on fitness according to the elite strategy, and further selects the same number of robots as parent individuals from the remaining robots that were not selected according to the tournament selection. The data is output to the next generation creation unit 5.

  The next generation creation unit 5 creates a next generation genotype based on a genetic algorithm from the genotype of the parent individual selected by the selection unit 4, and a plurality of next generations having a sensor form specified by each genotype A robot is virtually created, and the data of the created next-generation robot is output to the behavior learning unit 3. For example, the next generation creation unit 5 creates the next generation genotype by crossover and mutation from the genotype of the parent individual selected by the selection unit 4.

  The selection operation and the gene operation by the selection unit 4 and the next generation creation unit 5 are not particularly limited to the above example, and the lower (for example, 10%) robots are removed based on the fitness, and the tournament from the remaining ones. Various changes such as selection, roulette selection, expected value selection, ranking selection, etc., selecting the same number of parents and adding genetic manipulations are possible. In addition, various crossover methods such as one-point crossover, multipoint crossover, and uniform crossover can be used.

  The behavior learning unit 3 that has received the next-generation robot data output from the next-generation creation unit 5 causes the next-generation robot to virtually perform reinforcement learning, for example, Q learning using the data, and the learning result The fitness based on the above is output to the selection unit 4 together with the data of each robot.

  By repeating the processes by the selection unit 4, the next generation creation unit 5 and the behavior learning unit 3 for a predetermined generation, a final generation robot is virtually created, and the selection unit 4 has the highest fitness among the last generation robots. A high robot is selected, and the data is output to the output unit 6 as the best individual.

  The output unit 6 informs the user of the form of the sensor finally determined by displaying or printing the data of the best individual. In addition, as a form of the sensor which the output part 6 outputs, any of a genotype, a display type, etc. may be used.

  In the present embodiment, the initial generation creation unit 2 corresponds to an example of an initial generation creation unit, the behavior learning unit 3 corresponds to an example of a learning unit, and the selection unit 4 and the next generation creation unit 5 serve as next generation creation units. The selection unit 4 corresponds to an example of a selection unit, and the next generation generation unit 5 corresponds to an example of a generation unit.

  Next, sensor design processing by the sensor design apparatus configured as described above will be described. FIG. 2 is a flowchart for explaining sensor design processing by the sensor design apparatus shown in FIG.

  First, when a user inputs model data or the like for defining sensors, actuators, controllers, and the like constituting the robot using the input unit 1, in step S1, the initial generation creating unit 2 inputs the input model data. Etc. are used to create a plurality of genotypes so that the genotypes for specifying the sensor morphology are random.

  Here, an example of a robot whose sensor form is designed by the sensor design apparatus will be described. 3 is a schematic bottom view showing an example of a robot to be designed by the sensor design apparatus shown in FIG. 1, FIG. 4 is a schematic diagram showing a task environment of the robot shown in FIG. 3, and FIG. It is a schematic diagram which shows the state of the robot on the course shown in FIG.

  The robot 10 shown in FIG. 3 is rotatable to a sensor 11 composed of an infrared light emitting LED and a light receiving element, a fixed base 12 to which the sensor 11 is fixed, a main body 13 that holds the fixed base 12, and a main body 13. Two wheels 14 to be supported are provided. The robot 10 is a robot that moves along a line drawn on the floor and aims at a goal, and is called a line tracer.

  The course shown in FIG. 4 is a rectangle of 2 m × 5 m, and check points P1 to P4 are provided at each quarter point. As shown in FIG. 5, the robot 10 moves along the line LI. Aim for the goal. Here, the sensor 11 of the robot 10 outputs a value corresponding to the color of the floor surface to a controller (not shown) in the main body unit 13 in order to detect the line LI provided on the course. By controlling a motor and a drive circuit (not shown) in the unit 13, the wheel 14 is driven so as to trace the line LI according to the output of the sensor 11.

  The sensor 11 can be arranged by selecting an arbitrary number and position from 32 pre-defined 8 × 4 matrix arrangement possible positions. The position and the number of sensors 11 advantageous to the above are automatically designed. In this case, in step S1, model data and the like for modeling the line tracer and the course are input, and the initial generation creating unit 2 uses the input model data and the like to determine the position and number of sensors 11. A 32-bit genotype for identification is encoded, and 50 genotypes are randomly generated.

  Next, in step S2, the initial generation creation unit 2 virtually creates a plurality of initial generation robots having a sensor form specified by each created genotype, and performs behavior learning on the data of the created initial generation robots. Output to part 3. At this time, the behavior learning unit 3 initializes the time step t and the like.

  For example, in the case of the above-described line tracer, in step S2, the initial generation creation unit 2 virtually creates 50 initial generation robots having the positions and arrangements of the sensors specified by each of the 50 genotypes. The number of individuals (the number of groups) of the initial generation robot is 50.

Next, in step S3, action learning unit 3, using the sensor to the controller of the virtual robot modeled with data of the initial generation robots to observe the state s _t environment. For example, the environmental state _st is composed of the output of each sensor, the output of the sensor is binary, and in the case of a robot having four sensors, the number of states is 2 × 2 × 2 × 2 = 16. Note that the output value of the sensor is not particularly limited to the above example, and multiple values may be used.

Next, in step S4, action learning unit 3, to perform an action a _t in accordance with the ε greedy strategy as an action selection strategy to the controller. Here, the ε greedy strategy is a strategy in which a random action is selected with a probability of ε, and in other cases, an action having the maximum Q value is taken.

For example, if the line tracer, the action _{a t} the controller can take, straight _{a t0, a t1} to turn left, _{a t2} to turn right, _{a t3} to Sharp left, five _{a t4} to Sharp right There, when the wheel angular velocity (rad / s) to omega _L of the right and left of the robot, when the omega _R, in the case of _{a t0} of _{(ω L, ω R) =} (15,15), a t1 (ω L , Ω _R ) = (13, 6), (ω _L , ω _R ) = (6, 13) for a _t2 , and (ω _L , ω _R ) = (8, 2) for a _t3 , a _In the case of _t4 , the angular velocity of the wheel is controlled so that (ω _L , ω _R ) = (2, 8), and ε = 0.01 can be used.

Next, in step S5, action learning unit 3, reward r _t to the controller according to the state. For example, in the case of the above-described line tracer, it is possible to use a reward in which punishment increases as the center of the robot moves off the line. The eight rows of sensors shown in FIG. Assuming that the first column,..., The seventh column, the observed state is (1) when the sensor in the 0th column or the sensor in the seventh column is on the line, r _t = −20.0, (2) the first R _t = -10.0 if the column sensor or the sixth column sensor is on the line, (3) r _t = 0 if the second column sensor or the fifth column sensor is on the line. 0, (4) r _t = 1.0 if the third row sensor or fourth row sensor is on line, (5) r _t = −100.0 if all sensors are off line The reward r _t can be used. In addition, said conditions (1)-(4) are used preferentially in order of (4), (3), (2), (1).

Next, in step S6, the behavior learning unit 3 causes the controller to update the Q value according to the above equation (1). According to the above equation (1), to obtain the reward _{r t} by the action _{a t} the robot has taken _{Q (s t, a} t) is updated, if the reward is positive, _{Q (s t, a} t ) Will increase, the next time you are in the same state, you are more likely to take that action. For example, in the case of the above line tracer, the learning rate α = 0.8 and the discount rate γ = 0.999 can be used in the equation (1).

  Next, in step S7, the behavior learning unit 3 determines whether or not the time step t has reached the maximum number of steps and one trial has ended, and if the maximum number of steps has not been reached, the time step t is set to t + 1. The process returns to step S3 and the subsequent processing is continued. If the maximum number of steps has been reached, the process proceeds to step S8. For example, in the case of the above-described line tracer, the maximum number of steps = 2000 can be used, and when the line tracer reaches the goal within the maximum number of steps, one trial is completed, so the process proceeds to step S8. .

  Next, in step S8, the behavior learning unit 3 determines whether or not the learning has ended because the number of trials has reached the number of episodes. If learning has not ended, the learning conditions and the like are initialized and step S3 is performed. Returning to step S3, the next trial is continued, and when learning is completed, the process proceeds to step S9. For example, in the case of the above-described line tracer, the number of episodes = 100 can be used, and the position and orientation of the robot are initialized at each trial opportunity (episode) to continue the next trial.

  When learning is completed, in step S9, the behavior learning unit 3 calculates the fitness based on the learning result, and outputs the corresponding responsiveness to the selection unit 4 together with the data of each learned robot. For example, in the case of the above-described line tracer, 100 trials are performed per individual, and the fitness φ is calculated by the following equation (2).

Here, N is the number of episodes (100 times), H _i is the degree of achievement of the i-th trial, t _i is the arrival time (sec) to the check point P _i shown in FIG. 4, and T _max is the maximum trial time (128 sec). ). From the above equation, the individual (robot) with advanced learning has higher achievement in each trial and higher fitness. That is, it can be said that an individual with high fitness has a form of a sensor that is easy to learn. Note that the fitness calculation method is not particularly limited to the above example, and various modifications are possible.

  Next, in step S <b> 10, the selection unit 4 determines whether the robot whose learning has been completed is the last generation robot based on the robot data output from the behavior learning unit 3. In the case of the final generation robot, the selection unit 4 selects the robot with the highest fitness, and outputs the data as the best individual to the output unit 6, and the output unit 6 displays the genotype of the sensor of the best individual. . For example, in the case of the above-mentioned line tracer, 50 is used as the final generation number, and after learning of the 50th generation robot, the genotype representing the position and number of sensors of the robot with the highest fitness is displayed. The

  On the other hand, if it is not the final generation robot, in step S11, the selection unit 4 selects a predetermined number of robots with higher fitness according to the elite strategy, and outputs the data to the next generation creation unit 5 as a parent individual. For example, in the case of the above-described line tracer, the number of elite is set to 5, and among the 50 robots, the 5 robots having the top 5 fitness levels are selected as parent individuals and are left unconditionally for the next generation. .

  Next, in step S12, the selection unit 4 randomly selects a predetermined number of individuals from the remaining robots that have not been selected according to the tournament selection, selects the individual with the highest fitness among them, and performs this process. By repeating until the number of groups is obtained, the parent individual is selected and the data is output to the next generation creation unit 5. For example, in the case of the above-described line tracer, 45 robots are selected as parent individuals by tournament selection from 45 robots that were not selected.

  Next, in step S13, the next generation creation unit 5 creates the next generation genotype by crossover and mutation from the genotype of the parent individual selected by the selection unit 4, and the sensor specified by each genotype. A plurality of next-generation robots having a form are virtually created, and data of the created next-generation robots is output to the behavior learning unit 3. For example, in the case of the above line tracer, gene manipulation is performed from 50 robots selected as parent individuals with mutation rate = 0.03 and crossover rate = 1.0, and 50 next-generation robots are created. Is done.

  After the process of step S13, the process returns to step S3, the next generation robot learns, and further next generation robots are sequentially created by the genetic algorithm, and the learning of the last generation robot is completed until steps S3 to S13 are completed. The process is repeated.

  Through the above processing, in this embodiment, a plurality of genotypes for specifying the sensor form are created, and a plurality of initial generation robots having the sensor form specified by each genotype are virtually created. Then, Q learning is performed on a plurality of initial generation robots, and the fitness of each robot is calculated based on the learning result of each learned robot.

  Next, based on the calculated fitness of each robot, a predetermined number of robots with high learning performance are selected as a parent individual from a plurality of robots, and the same number of robots are selected from the remaining robots by tournament selection. The next generation genotype is created by crossover and mutation from the selected parent individual's genotype, and multiple robots with the form of sensors specified by each next generation genotype are next generation Virtually created as a robot, let multiple next-generation robots perform Q-learning again, the fitness of each robot is calculated based on the learning results, and these next-generation creation processing and learning processing are repeated until the final generation As a result, the form of the sensor is determined.

  Therefore, based on the learning result, a robot with high fitness, that is, a robot with high learning performance, is selected as a parent individual robot without dying, and the genotype that identifies the form of the sensor can be evolved, It is possible to automatically design sensor forms that are advantageous for robot behavior learning, and to design sensor forms autonomously based on learning results in order to construct a state space suitable for the robot's learning ability it can.

  It is also possible to acquire a learning algorithm that is optimal for the form of a sensor that is automatically designed by learning that is virtually performed at the time of design. Furthermore, since the robot is specially designed for the robot, it is possible to construct a sensor that is the interface between the physical world and the information world from the bottom up, and the actuator is fixed. There is also an advantage that it is easy to realize on hardware.

  Next, a sensor design result by the sensor setting device will be specifically described taking the line racer shown in FIG. 3 as an example. FIG. 6 is a diagram showing changes in the fitness of each generation of the line tracer designed by the sensor setting device shown in FIG. 1 and the number of sensors of the best individual in each generation. In the figure, the solid line indicates the maximum fitness, the broken line indicates the number of sensors of the best individual, and the alternate long and short dash line indicates the average fitness when the experiment is performed 10 times.

  It can be seen from FIG. 6 that the fitness of the best individual increases as the number of sensors decreases. For example, the fitness of the fifth generation (around 7 sensors) is about 0.2, whereas the fitness of the 50th generation (around 5 sensors) is increased to about 0.4. As a result, in the case of the line trace environment shown in FIG. 4, about 5 sensors are advantageous for learning, and by using a learning device, the number of sensors (8) that are normally used in the microcomputer car rally can be reduced. It was found that it can be reduced.

  FIG. 7 is a diagram showing a representative example of the sensor form designed by the sensor design apparatus shown in FIG. 1, and the black circles in the figure represent the sensor. For example, when there are four sensors, the sensor arrangement shown in FIG. 7A is designed, and when there are five sensors, the sensor arrangement shown in FIG. 7B is designed and there are six sensors. The sensor arrangement shown in FIG. 7C was designed.

  Next, the learning results were compared using the sensor form designed by the sensor design apparatus shown in FIG. 1 and the sensor form designed by hand. FIG. 8 is a diagram showing an example of a sensor form designed by the sensor design apparatus shown in FIG. 1 and a sensor form designed by hand, and a black circle in the figure represents a sensor.

  The sensor form shown in FIG. 8C is a sensor form designed by the sensor design apparatus shown in FIG. 1, and five sensors are arranged at predetermined intervals in a plane. The sensor form shown in (a) of FIG. 8 is a sensor form used as standard in the microcomputer car rally, and eight sensors are arranged at equal intervals on a straight line. The sensor configuration shown in (b) of FIG. 8 is a sensor configuration of a microcomputer car having the same dimension of the state space as the sensor configuration shown in (c) of FIG. 8, and five sensors are arranged on a straight line at predetermined intervals. ing.

  FIG. 9 is a diagram showing a change in fitness with respect to the number of episodes of each sensor form shown in FIG. The change in the fitness with respect to the number of episodes shown in FIG. 9 represents the change in the speed of reaching the goal, the solid line shows the change in the sensor form shown in (c) of FIG. 8, and the alternate long and short dash line in FIG. ), And the broken lines indicate changes in the sensor form shown in FIG.

  From FIG. 9, the sensor form designed by the sensor design apparatus shown in FIG. 1, that is, the sensor form shown in FIG. 8 (c) is more complete than the manual sensor form shown in FIG. 8 (a) and (b). It can be seen that the episode shows high fitness. This trend was the same even when the number of episodes was increased. Further, the sensor configuration shown in FIG. 8C is superior to the sensor configuration shown in FIG. 8A with a large state space in terms of convergence speed and performance. Thus, it was found that more effective learning can be achieved by appropriately observing the physical world.

  Next, the arrangement and the number of sensor forms shown in FIG. 8C designed by the sensor design apparatus shown in FIG. 1 will be considered. First, the sensor configuration shown in FIG. 8C has the following characteristics with respect to the sensor arrangement.

  (1) The sensor arrangement is asymmetrical. This is because the current task is set to turn the course counterclockwise, so there are many left curves except for the S-curve, and the fitness of the sensor form that is good at the left curve has increased. it is conceivable that.

  (2) Sensors are distributed in the front-rear direction. This is because in a horizontal arrangement, the line context cannot be read, and it cannot be determined whether the line tracer is on a curve or on a straight line, but the sensors are scattered back and forth. This is considered to be because the shape of a line such as a straight line or a curve can be read from the context of the line.

  Next, with regard to the number of sensors, automatically designed sensor forms often have around five sensors, and the number of sensors determines the dimension of the state space, which affects the learning speed and the amount of information that can be acquired. give. Generally, learning converges faster with fewer dimensions, but the performance after learning convergence is considered to be lower than when there are many dimensions. However, in the above results, even after learning has converged, a smaller number of sensors showed higher performance. As a result, it was found that if a small state space can be used effectively, not only the convergence speed but also the learning effect can be improved.

  Thus, by using this sensor design apparatus, a state space suitable for the task environment and the learning ability of the robot can be configured, and adaptive behavior can be executed by the robot.

  Next, a line tracer having the form of the sensor designed as described above (the line tracer shown in FIG. 8C) is actually created to perform line tracing, and the line tracer designed manually (FIG. 8). (A line tracer shown in (a) and (b)). As the controller, the line tracer shown in (a) of FIG. 8 uses a hand-coded version of the attached sample program, and the line tracer shown in (b) and (c) of FIG. A learning result is used to determine an action based on the Q value obtained at the time of design, and an action having the maximum Q value with respect to the state observed by the sensor is selected.

  Each line tracer mentioned above is tried 5 times, and one trial is ended when the course goes 10 laps or out of the course, and the average lap time (seconds) is measured to compare the running performance, and the robustness is improved. The average stay lap number was measured for comparison. The average lap time of the line tracer shown in (c) of FIG. 8 is 13.5 seconds, the average stay lap number is 8.4, and the average lap time of the line tracer shown in (a) of FIG. The number of stay laps is 4.4, the average lap time of the line tracer shown in FIG. 8B is 16.0 seconds, the average stay lap number is 3.0, and is designed by the sensor design apparatus shown in FIG. Further, the line tracer having the sensor form was excellent in both running performance and robustness.

  Next, how the above-described line tracers use the sensor arrangement will be described by taking as an example the capture of a right-angled corner that is considered to be difficult in line tracing.

  First, the line tracer shown in FIG. 8A employs a method in which a cross line existing immediately before a right corner is used as a signal, and when the cross line immediately before the right corner is read, Go to mode and clear the right corner using a special control law. In this way, the manual controller can use the designer's knowledge of the course that there is a right-angle corner after the cross line, so it breaks through the right-angle corner using a control law different from the normal curve. be able to.

  FIG. 10 is a schematic diagram for explaining how to use the sensor form at the right-angled corner of the line tracer shown in FIG. Since the line tracer shown in FIG. 8C cannot use the designer's knowledge about the course, the right-angled corner must be cleared using the same control law as that of a normal curve. For this reason, when approaching a right-angled corner, the sensor reacts as shown in FIG. Since this is the same sensor state as the left curve, the line tracer turns slightly to the left. However, since the course is actually a right angle, the course is likely to deviate from the course as shown in FIG. At this time, since the fourth sensor detects the center line and the line tracer makes a large left turn, the right-angled corner can be cleared.

  On the other hand, since the line tracer shown in FIG. 8B has sensors arranged in a horizontal row, the line tracer often goes out of course at a right angle corner. This is presumably because the sensors were not distributed back and forth, so that the context on the course could not be learned.

  As described above, the line tracer designed by the sensor design apparatus shown in FIG. 1 is superior in running performance and robustness as compared to manual design by appropriately observing the physical world, and also in learning performance. I found it excellent.

  In the above description, the line tracer has been described as an example, but the robot to which the present invention is applied is not particularly limited to this example, and can be applied to various robots as long as the robot uses a sensor.

It is a block diagram which shows the structure of the sensor design apparatus by one embodiment of this invention. It is a flowchart for demonstrating the sensor design process by the sensor design apparatus shown in FIG. It is a bottom face schematic diagram which shows an example of the robot used as the design object of the sensor design apparatus shown in FIG. It is a schematic diagram which shows the task environment of the robot shown in FIG. It is a schematic diagram which shows the state of the robot on the course shown in FIG. It is a figure which shows the change of the fitness in each generation of the line tracer designed by the sensor setting apparatus shown in FIG. 1, and the number of sensors of the best individual in each generation. It is a figure which shows the typical example of the sensor form designed by the sensor design apparatus shown in FIG. It is a figure which shows the example of the sensor form designed by the sensor design apparatus shown in FIG. 1, and the sensor form designed manually. It is a figure which shows the change of the fitness with respect to the number of episodes of each sensor form shown in FIG. It is a schematic diagram for demonstrating the utilization method of the sensor form in the right-angled corner of the line tracer shown in FIG.8 (c).

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Input part 2 Initial generation creation part 3 Behavior learning part 4 Selection part 5 Next generation creation part 6 Output part

Claims (8)

A sensor design device for designing the form of a sensor used for robot behavior learning,
An initial generation creating means for creating a plurality of genotypes for specifying the form of the sensor and virtually creating a plurality of robots having the form of the sensor specified by each genotype;
Learning means for virtually learning a plurality of robots created by the initial generation creating means, and calculating fitness of each robot based on a learning result;
Based on the fitness of each robot calculated by the learning means, select a plurality of robots that are parent individuals, create a next-generation genotype based on a genetic algorithm from the genotypes of the selected plurality of robots, A next generation creation means for virtually creating a plurality of robots having a sensor form specified by the next generation genotype,
The learning means causes the robot created by the next generation creating means to virtually perform learning again, calculates the fitness of each robot based on the learning result,
A sensor design apparatus for determining a sensor form by repeating a predetermined number of processes by the next generation creating means and the learning means. The sensor design apparatus according to claim 1, wherein the learning unit causes the robot to perform learning using Q-learning. The next generation creation means is:
Based on the fitness of each robot calculated by the learning means, a predetermined number of robots with high learning performance are selected as parent individuals from the plurality of robots, and the same number of robots are selected by tournament selection from the remaining robots. A selection means for selecting as a parent individual;
A next generation genotype is created from a genotype of a parent individual selected by the selection means based on a genetic algorithm, and a plurality of robots having a sensor form specified by each next generation genotype are used as next generation robots. A creation means for creating virtually,
3. The learning unit according to claim 1, wherein the learning unit virtually causes the next generation robot created by the creating unit to perform learning again, and calculates the fitness of each robot based on the learning result. Sensor design device. The sensor design device according to any one of claims 1 to 3, wherein the genotype specifies at least one of the position, number, resolution, and sensing interval of the sensor. The next-generation creation means creates a next-generation genotype using at least one of crossover and mutation from the genotype of the robot as a parent individual, and a plurality of robots having a sensor form specified by each genotype The sensor design device according to claim 1, wherein the sensor design device is created virtually. A sensor design method for designing a form of a sensor used for behavioral learning of a robot using a sensor design apparatus including an initial generation creation means, a learning means, and a next generation creation means,
A first step in which the initial generation creating means creates a plurality of genotypes for specifying the form of the sensor and virtually creates a plurality of robots having the form of the sensor specified by each genotype; ,
A second step in which the learning means virtually causes the plurality of robots created by the initial generation creation means to perform learning, and calculates the fitness of each robot based on the learning results;
The next generation creation means selects a plurality of robots as parent individuals based on the fitness of each robot calculated by the learning means, and generates a next generation based on a genetic algorithm from a genotype of the selected plurality of robots. A third step of creating a genotype and virtually creating a plurality of robots having the form of a sensor specified by each next generation genotype;
The learning means includes a fourth step of causing the robot created in the third step to virtually re-learn and calculating the fitness of each robot based on the learning result;
A sensor design method characterized by determining a sensor form by repeating a predetermined number of processes in the third and fourth steps. A sensor design program for designing the form of a sensor used for robot behavior learning,
An initial generation creating means for creating a plurality of genotypes for specifying the form of the sensor and virtually creating a plurality of robots having the form of the sensor specified by each genotype;
Learning means for virtually learning a plurality of robots created by the initial generation creating means, and calculating fitness of each robot based on a learning result;
Based on the fitness of each robot calculated by the learning means, select a plurality of robots that are parent individuals, create a next-generation genotype based on a genetic algorithm from the genotypes of the selected plurality of robots, The computer functions as a next generation creation means for virtually creating a plurality of robots having the form of a sensor specified by the next generation genotype,
The learning means causes the robot created by the next generation creating means to virtually perform learning again, calculates the fitness of each robot based on the learning result,
A sensor design program for determining a sensor form by repeating a predetermined number of processes by the next generation creating means and the learning means. A robot having a sensor designed by the sensor design device according to claim 1.

Images