JP2022122670A - Robot model learning device, robot model machine learning method, robot model machine learning program, robot control device, robot control method, and robot control program - Google Patents

Abstract

To enable a robot model to learn efficiently through machine learning.SOLUTION: A robot control device 40 is configured to: acquire an actual value of a position attitude of a robot 10 and an actual value of external force applied to the robot 10, and executes a robot model LM including a state transition model for calculating a prediction value of the position attitude of the robot 10 and an external force model for calculating a prediction value of the external force applied to the robot 10, on the basis of an actual value of the position attitude at a certain time and an action command which can be given to the robot 10; calculate a reward on the basis of an error of the position attitude and the prediction value of the external force; generate a plurality of action command candidates and give them to the robot model LM, for each control period; determine an action command that maximizes the reward on the basis of rewards calculated in correspondence with the plurality of action command candidates respectively; and update the external model so as to minimize an error between the prediction value of the external force calculated using the external force model according to the determined action command and the actual value of the external force corresponding to the prediction value of the external force.SELECTED DRAWING: Figure 1

Description

The disclosed technology relates to a robot model learning device, a robot model machine learning method, a robot model machine learning program, a robot control device, a robot control method, and a robot control program.

Machine learning is used to learn a robot model in order to automatically acquire the control law necessary for a robot to accomplish a task.

For example, Patent Literature 1 discloses a control device for controlling an industrial robot having a function of detecting force and moment applied to a manipulator, comprising a control unit for controlling the industrial robot based on a control command; a data acquisition unit that acquires at least one of a force and a moment applied to a manipulator of a robot for use as acquired data; force state data including information related to the force applied to the manipulator based on the acquired data; a preprocessing unit that generates control command adjustment data indicating control command adjustment behavior as state data, and executes machine learning processing related to the control command adjustment behavior of the manipulator based on the state data. A technique for doing so is disclosed.

JP 2020-055095 A

However, it is difficult to set parameters and design a reward function when learning a robot model by machine learning, and it is difficult to learn efficiently.

The disclosed technique has been made in view of the above points, and provides a robot model learning apparatus, a robot model machine learning method, and a robot model that enable efficient learning when a robot model is learned by machine learning. machine learning program, robot control device, robot control method, and robot control program.

A first aspect of the disclosure is a robot model learning apparatus comprising: an acquisition unit that acquires actual values of the position and orientation of a robot and actual values of an external force applied to the robot; A robot model that includes a state transition model that calculates a predicted value of the position and orientation of the robot at the next time based on an action command that can be given to the robot, and an external force model that calculates a predicted value of an external force applied to the robot. a model execution unit that executes the robot model; and a reward calculation unit that calculates a reward based on the error between the predicted value of the position and orientation and the target value of the position and orientation to be reached, and the predicted value of the external force. generating a plurality of candidates for the action command in each control cycle and giving them to the robot model; between an action determination unit that determines an action command to be converted, the predicted value of the external force calculated by the external force model based on the determined action command, and the actual value of the external force corresponding to the predicted value of the external force and an external force model updating unit that updates the external force model so that the difference between is reduced.

In the first aspect, an error between the predicted value of the position and orientation calculated by the state transition model based on the determined action command and the actual value of the position and orientation corresponding to the predicted value of the position and orientation. and a state transition model updating unit that updates the state transition model so that

In the first aspect, when the external force is a corrected external force that suppresses the expansion of the error, the remuneration calculation unit calculates the remuneration using a predicted value of the corrected external force as a factor for decreasing the remuneration. may be calculated.

In the first aspect, when the external force is a hostile external force that suppresses reduction of the error, the reward calculation unit calculates the reward using the predicted value of the hostile external force as a factor for increasing the reward. may be calculated.

In the first aspect, the remuneration calculation unit calculates the remuneration by calculation using a predicted value of the corrected external force as a reduction factor of the remuneration when the external force is a corrected external force that suppresses the expansion of the error. , when the external force is a hostile external force that suppresses the reduction of the error, the reward may be calculated by calculation using a predicted value of the hostile external force as an increase factor of the reward.

In the first aspect, the reward calculation unit makes the range of change in the amount of decrease in the reward based on the predicted value of the corrected external force during task execution smaller than the range of change in the amount of decrease in the reward based on the error, and the task The remuneration may be calculated by a calculation in which a change range of the increase amount of the remuneration based on the predicted value of the hostile external force during execution is smaller than a change range of the remuneration based on the error.

In the first aspect, the external force model includes a corrected external force model that outputs a predicted value of the corrected external force when the external force is the corrected external force, and the hostile external force when the external force is the hostile external force. and a hostile external force model that outputs a predicted value of the corrected external force calculated by the corrected external force model based on the determined action command when the external force is the corrected external force. and a modified external force model updating unit that updates the modified external force model so that the difference between the predicted value of the external force and the actual value of the external force becomes smaller; and a hostile external force model updating unit that updates the hostile external force model so that the difference between the predicted value of the hostile external force calculated by the hostile external force model and the actual value of the external force becomes smaller.

In the first aspect, the robot model includes an integrated external force model comprising the corrected external force model and the hostile external force model, the corrected external force model and the hostile external force model are neural networks, and one of the hostile external force models Alternatively, at least one layer of a plurality of intermediate layers and output layers integrates outputs of layers preceding the corresponding layer of the modified external force model by a progressive neural network technique, and the hostile external force model predicts an external force. A value and identification information indicating whether the external force is a corrected external force or a hostile external force, the integrated external force model uses the output of the hostile external force model as its own output, and the reward calculation unit determines that the identification information indicates the corrected external force. If the identification information indicates a hostile external force, the reward is calculated using the predicted value of the external force as a factor to decrease the reward, and if the identification information indicates a hostile external force, a calculation using the predicted value of the external force as a factor to increase the reward The reward may be calculated.

In the above first aspect, further comprising a receiving unit that receives a designation as to whether the external force is the corrected external force or the hostile external force, and if the designation is the corrected external force, causes the corrected external force model updating unit to operate. The configuration may further include a learning control unit that validates and validates the operation of the hostile external force model updating unit when the designation is the hostile external force.

In the first aspect, when it is determined whether the external force is the corrected external force or the hostile external force based on the actual values of the position and orientation and the actual values of the external force, and the result of the determination is the corrected external force. may further include a learning control unit that validates the operation of the corrected external force model updating unit, and validates the operation of the hostile external force model updating unit when the determination result is the hostile external force.

A second aspect of the disclosure is a machine learning method for a robot model, which is based on the actual values of the position and orientation of the robot at a certain time and the action command that can be given to the robot, and the position and orientation of the robot at the next time. and an external force model for calculating the predicted value of the external force applied to the robot are prepared. is obtained, and a plurality of candidates for the action command are generated and given to the robot model for each control cycle, and a plurality of candidates calculated by the state transition model corresponding to the plurality of candidates for the action command are obtained. based on a plurality of predicted values of the external force calculated by the external force model corresponding to a plurality of errors between the predicted values of the position and orientation and the target values of the position and orientation to be reached and the plurality of candidates of the action command; Then, an action command that maximizes the reward is determined based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command, and the external force model calculates the action command based on the determined action command. The external force model is updated so that the difference between the predicted value of the external force and the actual value of the external force corresponding to the predicted value of the external force is reduced.

In the second aspect, further, a difference between the position/orientation predicted value calculated by the state transition model based on the determined action command and the position/orientation actual value corresponding to the position/orientation predicted value The state transition model may be updated so that the error of is reduced.

In the second aspect, when the external force is a corrected external force that suppresses the expansion of the error, the reward is calculated by calculation using a predicted value of the corrected external force as a factor for reducing the reward. good too.

In the second aspect, when the external force is a hostile external force that suppresses reduction of the error, the reward is calculated by calculation using the predicted value of the hostile external force as a factor for increasing the reward. good too.

In the second aspect, when the external force is a corrected external force that suppresses the expansion of the error, the reward is calculated by calculation using the predicted value of the corrected external force as a factor for reducing the reward, and the external force is the error , the reward may be calculated by calculation using the predicted value of the hostile external force as a factor for increasing the reward.

In the second aspect, the external force model includes a corrected external force model that outputs a predicted value of the corrected external force when the external force is the corrected external force, and the hostile external force when the external force is the hostile external force. and a hostile external force model that outputs a predicted value of the external force, and when the external force is the corrected external force, the predicted value of the corrected external force calculated by the corrected external force model based on the determined action command and the external force Updating the modified external force model so that the difference from the actual value is reduced, and predicting the hostile external force calculated by the hostile external force model based on the determined action command when the external force is the hostile external force. The hostile external force model may be updated so that the difference between the value and the actual value of the external force becomes smaller.

In the second aspect, the correction external force is applied to the robot when the error is increasing, and the hostile external force is applied to the robot when the error is decreasing. good too.

A third aspect of the disclosure is a machine learning program for a robot model, which is based on the actual values of the position and orientation of the robot at a certain time and the action command that can be given to the robot, and the position and orientation of the robot at the next time. A machine learning program for machine learning a robot model including a state transition model for calculating a predicted value of and an external force model for calculating a predicted value of an external force applied to the robot, the machine learning program for machine learning the position and orientation of the robot at each control cycle A performance value and a performance value of the external force applied to the robot are obtained, a plurality of candidates for the action command are generated for each control cycle, and given to the robot model, and the plurality of candidates for the action command are obtained. Calculated by the external force model corresponding to a plurality of errors between the plurality of predicted values of the position and orientation calculated by the state transition model and the target values of the position and orientation to be reached, and the plurality of candidates for the action command. determining an action command that maximizes the reward based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command based on the plurality of predicted values of the external force; updating the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the external force model and the actual value of the external force corresponding to the predicted value of the external force is reduced; let it happen

A fourth aspect of the disclosure is a robot control device, which predicts the position and orientation of the robot at the next time based on the actual values of the position and orientation of the robot at a certain time and the action command that can be given to the robot. and an external force model for calculating a predicted value of the external force applied to the robot; a model execution unit that executes a robot model; an acquisition unit that acquires a value, and a reward based on the error between the predicted value of the position and orientation calculated by the robot model and the target value of the position and orientation to be reached, and the predicted value of the external force calculated by the robot model. and a reward calculation unit that calculates a plurality of candidates for the action command for each control cycle, and provides the plurality of candidates for the action command to the robot model, and the reward calculation unit calculates each of the plurality of candidates for the action command. a behavior determination unit that determines a behavior directive that maximizes the reward based on the reward.

A fifth aspect of the disclosure is a robot control method, which predicts the position and orientation of the robot at the next time based on the actual values of the position and orientation of the robot at a certain time and the action command that can be given to the robot. and an external force model for calculating a predicted value of the external force applied to the robot are prepared, and in each control cycle, the actual values of the position and orientation and the actual values of the external force applied to the robot are prepared. is obtained, and a plurality of candidates for the action command are generated and given to the robot model for each control cycle, and a plurality of the positions and orientations calculated by the state transition model corresponding to the plurality of candidates for the action command are obtained. based on a plurality of predicted values of the external force calculated by the external force model corresponding to a plurality of errors between the predicted value of and the target value of the position and orientation to be reached, and the plurality of candidates for the action command, An action command that maximizes the reward is determined based on a plurality of rewards calculated corresponding to the plurality of action command candidates, and the robot is controlled based on the determined action command.

A sixth aspect of the disclosure is a robot control program, which predicts the position and orientation of the robot at the next time based on the actual values of the position and orientation of the robot at a certain time and the action command that can be given to the robot. and an external force model for calculating a predicted value of an external force applied to the robot, the program for controlling the robot using a robot model, wherein the position and orientation results are obtained in each control cycle values and actual values of the external force applied to the robot, generate a plurality of candidates for the action command for each control cycle, and give the candidates to the robot model; The plurality of errors calculated by the external force model corresponding to the plurality of candidates for the action command and the plurality of errors between the plurality of predicted values of the position and orientation calculated by the transition model and the target values of the position and orientation to be reached. Based on the predicted value of the external force of, based on a plurality of rewards calculated corresponding to the plurality of candidates for the action command, determine an action command that maximizes the reward, and based on the determined action command A computer is caused to perform each process to control the robot by using a computer.

According to the disclosed technology, it is possible to learn efficiently when learning a robot model by machine learning.

1 is a configuration diagram of a robot system according to a first embodiment; FIG. (A) is a diagram showing a schematic configuration of the robot 10, and (B) is an enlarged diagram of the tip side of the arm of the robot. It is a figure for demonstrating corrected external force and hostile external force. It is a block diagram which shows the hardware constitutions of a robot control apparatus. 1 is a configuration diagram of a robot model; FIG. FIG. 11 is a configuration diagram of a robot model according to a modified example; FIG. 11 is a configuration diagram of a robot model according to a modified example; It is a block diagram of an integrated external force model. 4 is a flowchart of learning processing according to the first embodiment; 9 is a flowchart of learning processing according to the second embodiment; FIG. 11 is a configuration diagram of a robot system according to a third embodiment; 10 is a flowchart of learning processing according to the third embodiment; (A) is a graph showing a learning curve for position error, and (B) is a graph showing a learning curve for external force. (A) is a graph representing the number of task successes at different frictions, and (B) is a graph representing the number of task successes at different peg masses. FIG. 11 is a graph representing the number of task successes for different peg materials; FIG.

An example of embodiments of the technology disclosed herein will be described below with reference to the drawings. In each drawing, the same or equivalent components and portions are given the same reference numerals. Also, the dimensional ratios in the drawings may be exaggerated for convenience of explanation, and may differ from the actual ratios.

<First Embodiment>

FIG. 1 shows the configuration of a robot system 1 according to this embodiment. The robot system 1 has a robot 10 , a state observation sensor 20 , tactile sensors 30A and 30B, a robot control device 40 , a display device 50 and an input device 60 .

(robot)

2A and 2B are diagrams showing a schematic configuration of the robot 10. FIG. The robot 10 in this embodiment is a 6-axis vertical articulated robot, and a gripper (hand) 12 is provided at the tip 11 a of an arm 11 via a flexible portion 13 . The robot 10 performs a fitting operation in which a part (for example, a peg) is gripped by the gripper 12 and fitted into a hole. Although the robot 10 is a real robot in this embodiment, it may be a virtual robot in a simulation.

As shown in FIG. 2A, the robot 10 has an arm 11 with six degrees of freedom and joints J1 to J6. The joints J1 to J6 connect the links so as to be rotatable in the directions of arrows C1 to C6 by motors (not shown). Here, a vertical articulated robot is taken as an example, but a horizontal articulated robot (scalar robot) may be used. Also, although a 6-axis robot has been exemplified, a multi-joint robot with other degrees of freedom such as a 5-axis or 7-axis robot, or a parallel link robot may be used.

The gripper 12 has a pair of gripping portions 12a, and grips the component by controlling the gripping portions 12a. The gripper 12 is connected to the tip 11a of the arm 11 via the flexible portion 13 and moves as the arm 11 moves. In this embodiment, the flexible portion 13 is composed of three springs 13a to 13c arranged in a positional relationship such that the base of each spring is at each vertex of an equilateral triangle, but the number of springs may be any number. Also, the flexible portion 13 may be any other mechanism as long as it is a mechanism that generates a restoring force against positional fluctuations and obtains flexibility. For example, the flexible part 13 may be an elastic body such as a spring or rubber, a damper, a pneumatic or hydraulic cylinder, or the like. The flexible portion 13 is preferably constituted by passive elements. The flexible portion 13 allows the distal end 11a of the arm 11 and the gripper 12 to move relative to each other horizontally and vertically by 5 mm or more, preferably 1 cm or more, more preferably 2 cm or more.

A mechanism may be provided to switch the gripper 12 between a flexible state and a fixed state with respect to the arm 11 .

In addition, although the configuration in which the flexible portion 13 is provided between the tip 11a of the arm 11 and the gripper 12 is illustrated here, the flexible portion 13 is provided in the middle of the gripper 12 (for example, in the middle of the finger joint or in the middle of the columnar portion of the finger), in the middle of the arm ( For example, it may be provided at any of the joints J1 to J6 or in the middle of the columnar portion of the arm). Also, the flexible portion 13 may be provided at a plurality of these locations.

The robot system 1 uses machine learning (for example, model-based reinforcement learning) to acquire a robot model for controlling the robot 10 having the flexible part 13 as described above. Since the robot 10 has the flexible part 13, it is safe to bring the gripped part into contact with the environment, and even if the control cycle is slow, it is possible to perform fitting work. On the other hand, it is difficult to obtain an analytical robot model because the positions of the gripper 12 and parts are uncertain due to the flexible portion 13 . Therefore, in this embodiment, machine learning is used to acquire a robot model.

(state observation sensor)

The state observation sensor 20 observes the position and orientation of the gripper 12 as the state of the robot 10 and outputs the observed position and orientation as actual values. As the state observation sensor 20, for example, a joint encoder of the robot 10, a visual sensor (camera), a motion capture, or the like is used. When a motion capture marker is attached to the gripper 12 , the position and orientation of the gripper 12 can be identified, and the orientation of the component (workpiece) can be estimated from the position and orientation of the gripper 12 .

Also, the position and orientation of the gripper 12 itself and the part gripped by the gripper 12 can be detected as the state of the robot 10 by the visual sensor. If the flexible portion is between the gripper 12 and the arm 11 , the position and orientation of the gripper 12 with respect to the arm 11 can also be identified by a displacement sensor that detects the displacement of the gripper 12 with respect to the arm 11 .

(tactile sensor)

Although not shown in FIG. 2, tactile sensors 30A and 30B are attached to the gripper body 12b of the gripper 12 as shown in FIG.

As an example, the tactile sensors 30A and 30B are provided at positions along the direction in which the pair of holding portions 12a face each other. The tactile sensors 30A and 30B are, for example, sensors that detect three-axis or six-axis forces, and can detect the magnitude and direction of an external force applied to them. The user applies an external force to the gripper 12 by gripping the gripper main body 12b and moving the gripper 12 so that the hands (fingers) are in contact with both the tactile sensors 30A and 30B.

The external force includes a corrective external force that is advisory so that the task (work) executed by the robot 10 will succeed, and an adversarial external force that is adversarial so that the task will fail. The corrected external force is an external force that suppresses the expansion of the error between the predicted value of the position and orientation of the robot 10 predicted by the robot model and the target value of the position and orientation that the robot 10 should reach. The hostile external force is an external force that reduces the expansion of the error between the predicted value of the position and orientation of the robot 10 predicted by the robot model and the target value of the position and orientation that the robot 10 should reach.

Specifically, when the task to be executed by the robot 10 is to insert the peg 70 into the hole 74 provided in the table 72 as shown in FIG. It is necessary to move peg 70 in the direction of arrow A. In this case, the force applied in the direction of arrow A, which is the correct direction for successfully completing the task of inserting peg 70 into hole 74, is the corrected force. On the other hand, the external force applied in the direction of arrow B, which is the direction that causes the task to fail and is opposite to the direction of arrow A, is the hostile external force.

In the case of FIG. 3, when the user grips the gripper 12 and applies a corrective external force in the direction of arrow A, the force detected by the tactile sensor 30A becomes larger than that detected by the tactile sensor 30B, and it can be determined that the corrective external force is applied. On the other hand, when a hostile external force is applied in the direction of arrow B, the force detected by the tactile sensor 30B is greater than that detected by the tactile sensor 30A, and it can be determined that a hostile external force is being applied.

In this embodiment, the case where the gripper main body 12b is provided with the two tactile sensors 30A and 30B will be described, but the present invention is not limited to this. For example, three or more tactile sensors may be provided at equal intervals around the gripper body 12b. If three or more tactile sensors are provided and their detection results are combined to at least determine the direction of the external force in the plane perpendicular to the axis of the gripper 12, each tactile sensor detects only the magnitude of the external force. There may be.

(robot controller)

The robot control device 40 functions as a learning device that learns a robot model by machine learning. The robot control device 40 also functions as a control device that controls the robot 10 using a learned robot model.

FIG. 4 is a block diagram showing the hardware configuration of the robot control device according to this embodiment. As shown in FIG. 4, the robot control device 40 has the same configuration as a general computer (information processing device), and includes a CPU (Central Processing Unit) 40A, a ROM (Read Only Memory) 40B, a RAM (Random Access Memory). ) 40C, storage 40D, keyboard 40E, mouse 40F, monitor 40G, and communication interface 40H. Each component is communicatively connected to each other via a bus 40I.

In this embodiment, the ROM 40B or the storage 40D stores a program for machine learning a robot model and a robot control program. The CPU 40A is a central processing unit that executes various programs and controls each configuration. That is, the CPU 40A reads a program from the ROM 40B or the storage 40D and executes the program using the RAM 40C as a work area. The CPU 40A performs control of the above components and various arithmetic processing according to programs recorded in the ROM 40B or the storage 40D. The ROM 42 stores various programs and various data. The RAM 40C temporarily stores programs or data as a work area. The storage 40D is configured by a HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory, and stores various programs including an operating system and various data. The keyboard 40E and mouse 40F are examples of the input device 60 and are used for various inputs. The monitor 40G is, for example, a liquid crystal display, and is an example of the display device 50. FIG. The monitor 40G may employ a touch panel system and function as the input device 60 . The communication interface 40H is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark), for example.

Next, the functional configuration of the robot control device 40 will be described.

As shown in FIG. 1, the robot control device 40 includes an acquisition unit 41, a model execution unit 42, a reward calculation unit 43, an action determination unit 44, an external force model update unit 45, a learning control unit 46, and a user It has an interface (UI) control unit 47 . Each functional configuration is realized by CPU 40A reading a machine learning program stored in ROM 40B or storage 40D, developing it in RAM 40C, and executing it. Some or all of the functions may be realized by a dedicated hardware device.

The acquisition unit 41 acquires the actual values of the position and orientation of the robot 10 and the actual values of the external force applied to the robot 10 . The position and orientation of the robot 10 are, for example, the position and orientation of the gripper 12 as an end effector of the robot 10 . The external force applied to the robot 10 is, for example, an external force applied to the gripper 12 as an end effector of the robot 10 . Actual values of the external force are measured by the tactile sensors 30A and 30B. In the case where the robot 10 is such that it is difficult to identify which part is the end effector, the position and orientation of the robot may be measured and an external force may be applied to the part of the robot that affects the object to be manipulated. The location should be specified.

In this embodiment, the gripper 12 is provided at the tip 11a of the arm 11 via the flexible portion 13. Therefore, when an external force is applied to the gripper 12, the gripper 12 can be physically flexibly displaced. A configuration that can be displaced by control in response is preferred. The technology disclosed herein can also be applied by manually applying an external force to a rigid robot that does not have flexibility.

In the present embodiment, the position and orientation of the robot 10 are represented by values of a maximum of six degrees of freedom, ie, three degrees of freedom of position and three degrees of freedom of orientation. good too. For example, in the case of a robot in which the posture of the end effector does not change, the "position and posture" may be only three degrees of freedom.

The model execution unit 42 executes the robot model LM.

As shown in FIG. 5, the robot model LM is based on actual values (measured values) of the position and orientation at a certain time and action commands (candidate values or determined values) that can be given to the robot 10, and the robot model at the next time. A state transition model DM for calculating predicted values of the position and orientation of robot 10 and an external force model EM for calculating predicted values of external force applied to robot 10 are included.

The robot model LM is “based on” (=input) and “calculated” (=output) when the model execution unit 42 executes the model using input data. , to calculate (generate) output data by executing a model.

The external force model EM includes a modified external force model EM1 that outputs predicted values of corrected external forces and a hostile external force model EM2 that outputs predicted values of hostile external forces.

The reward calculation unit 43 calculates a reward based on the error between the position/orientation predicted value and the position/orientation target value to be reached and the external force predicted value. The position/orientation to be reached may be the position/orientation to be reached when the task is completed, or may be the position/orientation as an intermediate target before the task is completed.

When the external force is a corrected external force that suppresses the spread of the error, the remuneration calculation unit 43 calculates the remuneration by calculation using the predicted value of the corrected external force as a reduction factor of the remuneration.

A corrective external force, which is an external force that suppresses the expansion of an error, is an external force that slows down the speed of expansion when the position and orientation error expands, not an external force that turns the expansion of the error into a contraction. good too.

Calculation using the predicted value of the corrected external force as a factor for decreasing the reward means that the reward calculated using the predicted value of the corrected external force calculated by the external force model is higher than the reward when the calculated value is calculated with the predicted value of the corrected external force set to 0. means less reward. Decrease does not mean temporal decrease, and even if the predicted value of the corrected external force calculated by the corrected external force model EM1 is used for calculation, the reward does not necessarily decrease with the lapse of time.

Even if the position and orientation error increases, it is preferable to apply the correction external force when the position and orientation error is large (for example, when the peg 70 is far away from the hole 74 in FIG. 3). is small (when the peg 70 is near the hole 74), no correction external force need be applied. Further, when the position/orientation error is large, it is preferable to apply a larger correction external force as the speed at which the position/orientation error increases increases.

In addition, the reward calculation unit 43 calculates the reward by performing a calculation such that the amount of change in the amount of decrease in the reward based on the predicted value of the corrected external force during task execution is smaller than the width of change in the reward based on the error.

Further, when the external force is a hostile external force that suppresses the reduction of the error, the remuneration calculation unit 43 calculates the remuneration by calculation using the predicted value of the hostile external force as a factor for increasing the remuneration.

A hostile external force, which is an external force that suppresses the reduction of the error, is an external force that slows down the speed of reduction when the position and orientation error is reduced, not an external force that turns the reduction of the error into an increase. good too.

Calculations that use the predicted value of hostile external force as a factor to increase the reward are those calculated using the predicted value of hostile external force calculated by the hostile external force model compared to the reward when the predicted value of hostile external force is set to 0. This means that the reward for Note that the increase does not mean a temporal increase, and even if the predicted value of the hostile external force calculated by the hostile external force model EM2 is used for calculation, the reward does not necessarily increase over time.

Even if the position and orientation error is reduced, it is preferable to apply the hostile external force when the position and orientation error is small (for example, when the peg 70 is near the hole 74 in FIG. 3), and when the position and orientation error is large. does not have to apply a hostile external force. Further, when the position and orientation error is small, it is preferable to apply a larger hostile external force as the speed of reduction of the position and orientation error increases.

Further, the reward calculation unit 43 calculates the reward by performing a calculation such that the amount of change in the amount of increase in the reward based on the predicted value of the hostile external force during execution of the task is smaller than the width of change in the reward based on the error.

The action determining unit 44 generates a plurality of action command candidates for each control cycle and gives them to the robot model LM. Determine the behavioral directive that maximizes reward.

The action command is a speed command in this embodiment, but may be a position command, a torque command, a combination command of speed, position, torque, or the like. Also, the action command may be a sequence of action commands over multiple times. Also, the plurality of candidates for action commands may be candidates for a plurality of series of action commands.

Maximizing the reward only needs to be maximized as a result of searching within a limited time, and the reward does not have to be the true maximum value in the situation.

The external force model updating unit 45 updates the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action command and the actual value of the external force corresponding to the predicted value of the external force becomes small. to update.

The external force model updater 45 includes a corrected external force model updater 45A and a hostile external force model updater 45B.

The corrected external force model updating unit 45A corrects so that the difference between the predicted value of the corrected external force calculated by the corrected external force model EM1 based on the action command determined by the action determination unit 44 and the actual value of the corrected external force becomes smaller. Update the external force model EM1.

The hostile external force model updating unit 45B adjusts the hostile force so that the difference between the predicted value of the hostile external force calculated by the hostile external force model EM2 based on the action command determined by the action determining unit 44 and the actual value of the hostile external force becomes small. Update the external force model EM2.

The learning control unit 46 determines whether the external force is the corrected external force or the hostile external force based on the actual values of the position and orientation and the actual values of the external force. The operation is validated, and if the determination result is hostile external force, the operation of the hostile external force model updating unit 45B is validated. Furthermore, if the determination result is not the corrected external force, the operation of the corrected external force model updating unit 45A is invalidated, and if the determination result is not the hostile external force, the operation of the hostile external force model updating unit 45B is invalidated.

In this embodiment, the learning control unit 46 automatically determines whether the external force is a corrected external force or a hostile external force based on the actual values of the position and orientation and the actual values of the external force. The unit 46 may further include a receiving unit that receives designation as to whether the external force is the corrected external force or the hostile external force.

In this case, the user operates the input device 60 to specify whether the external force applied to the gripper 12 is a corrected external force or a hostile external force, and applies the external force to the gripper 12 .

Then, the learning control unit 46 validates the operation of the corrected external force model updating unit 45A when the designation is the corrected external force, and validates the operation of the hostile external force model updating unit 45B when the designation is the hostile external force. Furthermore, if the specified external force is not corrected, the operation of the corrected external force model updating unit 45A is invalidated, and if the specified external force is not the hostile external force, the operation of the hostile external force model updating unit 45B is invalidated.

In the example of FIG. 5, the state transition model DM, the corrected external force model EM1, and the hostile external force model EM2 are independent models, but the configuration of the robot model LM is not limited to this. For example, like the robot model LM1 shown in FIG. 6, it may be composed of a common part CM, a state transition model specific part DMa, a corrected external force model specific part EM1a, and an enemy external force model specific part EM2a. In this case, the common part CM performs common processing for the state transition model DM, the corrected external force model EM1, and the hostile external force model EM2. The corrected external force model specific part EM1a performs processing specific to the corrected external force model EM1. The hostile external force model specific part EM2a performs processing specific to the hostile external force model EM2.

Also, like the robot model LM2 shown in FIG. 7, the configuration may include an integrated external force model IM including the corrected external force model EM1 and the hostile external force model EM2 shown in FIG. In this case, the integrated external force model IM outputs a predicted value of the external force and also outputs identification information for identifying whether the external force is a corrected external force or a hostile external force.

The integrated external force model IM may be obtained by integrating the corrected external force model EM1 and the hostile external force model EM2 by a progressive neural network technique. In this case, the modified external force model EM1 and the hostile external force model are configured by neural networks. Then, at least one of the one or more intermediate layers and the output layer of the hostile external force model EM2 converts the output of the layer preceding the corresponding layer of the modified external force model EM1 into a progressive neural network (PNN). integrated by the method of

In the example of FIG. 8, the output layer OUT2 of the hostile external force model EM2 is supplied with the output of the intermediate layer MID2A, which is the preceding layer of the corresponding output layer OUT1 of the corrected external force model EM1. Also, the intermediate layer MID2B of the hostile external force model EM2 receives the output of the intermediate layer MID1A, which is the layer preceding the corresponding intermediate layer MID2A of the modified external force model EM1.

Machine learning of the modified external force model EM1 is first performed on such integrated external force model IM, and then machine learning of the hostile external force model EM2 is performed. While learning the modified external force model EM1 by applying the modified external force, update the weight parameter between each layer of the modified external force model EM1 so that the error of the predicted value of the modified external force with respect to the actual value of the modified external force becomes small, The hostile external force model EM2 is not updated. Same as the weight parameter of the path from one layer (for example, MID1A) of the modified external force model EM1 to the next layer (MID2A) The weight parameter of the path from the layer (MID1A) to the layer (MID2B) of the hostile external force model is always the same value. One layer of the hostile external force model EM2 (e.g., MID2B) is composed of the weighted inputs from the modified external force model layer (e.g., MID1A) to that layer and the weighted inputs from the preceding layer (MID1B) of the hostile external force model EM2. The sum with the input is taken as the output to the subsequent layer (OUT2) of the hostile external force model EM2. After the learning of the modified external force model EM1 is completed, while the hostile external force is added and the hostile external force model EM2 is being learned, the hostile external force model is modified so that the error in the predicted value of the hostile external force with respect to the actual value of the hostile external force is reduced. The weight parameter between each layer of EM2 is updated, but the modified external force model EM1 is not updated. In the operation phase after the end of the learning of the hostile external force model EM2, the output of the hostile external force model EM2 is used as the external force prediction value, and the output of the corrected external force model EM1 is not used. By machine-learning the external force in this way, the model that integrates the corrected external force model EM1 and the hostile external force model EM2 can learn the hostile external force without destroying the learning result of the corrected external force performed previously. It can be carried out.

The integrated external force model IM identifies whether the predicted value of the external force is the predicted value of the corrected external force or the predicted value of the hostile external force by an identification unit (not shown), and outputs the identification result as identification information. In this case, if the identification information indicates that the predicted value of the modified external force , the reward is calculated using the predicted value of the external force as a factor for increasing the reward.

It should be noted that the progressive neural network technique refers to, for example, the technique described in the following references.

(Reference) Rusu et al., “Progressive neural networks,” arXiv preprint arXiv:1606.04671, 2016.

In addition, there are the following reference articles regarding progressive neural networks.

(Reference article) Continuous learning in multiple games
https://wba-initiative.org/wp-content/uploads/2015/05/20161008-hack2-noguchi.pdf

(Learning process of robot model)

FIG. 9 is a flow chart showing the flow of machine learning processing for learning the robot model LM using machine learning. The machine learning process of FIG. 9 is executed by the CPU 40A reading a machine learning program stored in the ROM 40B or the storage 40D, developing it in the RAM 40C.

The processes of steps S100 to S108, which will be described below, are executed at regular time intervals according to the control cycle. The control cycle is set to a time during which the processes of steps S100 to S108 can be executed.

In step S100, the CPU 40A waits until a predetermined time corresponding to the length of the control cycle has elapsed since the previous control cycle was started. It should be noted that the process of step S100 may be omitted, and the process of the next control period may be started immediately after the process of the previous control period is completed.

In step S101, the CPU 40A acquires actual values (measured values) of the position and orientation of the robot 10 from the state observation sensor 20, and acquires actual values (measured values) of the external force from the tactile sensors 30A and 30B.

In step S102, the CPU 40A, as the acquiring unit 41, determines whether or not the actual values of the position and orientation acquired in step S101 satisfy a predetermined end condition. Here, the case where the end condition is satisfied is, for example, the case where the error between the actual value of the position and orientation and the target value of the position and orientation to be reached is within a specified value. The position and orientation to be reached is the position and orientation of the robot 10 when the robot 10 can insert the peg 70 into the hole 74 in this embodiment.

If the determination in step S102 is affirmative, the routine ends. On the other hand, if the determination in step S102 is negative, the process proceeds to step S103.

In step S103, the CPU 40A, as the external force model updating unit 45, updates the external force model EM. Specifically, first, it is determined whether the external force is a corrected external force or a hostile external force based on the actual values of the position and orientation and the actual values of the external force acquired in step S101. For example, when the error between the position/orientation actual value and the position/orientation target value to be reached is increasing, an external force detected as a force in a direction that suppresses the increase of the error is determined as a corrected external force, and the error An external force detected as a force in a direction that suppresses the reduction of the error when is shrinking can be discriminated as a hostile external force, but the discriminating method is not limited to this.

When the determined external force is the corrected external force, the difference between the predicted value of the corrected external force calculated by the corrected external force model EM1 based on the determined action command and the actual value of the corrected external force is reduced. The corrected external force model parameters of the corrected external force model EM1 are updated.

On the other hand, when the determined external force is a hostile external force, the difference between the predicted value of the hostile external force calculated by the hostile external force model EM2 based on the determined action command and the actual value of the hostile external force is reduced. Update the hostile external force model parameters of the hostile external force model EM2.

In step S<b>104 , the CPU 40</b>A, as the action determination unit 44 , generates a plurality of candidates for the action command (or action command series) for the robot 10 . In this embodiment, for example, n (eg, 300) speed command value candidates are randomly generated and output to the model execution unit 42 as action command candidate values.

In step S105, the CPU 40A, as the model execution unit 42, calculates predicted values of the position and orientation and predicted values of the external force for each of the plurality of action command candidates generated in step S104. Specifically, actual position and orientation values and n candidate values for action commands are input to the robot model LM. Calculate the predicted value of the external force.

In step S106, the CPU 40A, as the reward calculation unit 43, calculates a reward for each set of the position/orientation predicted value and the corrected external force corresponding to the n candidate values of the action command. That is, n rewards are calculated.

The reward r1 when the external force is the corrected external force can be calculated using the following formula (1).

・・・（１）

... (1)

Here, r ^R is the error between the position/orientation predicted value and the position/orientation target value to be reached. s1 ^H is the corrected external force. α1 is a weight and is set in advance. α1 is the width of change in the amount of decrease in reward r1 based on the predicted value of the corrected external force during task execution, and the width of change in reward r1 based on the error between the predicted value of the position and orientation and the target value of the position and orientation to be reached. is set to be less than

On the other hand, the reward r2 when the external force is a hostile external force can be calculated using the following equation (2).

・・・（２）

... (2)

where s2 ^H is the hostile external force. α2 is a weight and is set in advance. α2 is the width of change in the amount of increase in reward r2 based on the predicted value of hostile external force during task execution, and the width of change in reward r2 based on the error between the predicted value of position and orientation and the target value of position and orientation to be reached. is set to be less than

As shown in the above formulas (1) and (2), when the external force is the same, the larger the error between the position/orientation predicted value and the target position/orientation value to be reached, the smaller the reward. Also, as shown in the above formula (1), when the error is the same, the larger the corrected external force, the smaller the reward. Also, as shown in the above formula (2), when the error is the same, the greater the hostile external force, the greater the reward.

In step S<b>107 , the CPU 40</b>A, acting as the action determination unit 44 , determines an action command for maximizing the reward and outputs it to the robot 10 . For example, a relational expression representing the correspondence between n candidate values of action commands and rewards is calculated, and the candidate value of the action command corresponding to the maximum reward on the curve represented by the calculated relational expression is taken as the determined value. do. A so-called cross-entropy method (CEM) may also be used to identify behavioral commands that can maximize the reward. This provides a behavioral command that maximizes the reward.

Steps S104 to S106 may be repeated a predetermined number of times. In this case, the CPU 40A, as the action determination unit 44, after executing step S106 for the first time, extracts m candidate values of the action command with the highest reward from n pairs of the candidate values of the action command and the reward. Then, the average and variance of m candidate values for the action command are obtained, and a normal distribution is generated according to them. In the second step S104, the CPU 40A, as the action determining unit 44, generates n new speed command candidate values not randomly but so that the probability density matches the obtained normal distribution. Similarly, steps S104 to S106 are executed a predetermined number of times. In this way, it is possible to increase the accuracy of maximizing the reward.

The robot 10 operates according to the determined value of the action command. A user applies an external force to the robot 10 according to the motion of the robot 10 . Specifically, an external force is applied to the gripper 12 . The user applies a correction external force to the robot 10 when the error between the predicted values of the position and orientation and the target values of the position and orientation to be reached is increasing, and when the error is decreasing, the robot 10 It is preferable to apply a hostile external force against it. That is, for example, when the peg 70 is moving away from the hole 74 due to the operation of the robot 10 , the user applies a correcting external force to the gripper 12 in the direction in which the peg 70 approaches the hole 74 . Also, for example, when the peg 70 is moving toward the hole 74 due to the motion of the robot 10 , the peg 70 applies a hostile external force to the gripper 12 in the direction away from the hole 74 .

In addition, in the process of machine-learning the external force model, it is preferable to apply the corrected external force first. This is because applying the hostile external force first may slow down learning. Also, the ratio of adding the corrected external force and the hostile external force may be 1:1, or the ratio of the corrected external force may be increased. As for the order in which the corrected external force and the hostile external force are applied, the corrected external force may be applied a plurality of times and then the hostile external force may be applied a plurality of times, or the corrected external force and the hostile external force may be applied alternately.

Further, the corrected external force or the hostile external force may be automatically applied by a robot or the like that applies the external force instead of being applied by a human being.

In step S108, the CPU 40A, as the model execution unit 42, calculates the predicted value of the external force for the determined value of the action command determined in step S107, and returns to step S100.

In this manner, the processing of steps S100 to S108 is repeated in each control cycle until the actual values of the position and orientation satisfy the termination condition.

Thereby, the robot model LM is learned. In this way, the robot model LM includes the corrected external force model EM1 and the hostile external force model EM2, and the user learns the robot model LM while applying the corrected external force or the hostile external force to the robot 10. Therefore, the robot model LM can be learned efficiently. It is possible to obtain a robot model LM excellent in robustness against environmental changes such as changes in the shape and material of parts to be operated by the robot 10 and changes in the physical characteristics of the robot 10 over time.

Note that in the operation phase, the model execution unit 42 executes the robot model LM that has been learned through the learning process of FIG. The functional configuration of the robot control device 40 in the operation phase is a configuration in which the external force model updating unit 45 and the learning control unit 46 are omitted from the functional configuration of FIG. The robot control processing in the operation phase is the learning processing of FIG. is the robot control processing program.

The device that executes the robot model learning process in the learning phase and the device that executes the robot control process in the operation phase may be separate devices or may be the same device. For example, the learning device used for learning may be used as it is as the robot control device 40 to perform control using the learned robot model LM. Further, the robot control device 40 may perform control while continuing learning.

<Modified Example of First Embodiment>
In the first embodiment, the state transition model DM is configured such that the actual values of the position and orientation and the action command are input, but the actual values of the external force are not input. Instead of this, the state transition model DM may be configured to input the actual value of the external force. In this case, the state transition model DM calculates predicted values of the position and orientation based on the actual values of the position and orientation, the action command, and the actual values of the external force. However, the correction external force or the hostile external force is applied to the tactile sensors 30A and 30B only during the machine learning of the external force models EM1, EM2, EM1a, EM2a and IM. In the operation phase, the state transition model DM calculates the predicted values of the position and orientation while the state in which the input actual value of the external force is substantially zero continues. On the other hand, the fact that the external force model calculates the predicted value of the external force from the actual value of the position and orientation and the action command without inputting the actual value of the external force is the same in this modified example. Predicted values of external forces influence behavioral decisions through being used in reward calculations. Modifications similar to this modification can also be made in subsequent embodiments.

<Second embodiment>

Next, a second embodiment of the disclosed technology will be described. In addition, the same code|symbol is attached|subjected to the same part as 1st Embodiment, and detailed description is abbreviate|omitted.

Since the robot system 1 according to the second embodiment is the same as that of the first embodiment, description thereof is omitted.

(Learning process of robot model)

FIG. 10 is a flowchart showing the flow of machine learning processing according to the second embodiment. The machine learning process of FIG. 10 is executed by the CPU 40A reading a machine learning program stored in the ROM 40B or the storage 40D, developing it in the RAM 40C.

Since the processing of steps S100 to S103 and S108 is the same as the processing of FIG. 9, the description thereof is omitted.

In step S<b>104</b>A, the CPU 40</b>A, as the action determination unit 44 , generates one candidate for the action command (or action command series) for the robot 10 .

In step S106A, the CPU 40A, as the model execution unit 42, calculates the predicted value of the position and orientation and the predicted value of the external force for one candidate of the action command generated in step S104A. Specifically, the actual values of the position and orientation and the candidate values of the action command are input to the robot model LM, and the predicted values of the position and orientation corresponding to the candidate values of the action command and the predicted values of the modified external force or the predicted values of the hostile external force are input. Calculate

In step S106A, the CPU 40A, as the reward calculation unit 43, calculates a reward based on the set of the position/orientation predicted value and the external force predicted value corresponding to the candidate value of the action command. That is, when the external force is the corrected external force, the reward r1 is calculated by the above formula (1), and when the external force is the hostile external force, the reward r2 is calculated by the above formula (2).

In step S106B, CPU 40A determines whether or not the remuneration calculated in step S106A satisfies prescribed conditions. Here, the case where the prescribed condition is satisfied is, for example, the case where the remuneration exceeds the prescribed value, or the case where the processing loop of steps S104A to S106B is executed a prescribed number of times. The specified number of times is set to, for example, 10 times, 100 times, 1000 times, or the like.

In step S<b>107</b>A, the CPU 40</b>A, acting as the action determination unit 44 , determines an action command for maximizing the reward and outputs it to the robot 10 . For example, the action command itself may be used when the reward satisfies a prescribed condition, or it may be an action command predicted from the history of changes in reward corresponding to changes in the action command and capable of maximizing the reward.

<Third Embodiment>

Next, a third embodiment of the disclosed technology will be described. In addition, the same code|symbol is attached|subjected to the same part as 1st Embodiment, and detailed description is abbreviate|omitted.

(robot controller)

FIG. 11 shows the functional configuration of a robot control device 40X according to the third embodiment. The robot controller 40X differs from the robot controller 40 shown in FIG. 1 in that it includes a storage unit 48 and a state transition model update unit 49. Since other configurations are the same as those of the robot control device 40, description thereof will be omitted.

The storage unit 48 stores the actual values of the position and orientation of the robot 10 acquired by the acquisition unit 41 .

The state transition model update unit 49 updates the predicted values of the position and orientation calculated by the state transition model DM based on the action command determined by the action determination unit 44 and the actual values of the position and orientation corresponding to the predicted values of the position and orientation. The state transition model DM is updated so that the error between is reduced.

(Learning process of robot model)

FIG. 12 is a flowchart showing the flow of machine learning processing according to the third embodiment. The machine learning process of FIG. 12 is executed by the CPU 40A reading a machine learning program stored in the ROM 40B or the storage 40D, developing it in the RAM 40C.

The learning process of FIG. 12 differs from the learning apparatus of FIG. 9 in that the processes of steps S101A and S103A are added. Since other steps are the same as the processing in FIG. 9, description thereof is omitted.

In step S101A, the CPU 40A, as the acquisition unit 41, causes the storage unit 48 to store the actual values of the position and orientation of the robot 10 acquired in step S101.

At step S103A, the CPU 40A, as the state transition model updating unit 49, updates the state transition model DM. Specifically, first, for example, 100 position and orientation actual values x _t at times t randomly selected from among those stored in the storage unit 48, speed command values u _t as action commands, and values for time t+1. Obtain a set of position and orientation actual values x _t+1 . Next, new state transition model parameters are determined by modifying the previous state transition model parameters. The state transition model parameters are corrected with the goal of minimizing the error between the predicted values of the position and orientation at time t+1 calculated from the actual values of the position and orientation at time t and the actual values of the position and orientation at time t+1.

Then, new state transition model parameters are set in the state transition model DM. The new state transition model parameters are stored in the state transition model updating unit 49 for use as "previous model parameters" in the next control cycle.

Thus, in this embodiment, the state transition model DM can be learned together with the modified external force model EM1 and the hostile external force model EM2.

<Experimental example>

Next, an experimental example of the disclosed technique will be described.

FIG. 13 shows the result of learning a robot model by performing a task of inserting a peg into a hole while applying corrective external force and hostile external force to the robot through simulation. In this simulation, the robot model was learned by applying the corrected external force seven times and then applying the hostile external force seven times.

The horizontal axes in FIGS. 13A and 13B represent the number of times the external force is applied. The vertical axis of FIG. 13(A) represents the peg position error. The vertical axis of FIG. 13B represents the magnitude of the applied external force.

FIG. 13(A) shows the result of learning only the state transition model without applying external force by the conventional method (Baseline), and the corrected external force model and the hostile external force model by the corrected external force and hostile external force applied by subject 1 by the proposed method. The result of learning the robot model including the modified external force model and the hostile external force model by the modified external force and the hostile external force applied by the subject 2 different from the subject 1 by the proposed method (Proposed (participant 1)). (participant 2)). As shown in FIG. 13(A), when the positional error at the end of learning is compared between the conventional method and the proposed method (subject 1 and subject 2), the positional error is smaller in the proposed method than in the conventional method. I understand. Moreover, as shown in FIG. 13B, although the method of applying the external force differs between the subjects 1 and 2, it was found that the positional error was small regardless of the subject.

FIG. 14(A) shows the results of simulating the number of successful insertions of pegs by changing the coefficient of friction of the table provided with holes. FIG. 14(B) shows the results of simulating the number of successful peg insertions by changing the mass of the peg. As shown in FIGS. 14A and 14B, even when the coefficient of friction of the table and the mass of the peg are different, the number of successful peg insertions is greater in the proposed method than in the conventional method.

Further, FIG. 15 shows the result of performing a peg insertion task similar to the simulation on an actual machine using pegs made of different materials. There are three types of peg materials: metal (default), plastic, and sponge. As shown in FIG. 15, it was found that the number of successful insertions of the pegs by the proposed method is greater than that of the conventional method, regardless of the material of the pegs.

As can be seen from the configuration and operation of the above-described embodiment and the above-described experimental example, the efficiency of the machine learning of the robot model can be improved by performing the machine learning by applying the correction external force. In addition, by performing machine learning by applying a hostile external force, it is possible to improve robustness against changes in the frictional force and mass of the object to be grasped. In addition, machine learning by adding hostile external force also has the effect of increasing learning efficiency.

It should be noted that the above-described embodiment merely exemplifies a configuration example of the present invention. The disclosed technique is not limited to the above-described specific forms, and various modifications are possible within the scope of the technical idea.

For example, in the above embodiment, the peg fitting work was described as an example, but the work to be learned and controlled may be any work.

Further, various processors other than the CPU may execute the robot model learning process and the robot control process executed by the CPU by reading the software (program) in each of the above embodiments. The processor in this case is a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing such as an FPGA (Field-Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit) for executing specific processing. A dedicated electric circuit or the like, which is a processor having a specially designed circuit configuration, is exemplified. Also, the learning process and the control process may be performed by one of these various processors, or by a combination of two or more processors of the same or different type (e.g., multiple FPGAs, and a CPU and an FPGA). , etc.). Further, the hardware structure of these various processors is, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined.

In each of the above-described embodiments, the robot model learning program and the robot control program have been pre-stored (installed) in the storage 40D or ROM 40B, but the present invention is not limited to this. The program may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory. Also, the program may be downloaded from an external device via a network.
The following additional remarks are disclosed regarding the above embodiments.
(Appendix 1)
an acquisition unit (41) for acquiring actual values of the position and orientation of the robot and actual values of the external force applied to the robot;
A state transition model (DM) for calculating predicted values of the position and orientation of the robot at the next time based on the actual values of the position and orientation at a certain time and an action command that can be given to the robot, and applied to the robot. a robot model (LM) including an external force model (EM) for calculating predicted values of external forces;
a model execution unit (42) for executing the robot model;
a reward calculation unit (43) for calculating a reward based on the error between the predicted value of the position and orientation and the target value of the position and orientation to be reached and the predicted value of the external force;
For each control cycle, a plurality of candidates for the action command are generated and given to the robot model, and a reward is maximized based on the reward calculated by the reward calculation unit corresponding to each of the plurality of candidates for the action command. an action determination unit (44) that determines an action command to
updating the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action command and the actual value of the external force corresponding to the predicted value of the external force is reduced; an external force model updating unit (45) for
A robot model learning device with
(Appendix 2)
The error between the predicted position/orientation calculated by the state transition model based on the determined action command and the actual position/orientation corresponding to the predicted position/orientation is reduced. a state transition model updating unit (49) for updating the state transition model;
The robot model learning device according to Supplementary Note 1, comprising:
(Appendix 3)
When the external force is a corrected external force that suppresses the expansion of the error, the remuneration calculation unit calculates the remuneration by calculation using a predicted value of the corrected external force as a factor for reducing the remuneration. The robot model learning device according to appendix 2.
(Appendix 4)
The reward calculation unit calculates the reward by calculating a range of change in the decrease amount of the reward based on the predicted value of the corrected external force during execution of the task so as to be smaller than a range of change in the decrease amount of the reward based on the error. The robot model learning device according to appendix 3.
(Appendix 5)
The external force model includes a modified external force model (EM1) that outputs a predicted value of the modified external force when the external force is the modified external force,
When the external force is the corrected external force, the external force model updating unit updates the predicted value of the corrected external force calculated by the corrected external force model based on the determined action command and the actual value of the corrected external force. The robot model learning device according to appendix 3 or appendix 4, further comprising a corrected external force model updating unit (45A) that updates the corrected external force model so that the difference between the models becomes smaller.
(Appendix 6)
When the external force is a hostile external force that suppresses reduction of the error, the remuneration calculation unit calculates the remuneration by calculation using the predicted value of the hostile external force as a factor for increasing the remuneration. The robot model learning device according to appendix 2.
(Appendix 7)
The reward calculation unit calculates the reward by calculating a range of change in the amount of increase in the reward based on the predicted value of the hostile external force during execution of the task so that the range of change in the amount of increase in the reward is smaller than a range of change in the amount of increase in the reward based on the error. The robot model learning device according to appendix 6.
(Appendix 8)
The external force model includes a hostile external force model (EM2) that outputs a predicted value of the hostile external force when the external force is the hostile external force,
When the external force is the hostile external force, the external force model update unit updates the predicted value of the hostile external force calculated by the hostile external force model based on the determined action command and the actual value of the hostile external force. The robot model learning device according to appendix 6 or appendix 7, further comprising a hostile external force model updating unit (45B) that updates the hostile external force model so that the difference between the models becomes smaller.
(Appendix 9)
When the external force is a corrected external force that suppresses the expansion of the error, the reward calculation unit calculates the reward by calculation using a predicted value of the corrected external force as a factor for reducing the reward, and the external force is the error The robot model learning device according to appendix 1 or appendix 2, wherein the reward is calculated using a predicted value of the hostile external force as a factor for increasing the reward when the reward is a hostile external force that suppresses a reduction in the remuneration.
(Appendix 10)
The reward calculation unit determines that the range of change in the decrease amount of the reward based on the predicted value of the corrected external force during task execution becomes smaller than the range of change in the reward based on the error, and the hostile external force during task execution. The robot model learning device according to Supplementary Note 9, wherein the reward is calculated by a calculation in which the width of change in the amount of increase in the reward based on the predicted value of is smaller than the width of change in the reward based on the error.
(Appendix 11)
The external force model includes a modified external force model (EM1) that outputs a predicted value of the modified external force when the external force is the modified external force, and a predicted value of the hostile external force when the external force is the hostile external force. and an adversarial external force model (EM2) that outputs
When the external force is the corrected external force, the external force model update unit determines whether the difference between the predicted value of the corrected external force calculated by the corrected external force model based on the determined action command and the actual value of the external force is a corrected external force model updating unit that updates the corrected external force model so that it becomes smaller; and a prediction of the hostile external force calculated by the hostile external force model based on the determined action command when the external force is the hostile external force. 11. The robot model learning device according to appendix 9 or 10, further comprising a hostile external force model updating unit (45B) that updates the hostile external force model so that the difference between the value and the actual value of the external force is reduced.
(Appendix 12)
the robot model includes an integrated external force model (IM) comprising the modified external force model and the hostile external force model;
The modified external force model and the hostile external force model are neural networks,
At least one layer of the one or more intermediate layers and the output layer of the hostile external force model integrates the output of the previous layer of the corresponding layer of the modified external force model by a progressive neural network technique,
The integrated external force model outputs the output of the hostile external force model as a predicted value of the external force,
The integrated external force model outputs identification information as to whether the predicted value of the external force to be output is the predicted value of the corrected external force or the predicted value of the hostile external force,
The remuneration calculation unit calculates the remuneration by calculation using the predicted value of the external force as a reduction factor of the remuneration when the identification information indicates that it is a predicted value of the corrected external force, and the identification information is: 12. The robot model learning device according to Supplementary Note 11, wherein the reward is calculated by calculation using the predicted value of the external force as a factor for increasing the reward when indicating that it is a predicted value of the hostile external force.
(Appendix 13)
further comprising a receiving unit that receives designation of whether the external force is the corrected external force or the hostile external force;
a learning control unit that validates the operation of the corrected external force model update unit when the designation is the corrected external force, and validates the operation of the hostile external force model update unit when the designation is the hostile external force; The robot model learning device according to Supplementary Note 11 or Supplementary Note 12.
(Appendix 14)
determining whether the external force is the modified external force or the hostile external force based on the actual values of the position and orientation and the actual values of the external force; and updating the modified external force model when the result of the determination is the modified external force. further comprising a learning control unit (46) for activating the operation of the robot model unit, and activating the operation of the hostile external force model update unit when the result of the determination is the hostile external force. learning device.
(Appendix 15)
In addition to a state transition model (DM) that calculates predicted values of the robot's position and orientation at the next time based on actual values of the robot's position and orientation at a certain time and action commands that can be given to the robot, and the robot. Prepare a robot model (LM) including an external force model (EM) that calculates the predicted value of the external force applied,
Acquiring the actual values of the position and orientation and the actual values of the external force applied to the robot for each control cycle,
A plurality of candidates for the action command are generated for each control cycle and given to the robot model, and a plurality of predicted values of the position and orientation calculated by the state transition model corresponding to the plurality of candidates for the action command, and Based on a plurality of predicted values of the external force calculated by the external force model corresponding to a plurality of errors between target values of the position and orientation to be reached and a plurality of candidates of the action command, a plurality of the action commands Based on the multiple rewards calculated for the candidates, determine the action command that maximizes the reward,
updating the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action command and the actual value of the external force corresponding to the predicted value of the external force is reduced; do,
Machine learning methods for robot models.
(Appendix 16)
Further, an error between the predicted value of the position and orientation calculated by the state transition model based on the determined action command and the actual value of the position and orientation corresponding to the predicted value of the position and orientation is reduced. 16. The robot model machine learning method according to appendix 15.
(Appendix 17)
The robot model according to appendix 15 or 16, wherein when the external force is a corrected external force that is an external force that suppresses the expansion of the error, the reward is calculated using a predicted value of the corrected external force as a factor for decreasing the reward. machine learning methods.
(Appendix 18)
The robot model according to Supplementary Note 17, wherein the reward is calculated by a calculation such that the width of change in the decrease amount of the reward based on the predicted value of the corrected external force during task execution is smaller than the width of change in the reward based on the error. machine learning method.
(Appendix 19)
The external force model includes a modified external force model (EM1) that outputs a predicted value of the modified external force when the external force is the modified external force,
When the external force is the corrected external force, the difference between the predicted value of the corrected external force calculated by the corrected external force model based on the determined action command and the actual value of the external force is reduced. 19. The robot model machine learning method according to appendix 17 or appendix 18, wherein the modified external force model is updated.
(Appendix 20)
20. The robot model machine learning method according to appendix 19, wherein the correction external force is applied to the robot when the error is increasing.
(Appendix 21)
When the external force is a hostile external force that suppresses reduction of the error, the reward is calculated by calculation using a predicted value of the hostile external force as a factor for increasing the reward. machine learning methods.
(Appendix 22)
The robot model according to Supplementary Note 21, wherein the amount of change in the amount of increase in the reward based on the predicted value of the hostile external force during task execution is smaller than the range of change in the amount of increase in the reward based on the error. machine learning method.
(Appendix 23)
The external force model includes a hostile external force model (EM2) that outputs a predicted value of the hostile external force when the external force is the hostile external force,
When the external force is the hostile external force, the difference between the predicted value of the hostile external force calculated by the hostile external force model based on the determined action command and the actual value of the external force is reduced. 23. The robot model machine learning method according to appendix 21 or appendix 22, wherein the hostile external force model is updated.
(Appendix 24)
24. The robot model machine learning method according to appendix 23, wherein the hostile external force is applied to the robot when the error is shrinking.
(Appendix 25)
When the external force is a corrected external force that suppresses the expansion of the error, the reward is calculated by calculation using the predicted value of the corrected external force as a factor for reducing the reward, and the external force suppresses the reduction of the error. 17. The robot model machine learning method according to appendix 15 or 16, wherein the reward is calculated using a predicted value of the hostile external force as an increasing factor of the reward.
(Appendix 26)
The range of change in the amount of decrease in the reward based on the predicted value of the corrected external force during task execution is smaller than the range of change in the reward based on the error, and the range of change in the reward based on the predicted value of the hostile external force during task execution. 26. The robot model machine learning method according to appendix 25, wherein the reward is calculated by a calculation in which the width of change in the amount of increase in reward is smaller than the width of change in the reward based on the error.
(Appendix 27)
The external force model includes a modified external force model (EM1) that outputs a predicted value of the modified external force when the external force is the modified external force, and a predicted value of the hostile external force when the external force is the hostile external force. and an adversarial external force model (EM2) that outputs
When the external force is the corrected external force, the corrected external force is adjusted so that the difference between the predicted value of the corrected external force calculated by the corrected external force model based on the determined action command and the actual value of the external force becomes small. The model is updated so that, when the external force is the hostile external force, the difference between the predicted value of the hostile external force calculated by the hostile external force model based on the determined action command and the actual value of the external force becomes smaller. 27. The robot model machine learning method according to appendix 25 or appendix 26.
(Appendix 28)
Machine learning of the robot model according to Appendix 27, applying the corrective external force to the robot when the error is increasing, and applying the hostile external force to the robot when the error is decreasing. Method.
(Appendix 29)
In addition to a state transition model (DM) that calculates predicted values of the robot's position and orientation at the next time based on actual values of the robot's position and orientation at a certain time and action commands that can be given to the robot, and the robot. A machine learning program for machine learning a robot model (LM) including an external force model (EM) that calculates a predicted value of an external force applied,
Acquiring the actual values of the position and orientation and the actual values of the external force applied to the robot for each control cycle,
A plurality of candidates for the action command are generated for each control cycle and given to the robot model, and a plurality of predicted values of the position and orientation calculated by the state transition model corresponding to the plurality of candidates for the action command, and Based on a plurality of predicted values of the external force calculated by the external force model corresponding to a plurality of errors between target values of the position and orientation to be reached and a plurality of candidates of the action command, a plurality of the action commands Based on the multiple rewards calculated for the candidates, determine the action command that maximizes the reward,
updating the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action command and the actual value of the external force corresponding to the predicted value of the external force is reduced; do,
A robot model machine learning program that causes a computer to perform each process.
(Appendix 30)
In addition to a state transition model (DM) that calculates predicted values of the robot's position and orientation at the next time based on actual values of the robot's position and orientation at a certain time and action commands that can be given to the robot, and the robot. a model execution unit (42) that executes a robot model (LM) including an external force model (EM) that calculates a predicted value of the external force applied;
an acquisition unit (41) for acquiring actual values of the position and orientation of the robot and actual values of the external force applied to the robot;
A reward calculation unit (43) for calculating a reward based on the error between the predicted value of the position and orientation calculated by the robot model and the target value of the position and orientation to be reached, and the predicted value of the external force calculated by the robot model. )When,
For each control cycle, a plurality of candidates for the action command are generated and given to the robot model, and a reward is maximized based on the reward calculated by the reward calculation unit corresponding to each of the plurality of candidates for the action command. an action determination unit (44) that determines an action command to
A robot controller with
(Appendix 31)
In addition to a state transition model (DM) that calculates predicted values of the robot's position and orientation at the next time based on actual values of the robot's position and orientation at a certain time and action commands that can be given to the robot, and the robot. Prepare a robot model (LM) including an external force model (EM) that calculates the predicted value of the external force applied,
Acquiring the actual values of the position and orientation and the actual values of the external force applied to the robot for each control cycle,
A plurality of candidates for the action command are generated for each control cycle and given to the robot model, and a plurality of predicted values of the position and orientation calculated by the state transition model corresponding to the plurality of candidates for the action command, and Based on a plurality of predicted values of the external force calculated by the external force model corresponding to a plurality of errors between target values of the position and orientation to be reached and a plurality of candidates of the action command, a plurality of the action commands Based on the multiple rewards calculated for the candidates, determine the action command that maximizes the reward,
controlling the robot based on the determined action command;
Robot control method.
(Appendix 32)
In addition to a state transition model (DM) that calculates predicted values of the robot's position and orientation at the next time based on actual values of the robot's position and orientation at a certain time and action commands that can be given to the robot, and the robot. A program for controlling the robot using a robot model (LM) including an external force model (EM) that calculates a predicted value of the external force applied,
Acquiring the actual values of the position and orientation and the actual values of the external force applied to the robot for each control cycle,
A plurality of candidates for the action command are generated for each control cycle and given to the robot model, and a plurality of predicted values of the position and orientation calculated by the state transition model corresponding to the plurality of candidates for the action command, and Based on a plurality of predicted values of the external force calculated by the external force model corresponding to a plurality of errors between target values of the position and orientation to be reached and a plurality of candidates of the action command, a plurality of the action commands Based on the multiple rewards calculated for the candidates, determine the action command that maximizes the reward,
controlling the robot based on the determined action command;
A robot control program that causes a computer to perform each process.

1 robot system 10 robot 20 state observation sensors 30A, 30B tactile sensor 40 robot control device 41 acquisition unit 42 model execution unit 43 reward calculation unit 44 action determination unit 45 external force model update unit 45A hostile external force model update unit 45B corrected external force model update unit 49 State transition model update unit 70 Peg LM Robot model

Claims (21)

an acquisition unit that acquires the actual values of the position and orientation of the robot and the actual values of the external force applied to the robot;
A state transition model that calculates a predicted value of the position and orientation of the robot at the next time based on the actual values of the position and orientation at a certain time and an action command that can be given to the robot, and a prediction of the external force applied to the robot. a robot model including an external force model for calculating a value;
a model execution unit that executes the robot model;
a reward calculation unit that calculates a reward based on the error between the predicted value of the position and orientation and the target value of the position and orientation to be reached and the predicted value of the external force;
For each control cycle, a plurality of candidates for the action command are generated and given to the robot model, and a reward is maximized based on the reward calculated by the reward calculation unit corresponding to each of the plurality of candidates for the action command. an action determination unit that determines an action command to
updating the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action command and the actual value of the external force corresponding to the predicted value of the external force is reduced; an external force model updating unit that
A robot model learning device with The error between the predicted position/orientation calculated by the state transition model based on the determined action command and the actual position/orientation corresponding to the predicted position/orientation is reduced. a state transition model updating unit that updates the state transition model;
The robot model learning device according to claim 1, comprising: 2. When the external force is a corrected external force that suppresses the expansion of the error, the remuneration calculation unit calculates the remuneration by calculation using a predicted value of the corrected external force as a reduction factor of the remuneration. 3. The robot model learning device according to claim 2. 2. When the external force is a hostile external force that suppresses reduction of the error, the remuneration calculation unit calculates the remuneration by calculation using a predicted value of the hostile external force as an increase factor of the remuneration. 3. The robot model learning device according to claim 2. When the external force is a corrected external force that suppresses the expansion of the error, the reward calculation unit calculates the reward by calculation using a predicted value of the corrected external force as a factor for reducing the reward, and the external force is the error 3. The robot model learning device according to claim 1, wherein the reward is calculated using a predicted value of the hostile external force as an increase factor of the reward when the external force is a hostile external force that suppresses the reduction of the . The reward calculation unit determines that the range of change in the decrease amount of the reward based on the predicted value of the corrected external force during task execution becomes smaller than the range of change in the reward based on the error, and the hostile external force during task execution. 6. The robot model learning device according to claim 5, wherein the reward is calculated by a calculation in which the width of change in the amount of increase in the reward based on the predicted value of is smaller than the width of change in the reward based on the error. The external force model includes a corrected external force model that outputs a predicted value of the corrected external force when the external force is the corrected external force, and a predicted value of the hostile external force when the external force is the hostile external force. including a hostile force model and
When the external force is the corrected external force, the external force model update unit determines whether the difference between the predicted value of the corrected external force calculated by the corrected external force model based on the determined action command and the actual value of the external force is a corrected external force model updating unit that updates the corrected external force model so that it becomes smaller; and a prediction of the hostile external force calculated by the hostile external force model based on the determined action command when the external force is the hostile external force. 7. The robot model learning device according to claim 5, further comprising a hostile external force model updating unit that updates the hostile external force model so that a difference between the value and the actual value of the external force becomes smaller. The robot model includes an integrated external force model comprising the modified external force model and the hostile external force model;
The modified external force model and the hostile external force model are neural networks,
At least one layer of the one or more intermediate layers and the output layer of the hostile external force model integrates the output of the previous layer of the corresponding layer of the modified external force model by a progressive neural network technique,
The integrated external force model outputs the output of the hostile external force model as a predicted value of the external force,
The integrated external force model outputs identification information as to whether the predicted value of the external force to be output is the predicted value of the corrected external force or the predicted value of the hostile external force,
The remuneration calculation unit calculates the remuneration by calculation using the predicted value of the external force as a reduction factor of the remuneration when the identification information indicates that it is a predicted value of the corrected external force, and the identification information is: 8. The robot model learning device according to claim 7, wherein when indicating that it is a predicted value of a hostile external force, the reward is calculated by calculation using the predicted value of the external force as a factor for increasing the reward. further comprising a receiving unit that receives designation of whether the external force is the corrected external force or the hostile external force;
a learning control unit that validates the operation of the corrected external force model update unit when the designation is the corrected external force, and validates the operation of the hostile external force model update unit when the designation is the hostile external force; The robot model learning device according to claim 7 or 8. determining whether the external force is the modified external force or the hostile external force based on the actual values of the position and orientation and the actual values of the external force; and updating the modified external force model when the result of the determination is the modified external force. 9. The robot model learning according to claim 7 or 8, further comprising a learning control unit that validates the operation of the unit and validates the operation of the hostile external force model update unit when the determination result is the hostile external force. Device. A state transition model that calculates a predicted value of the robot's position and orientation at the next time based on the actual values of the robot's position and orientation at a certain time and an action command that can be given to the robot, and an external force applied to the robot. Prepare a robot model that includes an external force model that calculates the predicted value,
Acquiring the actual values of the position and orientation and the actual values of the external force applied to the robot for each control cycle,
A plurality of candidates for the action command are generated for each control cycle and given to the robot model, and a plurality of predicted values of the position and orientation calculated by the state transition model corresponding to the plurality of candidates for the action command, and Based on a plurality of predicted values of the external force calculated by the external force model corresponding to a plurality of errors between target values of the position and orientation to be reached and a plurality of candidates of the action command, a plurality of the action commands Based on the multiple rewards calculated for the candidates, determine the action command that maximizes the reward,
updating the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action command and the actual value of the external force corresponding to the predicted value of the external force is reduced; do,
Machine learning methods for robot models. Further, an error between the predicted value of the position and orientation calculated by the state transition model based on the determined action command and the actual value of the position and orientation corresponding to the predicted value of the position and orientation is reduced. 12. The robot model machine learning method according to claim 11, wherein the state transition model is updated to . 13. The reward according to claim 11 or 12, wherein when the external force is a corrected external force that suppresses the expansion of the error, the reward is calculated using a predicted value of the corrected external force as a reduction factor of the reward. Machine learning methods for robot models. 13. The reward according to claim 11 or 12, wherein when the external force is a hostile external force that is an external force that suppresses reduction of the error, the reward is calculated using a predicted value of the hostile external force as a factor for increasing the reward. Machine learning methods for robot models. When the external force is a corrected external force that suppresses the expansion of the error, the reward is calculated by calculation using the predicted value of the corrected external force as a factor for reducing the reward, and the external force suppresses the reduction of the error. 13. The machine learning method for a robot model according to claim 11 or 12, wherein when the hostile external force is , the reward is calculated by calculation using the predicted value of the hostile external force as a factor for increasing the reward. The external force model includes a corrected external force model that outputs a predicted value of the corrected external force when the external force is the corrected external force, and a predicted value of the hostile external force when the external force is the hostile external force. including a hostile force model and
When the external force is the corrected external force, the corrected external force is adjusted so that the difference between the predicted value of the corrected external force calculated by the corrected external force model based on the determined action command and the actual value of the external force becomes small. The model is updated so that, when the external force is the hostile external force, the difference between the predicted value of the hostile external force calculated by the hostile external force model based on the determined action command and the actual value of the external force becomes smaller. 16. The machine learning method for a robot model according to claim 15, wherein said hostile force model is updated such that: 17. The robot model machine according to claim 16, wherein the correcting external force is applied to the robot when the error is increasing, and the hostile external force is applied to the robot when the error is decreasing. learning method. A state transition model that calculates a predicted value of the robot's position and orientation at the next time based on the actual values of the robot's position and orientation at a certain time and an action command that can be given to the robot, and an external force applied to the robot. A machine learning program for machine learning a robot model including an external force model for calculating a predicted value,
Acquiring the actual values of the position and orientation and the actual values of the external force applied to the robot for each control cycle,
A plurality of candidates for the action command are generated for each control cycle and given to the robot model, and a plurality of predicted values of the position and orientation calculated by the state transition model corresponding to the plurality of candidates for the action command, and Based on a plurality of predicted values of the external force calculated by the external force model corresponding to a plurality of errors between target values of the position and orientation to be reached and a plurality of candidates of the action command, a plurality of the action commands Based on the multiple rewards calculated for the candidates, determine the action command that maximizes the reward,
updating the external force model so that the difference between the predicted value of the external force calculated by the external force model based on the determined action command and the actual value of the external force corresponding to the predicted value of the external force is reduced; do,
A robot model machine learning program that causes a computer to perform each process. A state transition model that calculates a predicted value of the robot's position and orientation at the next time based on the actual values of the robot's position and orientation at a certain time and an action command that can be given to the robot, and an external force applied to the robot. a model execution unit that executes a robot model including an external force model that calculates a predicted value;
an acquisition unit that acquires the actual values of the position and orientation of the robot and the actual values of the external force applied to the robot;
a reward calculation unit that calculates a reward based on an error between the predicted value of the position and orientation calculated by the robot model and the target value of the position and orientation to be reached, and the predicted value of the external force calculated by the robot model;
For each control cycle, a plurality of candidates for the action command are generated and given to the robot model, and a reward is maximized based on the reward calculated by the reward calculation unit corresponding to each of the plurality of candidates for the action command. an action determination unit that determines an action command to
A robot controller with A state transition model that calculates a predicted value of the robot's position and orientation at the next time based on the actual values of the robot's position and orientation at a certain time and an action command that can be given to the robot, and an external force applied to the robot. Prepare a robot model that includes an external force model that calculates the predicted value,
Acquiring the actual values of the position and orientation and the actual values of the external force applied to the robot for each control cycle,
A plurality of candidates for the action command are generated for each control cycle and given to the robot model, and a plurality of predicted values of the position and orientation calculated by the state transition model corresponding to the plurality of candidates for the action command, and Based on a plurality of predicted values of the external force calculated by the external force model corresponding to a plurality of errors between target values of the position and orientation to be reached and a plurality of candidates of the action command, a plurality of the action commands Based on the multiple rewards calculated for the candidates, determine the action command that maximizes the reward,
controlling the robot based on the determined action command;
Robot control method. A state transition model that calculates a predicted value of the robot's position and orientation at the next time based on the actual values of the robot's position and orientation at a certain time and an action command that can be given to the robot, and an external force applied to the robot. A program for controlling the robot using a robot model including an external force model for calculating a predicted value,
Acquiring the actual values of the position and orientation and the actual values of the external force applied to the robot for each control cycle,
A plurality of candidates for the action command are generated for each control cycle and given to the robot model, and a plurality of predicted values of the position and orientation calculated by the state transition model corresponding to the plurality of candidates for the action command, and Based on a plurality of predicted values of the external force calculated by the external force model corresponding to a plurality of errors between target values of the position and orientation to be reached and a plurality of candidates of the action command, a plurality of the action commands Based on the multiple rewards calculated for the candidates, determine the action command that maximizes the reward,
controlling the robot based on the determined action command;
A robot control program that causes a computer to perform each process.

Images