JP2023137880A - Information processing device, information processing method and program - Google Patents

Abstract

To provide an information processing device, an information processing method, and a program that enhance efficiency of machine learning.SOLUTION: In a system including an information processing device, a robot, and a sensor, an information processing device 10 includes: an acquisition unit 11 that acquires an expert trajectory for a specific task; a generator 12 that performs reinforcement learning on the basis of at least a first reward based on behavior information being identified as the expert trajectory, and a second reward based on the specific task being executed on the basis of the behavior information, and generates the behavior information on the basis of the acquired information and a result of the reinforcement learning; and a discriminator 13 that discriminates whether input information is the generated behavior information or the expert trajectory. The generator 12 performs reinforcement learning at a first point of time by setting a ratio of the second reward to the first reward as a first ratio, and performs reinforcement learning at a second point of time later than the first point of time by setting the ratio of the second reward to the first reward as a second ratio higher than the first ratio.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to an information processing device, an information processing method, and a program.

In recent years, a method called GAIL (Generative Adversarial Imitation Learning), which combines an imitation learning algorithm using inverse reinforcement learning and a generative adversarial network (GAN), has been attracting attention (non-patent literature). 1).

Inverse reinforcement learning is a method that estimates the reward function of the environment from the expert's action trajectory (expert data, Expert Trajectory), so imitation learning can be performed even when no reward is obtained from the environment. In imitation learning using inverse reinforcement learning, it is necessary to solve two problems: the problem of finding a reward function from the expert's behavioral trajectory, and the problem of finding the expert policy from the obtained reward function by reinforcement learning. be. On the other hand, in GAIL, by using the GAN mechanism, it is possible to determine the expert's policy from the expert's action trajectory.

Furthermore, Non-Patent Document 2 discloses a technique for machine learning control of a robot arm using GAIL. In Non-Patent Document 2, an image taken with a camera and information indicating the position and angular velocity of a joint of a robot arm are acquired. Further, a generator is trained using a hybrid reward γ that is a combination of the imitation learning reward γ _gail and the reinforcement learning reward γ _task as shown in the following equation (1). Note that the imitation learning reward γ _gail is a reward based on being able to fool the discriminator. Further, the reinforcement learning reward γ _task is a reward based on the completion of a task (work) by the robot.

Further, λ is a constant set in advance, and has a value between 0 and 1. s _t is the input data to the generator, and a _t is the output data from the generator. Note that when λ is 0, only reinforcement learning (RL) is used, and when λ is 1, normal GAIL (for example, as described in Non-Patent Document 1) is used.
γ (s _t , _at ) = λγ _gail (s _t , _at ) + (1-λ) γ _task (s _t , at ₎ ...(1)

Jonathan Ho and Stefano Ermon. "Generative adversarial imitation learning" NIPS, 2016. Yuke Zhu, et al. "Reinforcement and Imitation Learning for Diverse Visuomotor Skills" RSS, 2018

However, with the conventional technology, for example, machine learning may not be efficiently executed.

An object of the present disclosure is to provide an information processing device, an information processing method, and a program that can improve the efficiency of machine learning.

In a first aspect of the present disclosure, there is provided an acquisition unit that acquires information indicating a behavioral trajectory of an expert with respect to a specific task; Reinforcement learning is performed based on at least a reward and a second reward based on the execution of the specific task based on the behavioral information, and the information acquired by the acquisition unit and the result of the reinforcement learning are a generator that generates behavior information based on the information, and a discriminator that identifies whether input information is behavior information generated by the generator or information indicating a behavior trajectory of the expert. , the generator performs the reinforcement learning at a first point in time with a ratio of the second reward to the first reward as a first ratio, and sets the second reward to the first reward as a first ratio higher than the first ratio. An information processing device is provided that performs the reinforcement learning at a second time point after the first time point at a ratio of 2 to 2.

Further, in a second aspect of the present disclosure, there is provided a process of acquiring information indicating a behavioral trajectory of an expert with respect to a specific task, and a process of acquiring information indicating the behavioral trajectory of the expert with respect to a specific task, and Reinforcement learning is performed based on at least one reward and a second reward based on the execution of the specific task based on the behavioral information, and the information acquired by the acquisition process and the result of the reinforcement learning. and a process of identifying whether the input information is information generated by the generation process or information indicating the behavioral trajectory of the expert based on the generated behavior information. In the processing, the reinforcement learning is performed at a first time point by setting the ratio of the second reward to the first reward as a first ratio, and the second reward to the first reward is set to a second ratio higher than the first ratio. An information processing method is provided in which the reinforcement learning is performed at a second time point after the first time point.

Further, in a third aspect of the present disclosure, there is provided a process of acquiring information indicating a behavioral trajectory of an expert with respect to a specific task, and a process of acquiring information indicating the behavioral trajectory of the expert with respect to a specific task; Reinforcement learning is performed based on at least one reward and a second reward based on the execution of the specific task based on the behavioral information, and the information acquired by the acquisition process and the result of the reinforcement learning. causing a computer to execute a process of generating behavioral information based on the above, and a process of identifying whether the input information is information generated by the generating process or information indicating the behavioral trajectory of the expert, In the generation process, the reinforcement learning is performed at a first point in time with the ratio of the second reward to the first reward as a first ratio, and the second reward to the first reward is set to a higher ratio than the first ratio. A program is provided that performs the reinforcement learning at a second point in time that is later than the first point in time.

According to one aspect, the efficiency of machine learning can be improved.

1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. 1 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to an embodiment. 5 is a flowchart illustrating an example of learning processing of the information processing device according to the embodiment. 7 is a flowchart illustrating an example of inference processing of the information processing device according to the embodiment.

The principles of the present disclosure are explained with reference to several exemplary embodiments. It is to be understood that these embodiments are described for illustrative purposes only and do not suggest limitations as to the scope of the disclosure, and to assist those skilled in the art in understanding and practicing the disclosure. The disclosure described herein may be implemented in a variety of ways other than those described below.
In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. has.
Embodiments of the present disclosure will be described below with reference to the drawings.

<System configuration>
With reference to FIG. 1, the configuration of an information processing system 1 according to an embodiment will be described. FIG. 1 is a diagram illustrating an example of the configuration of an information processing system 1 according to an embodiment. In the example of FIG. 1, the information processing system 1 includes an information processing device 10, a robot 20, and a sensor 30. Note that the number of information processing devices 10, robots 20, and sensors 30 is not limited to the example in FIG. 1. Note that the information processing device 10 and the sensor 30 may be housed inside the housing of the robot 20. The information processing device 10, the robot 20, and the sensor 30 are connected so that they can communicate wirelessly or by wire.

The information processing device 10 is a device that controls the robot 20 using machine learning. For example, the information processing device 10 acquires, through the sensor 30, the movement of a human or the like when performing a task (work) using a tool as information indicating an expert's action trajectory, and performs learning based on the acquired information. Then, the information processing device 10 causes the robot 20 to use the tool in the same way as a human or the like to execute the task.

The robot 20 is a robot that performs tasks using various tools using an arm or the like. The robot 20 may be any device that can perform tasks using tools, and its external shape is not limited. The robot 20 can be used for various purposes such as home use, exploration use, and factory use. The sensor 30 is a sensor that measures the surroundings of the robot 20. Sensor 30 may be, for example, a camera or LiDAR.

<Hardware configuration>
FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device 10 according to the embodiment. In the example of FIG. 2, the information processing device 10 (computer 100) includes a processor 101, a memory 102, and a communication interface 103. These parts may be connected by a bus or the like. Memory 102 stores at least a portion of program 104. Communication interface 103 includes interfaces necessary for communication with other network elements.

When the program 104 is executed by the cooperation of the processor 101, the memory 102, etc., the computer 100 performs at least part of the processing of the embodiment of the present disclosure. Memory 102 may be of any type suitable for the local technology network. Memory 102 may be, by way of non-limiting example, a non-transitory computer-readable storage medium. Memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. Although only one memory 102 is shown in computer 100, there may be several physically different memory modules present in computer 100. Processor 101 may be of any type. Processor 101 may include one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, by way of non-limiting example. Computer 100 may have multiple processors, such as application specific integrated circuit chips, that are time dependent on a clock that synchronizes the main processors.

Embodiments of the present disclosure may be implemented in hardware or special purpose circuitry, software, logic or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.

The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer readable storage medium. A computer program product includes computer-executable instructions, such as instructions contained in program modules, that are executed on a device on a target real or virtual processor to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided among program modules as desired in various embodiments. Machine-executable instructions of program modules can be executed locally or within distributed devices. In distributed devices, program modules can be located in both local and remote storage media.

Program code for implementing the methods of this disclosure may be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing device. When the program code is executed by a processor or controller, the functions/acts illustrated in the flowcharts and/or implementing block diagrams are performed. Program code can run entirely on a machine, partially on a machine, as a standalone software package, partially on a machine, partially on a remote machine, or entirely on a remote machine or server. Ru.

The program can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media includes various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disk media, semiconductor memory, and the like. Magnetic recording media include, for example, flexible disks, magnetic tapes, hard disk drives, and the like. The magneto-optical recording medium includes, for example, a magneto-optical disk. Optical disc media include, for example, Blu-ray discs, CDs (Compact Discs)-ROMs (Read Only Memory), CD-Rs (Recordables), CD-RWs (ReWritables), and the like. Semiconductor memories include, for example, solid state drives, mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs (Random Access Memory), and the like. The program may also be provided to the computer on various types of temporary computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can provide the program to the computer via wired communication channels, such as electrical wires and fiber optics, or wireless communication channels.

<Configuration>
Next, with reference to FIG. 3, the configuration of the information processing device 10 according to the embodiment will be described. FIG. 3 is a diagram illustrating an example of the configuration of the information processing device 10 according to the embodiment. In the example of FIG. 3, the information processing device 10 includes an acquisition unit 11, a generator 12, a discriminator 13, and a control unit 14. Each of these units may be realized by cooperation between one or more programs installed in the information processing device 10 and hardware such as the processor 101 and the memory 102 of the information processing device 10.

The acquisition unit 11 acquires information indicating an expert's action trajectory for a specific task and information indicating an environment related to the robot 20 from a storage device inside the information processing device 10 or an external device.

The generator 12 generates information indicating behavior based on the information acquired by the acquisition unit 11 and the results of reinforcement learning. The generator 12 outputs to the discriminator 13 either the information indicating the expert's behavior trajectory acquired by the acquisition unit 11 or the information indicating the behavior generated based on the results of reinforcement learning. Based on the identification result, the reward γ _gail (an example of the "first reward") is determined. Furthermore, the generator 12 determines a reward γ _task (an example of a “second reward”) based on execution (completion, success) of a specific task by the robot 20.

Further, the generator 12 performs reinforcement learning at the first time point by setting the ratio of the reward γ _task to the reward γ _gail as the first ratio, and sets the reward γ _task to the reward γ _gail as the second ratio higher than the first ratio. Reinforcement learning is performed at a second point in time after the point in time.

The discriminator 13 identifies whether the information input from the generator 12 is information generated by the generator 12 or information indicating the expert's action trajectory acquired by the acquisition unit 11. The control unit 14 controls the robot 20 based on information indicating the behavior generated by the generator 12.

<Processing>
＜＜Learning phase＞＞
Next, with reference to FIG. 4, an example of the learning process of the information processing apparatus 10 according to the embodiment will be described. FIG. 4 is a flowchart illustrating an example of the learning process of the information processing device 10 according to the embodiment.

In step S101, the acquisition unit 11 of the information processing device 10 acquires information indicating an expert's action trajectory for a specific task. A particular task may be, for example, driving a nail with a hammer or scooping water with a glass. Here, the acquisition unit 11 acquires, as information indicating the expert's action trajectory, the position and posture of the human arm and the tool at each point in time when the tool (e.g., hammer, cup, etc.) is used by the human. You may also obtain information that indicates. Information indicating the expert's action trajectory may be generated, for example, by analyzing an image taken by the sensor 30, which is a camera, using a convolutional neural network (CNN). Further, the information indicating the expert's action trajectory may be generated based on data measured by a sensor 30 attached to at least one of a human arm and a tool, for example.

Subsequently, the acquisition unit 11 of the information processing device 10 acquires information indicating the environment (environment information, information indicating the action trajectory of the robot 20) (step S102). The environmental information may be generated, for example, by analyzing an image taken by the sensor 30, which is a camera, using CNN. Further, the environmental information may be generated based on data measured by a sensor 30 provided (attached) to at least one of an arm and a tool of the robot 20, for example. The environmental information may include, for example, information indicating the position and orientation of the tool. Further, the environmental information may include, for example, information indicating the positions and angular velocities of the joints of the arms of the robot 20.

Subsequently, the generator 12 of the information processing device 10 determines (updates) the hybrid reward γ _s (step S103). Here, the generator 12 may calculate the hybrid reward γ _s by determining the value of the weighting coefficient α in the following equation (2), for example.
γ _s (s _t , _at )=αγ _gail (s _t , _at )+(1-α) γ _task (s _t , at ₎ ...(2)

Here, α may be a variable from 0 to 1. s _t is environmental information, and a _t is information indicating behavior (behavior information). Note that when α is 0, only normal reinforcement learning is used, and when α is 1, only normal GAIL (imitation learning) is used.

The reward γ _gail is a reward for imitation learning by normal GAIL. The reward γ _task (an example of a “second reward”) is a reinforcement learning reward based on execution (completion, success) of a specific task by the robot 20.

Reward γ _gail for imitation learning by GAIL is a reward in normal GAIL. In other words, the reward γ _gail is based on the fact that the behavior information generated by the generator 12 is identified (determined, determined, discriminated) as information indicating the expert's behavior trajectory acquired by the acquisition unit 11 (the discriminator 13 The reward is based on whether the user was able to deceive the user, and whether the generated behavioral information is likely to be an expert's behavioral trajectory.

For example, the generator 12 may input data of a combination of s _t that is input data to the generator 12 and a _t that is output data from the generator 12 to the discriminator 13 . Then, the classifier 13 may calculate (estimate, infer) a value (accuracy, reliability) of the likelihood of the expert's action trajectory for the data in step S107, which will be described later. Then, the generator 12 may determine the value of the reward γ _gail to be higher as the value of the expert's behavior trajectory likelihood p calculated for the data by the discriminator 13 is higher.

In addition, the reinforcement learning reward γ _task is a reward based on the execution of a specific task by the robot 20. The control unit 14 controls the robot based on the behavior _information at generated by the generator 12, for example. Then, the generator 12 determines, for example, the position and posture of the tool used by the robot arm controlled based on the behavior information _at , based on the position and posture at the start of the task in the expert's behavior trajectory when the task is completed. When there is a change (transition) to the position, posture, etc. at the time, the value of the reinforcement learning reward γ _task may be set to a specific value other than 0.

The generator 12 may determine the value of α to be a smaller value as learning progresses. As a result, for example, in the early stage of learning, the influence of imitation learning by GAIL can be made relatively large, and in the latter stage of learning, the influence of reinforcement learning due to being able to execute a task can be made relatively large. Therefore, it is thought that the efficiency of machine learning can be improved through a learning process similar to that used by humans, in which the robot initially learns by watching what it sees, and once it has learned a certain amount, fine-tunes it by trial and error. In this case, the generator 12 sets the reward γ _task to the reward γ _gail at a first rate at the first time point, and sets the reward γ _task to the reward γ _gail at a higher rate than the first rate at a second time point after the first time point. It may also be a second ratio.

In this case, the generator 12 may determine the first ratio and the second ratio, for example, depending on at least one of the number of times reinforcement learning in step S104 is performed and the performance of the reinforcement learning result. Here, the generator 12 generates information indicating, for example, the time required for the robot 20 to execute a specific task, the power consumption required to execute the specific task, and the number of times the robot 20 generates information indicating the behavior related to the execution of the specific task. The performance value of the reinforcement learning result may be determined (identified, calculated) based on at least one of the following. In this case, the generator 12 may determine, for example, that the shorter the time required from the start of a specific task to the completion by the robot 20, the higher the value of the performance. Further, the generator 12 may determine, for example, that the less power is consumed by the robot 20 to perform a specific task, the higher the value of the performance is. Furthermore, the generator 12 may determine, for example, that the smaller the number of behavioral information generated until the robot 20 executes a specific task, the higher the value of the performance. Note that the value of power consumption in the robot 20 may be measured by the sensor 30 of the robot 20. Note that in the example of formula (2) above, an example was explained in which the hybrid reward γ _s is determined based on the reward γ _gail of imitation learning by GAIL and the reward γ _task of reinforcement learning, but the technology of the present disclosure but not limited to. For example, the generator 12 may determine the hybrid reward γ _s based on the reward γ _gail of imitation learning by GAIL and the reward γ _task of reinforcement learning, as well as the reward of other learning methods.

Subsequently, the generator 12 of the information processing device 10 performs reinforcement learning using the hybrid reward γ _s of equation (2) above (step S104). Next, the generator 12 of the information processing device 10 generates behavior information a _t for the environmental information s _t that is input data based on the result of reinforcement learning (step S105). As the information _at , for example, data for controlling the robot may be generated and output. In this case, the behavior information may include, for example, information indicating the angular velocity of each joint of the arm of the robot.

Subsequently, the discriminator 13 of the information processing device 10 calculates (estimates, infers) the value (accuracy, reliability) of the expert's behavior trajectory likelihood p for the data of the combination of the environmental information s _t and the behavior information a _t . and outputs it to the generator 12 (step S106).

Next, the discriminator 13 of the information processing device 10 uses, for example, a neural network (NN) to set the value of p to 0 if the behavioral information _at is generated by the generator 12. supervised learning is performed to calculate the value of p as 1 when the behavior _information at is the expert's behavior trajectory (step S107). Thereby, similarly to GAIL, the discriminator 13 and the generator 12 can be trained to compete with each other using GAN. Here, the discriminator 13 uses, for example, data of a combination of environmental information _st , behavior information _at , and a correct label indicating whether or not the behavior information _at is generated by the generator 12. You may also learn by supervised learning based on this.

Subsequently, the control unit 14 of the information processing device 10 causes the robot 20 to operate based on the behavior information generated by the generator 12 (step S108). Here, the control unit 14 may transmit a control command according to the behavior information to the robot 20. Subsequently, the generator 12 of the information processing device 10 determines whether to end learning (step S109). Here, the generator 12 determines, for example, when at least one of the number of times a specific task has been executed by the robot 20, the number of times reinforcement learning in step S104 has been performed, and the performance of the reinforcement learning result is greater than or equal to a threshold value. , it may be determined that learning has ended.

If it is determined that learning is not to be completed (NO in step S109), the process proceeds to step S102. On the other hand, if it is determined that learning is to be ended (YES in step S109), the learning process is ended.

＜＜Inference phase＞＞
Next, with reference to FIG. 5, an example of the inference processing of the information processing apparatus 10 according to the embodiment will be described. FIG. 5 is a flowchart illustrating an example of inference processing by the information processing device 10 according to the embodiment.

In step S201, the control unit 14 of the information processing device 10 determines (recognizes) the content of the task to be executed by the robot 20. Here, the control unit 14 may determine the content of the task based on, for example, input from the user such as voice or button operation. For example, the control unit 14 may determine the content of the task, such as driving a nail with a hammer or scooping water with a cup.

Subsequently, the acquisition unit 11 of the information processing device 10 acquires environmental information (step S202). The process in step S202 may be similar to the process in step S102 in FIG. 4, for example.

Subsequently, the generator 12 of the information processing device 10 generates behavior information a _t for the environmental information s _t that is input data, based on the result of reinforcement learning (step S203). The process in step S203 may be similar to the process in step S105 in FIG. 4, for example.

Subsequently, the control unit 14 of the information processing device 10 causes the robot 20 to operate based on the behavior information generated by the generator 12 (step S204). The process in step S204 may be similar to the process in step S108 in FIG. 4, for example.

<Modified example>
The information processing device 10 may be a device included in one housing, but the information processing device 10 of the present disclosure is not limited to this. Each part of the information processing device 10 may be realized by cloud computing configured by, for example, one or more computers. Information processing apparatuses such as these are also included in an example of the "information processing apparatus" of the present disclosure.

Note that the present invention is not limited to the above embodiments, and can be modified as appropriate without departing from the spirit.

1 Information processing system 10 Information processing device 11 Acquisition unit 12 Generator 13 Discriminator 14 Control unit 20 Robot 30 Sensor

Claims (7)

an acquisition unit that acquires information indicating an expert's action trajectory for a specific task;
At least based on a first reward based on the fact that the behavior information is identified as information indicating a behavior trajectory of the expert, and a second reward based on the fact that the specific task is executed based on the behavior information. a generator that performs reinforcement learning and generates behavioral information based on the information acquired by the acquisition unit and the result of the reinforcement learning;
a discriminator that identifies whether the input information is behavior information generated by the generator or information indicating a behavior trajectory of the expert;
The generator performs the reinforcement learning at a first time point by setting a ratio of the second reward to the first reward as a first ratio, and sets the second reward to the first reward as a second ratio higher than the first ratio. performing the reinforcement learning at a second point in time that is later than the first point in time as a proportion;
Information processing device. The generator determines the first ratio and the second ratio according to the number of times the reinforcement learning is performed.
The information processing device according to claim 1. The generator determines the first ratio and the second ratio according to the performance of the reinforcement learning result.
The information processing device according to claim 1 or 2. The generator is based on at least one of the time required to execute the specific task, the power consumption required to execute the specific task, and the number of times the behavior information related to the execution of the specific task is generated. , determining the value of said performance;
The information processing device according to claim 3. The expert's action trajectory for the specific task is the action trajectory of a human in a task using a specific tool,
The behavior information includes information indicating the angular velocity of the joint of the robot arm.
The information processing device according to any one of claims 1 to 4. a process of acquiring information indicating an expert's action trajectory for a specific task;
At least based on a first reward based on the fact that the behavior information is identified as information indicating a behavior trajectory of the expert, and a second reward based on the fact that the specific task is executed based on the behavior information. A process of performing reinforcement learning and generating behavioral information based on the information acquired by the acquisition process and the result of the reinforcement learning;
performing a process of identifying whether the input information is information generated by a generation process or information indicating an action trajectory of the expert;
In the generation process, the reinforcement learning is performed at a first point in time with the ratio of the second reward to the first reward as a first ratio, and the second reward to the first reward is set to a higher ratio than the first ratio. performing the reinforcement learning at a second time point later than the first time point at a rate of 2;
Information processing method. a process of acquiring information indicating an expert's action trajectory for a specific task;
At least based on a first reward based on the fact that the behavior information is identified as information indicating a behavior trajectory of the expert, and a second reward based on the fact that the specific task is executed based on the behavior information. A process of performing reinforcement learning and generating behavioral information based on the information acquired by the acquisition process and the result of the reinforcement learning;
causing a computer to perform a process of identifying whether the input information is information generated by a generation process or information indicating an action trajectory of the expert;
In the generation process, the reinforcement learning is performed at a first point in time with the ratio of the second reward to the first reward as a first ratio, and the second reward to the first reward is set to a higher ratio than the first ratio. performing the reinforcement learning at a second time point later than the first time point at a rate of 2;
program.

Images