JP2023145809A - Reinforcement learning device, reinforcement learning system, object operation device, model generation method and reinforcement learning program - Google Patents

Abstract

To provide a reinforcement learning device, reinforcement learning system, object operation device, model generation method and reinforcement learning program which can increase a success probability of a prescribed operation to an object.SOLUTION: A reinforcement learning device includes at least one memory and at least one processor. The at least one processor is configured to be able to execute processing of inputting information about a photographed image captured by an imaging device in which any of the position and attitude thereof changes and information about a target object image showing an object being an operation target operated by an end effector to a training model that outputs information for controlling the operation of the end effector, and updating a parameter of the training model on the basis of the operation result to the object when the operation of the end effector is controlled on the basis of the information output by the training model.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a reinforcement learning device, a reinforcement learning system, an object manipulation device, a model generation method, and a reinforcement learning program.

Input images taken with a fixed camera to successfully perform a predetermined operation (for example, a gripping operation using an end effector) on a specified type of object among multiple types of objects placed within a predetermined area. A reinforcement learning system that performs reinforcement learning on the motion of an end effector is known.

According to the reinforcement learning system, if an object of a specified type is placed in a position where it can be photographed, reinforcement learning can be repeated to increase the success probability of a predetermined operation. On the other hand, if the specified type of object is not placed in a position where it can be photographed, reinforcement learning cannot proceed and the probability of success of a predetermined operation cannot be increased.

M. Danielczuk et al., "Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Cluster", in ICRA, 2019. E. Jang, C. Devin, V. Vanhoucke, and S. Levine, "Grasp2Vec: Learning Object Representations from Self-Supervised Grasping", in CoRL, 2018.

The present disclosure provides a reinforcement learning device, a reinforcement learning system, an object manipulation device, a model generation method, and a reinforcement learning program that can increase the success probability of a predetermined operation on an object.

A reinforcement learning device according to one aspect of the present disclosure has, for example, the following configuration. That is,
at least one memory;
at least one processor;
The at least one processor includes:
The operation of the end effector is controlled by using information regarding a photographed image photographed by an imaging device whose position or orientation changes at least, and information regarding a target object image indicating an object to be operated by the end effector. inputting information into a training model that outputs information for
When the operation of the end effector is controlled based on the information output by the training model, the parameters of the training model can be updated based on the operation result for the object.

1 is a diagram illustrating an example of a system configuration of a reinforcement learning system. It is a diagram showing an example of the hardware configuration of each device that constitutes the reinforcement learning system. FIG. 1 is a first diagram showing an example of a functional configuration of a reinforcement learning device. It is a 1st flowchart which shows the flow of reinforcement learning processing. FIG. 2 is a first diagram showing an example of execution of reinforcement learning processing. FIG. 2 is a second diagram showing an example of execution of reinforcement learning processing. FIG. 2 is a second diagram showing an example of the functional configuration of the reinforcement learning device. It is a 2nd flowchart which shows the flow of reinforcement learning processing.

Each embodiment will be described below with reference to the accompanying drawings. Note that, in this specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, thereby omitting redundant explanation.

[First embodiment]
<System configuration of reinforcement learning system>
First, the system configuration of the reinforcement learning system will be explained. FIG. 1 is a diagram showing an example of the system configuration of a reinforcement learning system. As shown in FIG. 1, the reinforcement learning system 100 includes a manipulator 110 and a reinforcement learning device 120.

For example, the manipulator 110 performs a predetermined operation on a specified type of object (the object to be operated indicated by the target object image) from among the object group 130 in which a plurality of types of objects are placed together. This is a device that performs

The main body 113 of the manipulator 110 has a plurality of arms connected via a plurality of joints, and by controlling the joint angles of each arm, the position and posture of the distal end portion of the main body 113 of the manipulator 110 are controlled. It is configured to

A gripping mechanism 111 (an example of an end effector) that performs a predetermined operation (grasping operation in this embodiment) on a specified type of object is attached to the tip of the main body 113 of the manipulator 110. A gripping operation for a specified type of object is performed by controlling opening and closing of the gripping mechanism section 111.

Furthermore, an imaging device 112 is attached to the tip of the main body 113 of the manipulator 110. In other words, the imaging device 112 is configured so that its position and orientation change as the position and orientation of the gripping mechanism section 111 change. The imaging device 112 outputs captured images including R value, G value, and B value images at a predetermined frame period. Alternatively, the imaging device 112 may output captured images containing distance information to each position on the object surface at a predetermined frame period, in addition to the R value, G value, and B value images. Alternatively, the imaging device 112 may output a distance image including distance information to each position on the object surface at a predetermined frame period. Further, the captured image captured by the imaging device 112 may be a moving image. In the following, for the sake of simplicity, the imaging device 112 will be described as an example that outputs captured images including R value, G value, and B value images at a predetermined frame period.

Furthermore, the support base 114 that supports the main body 113 of the manipulator 110 is provided with a drive that "controls the operation of the gripping mechanism 111" (controls the position and posture of the gripping mechanism 111 and the opening and closing of the gripping mechanism 111). A control device 115 is built-in.

The drive control device 115 acquires a captured image captured by the imaging device 112 and transmits it to the reinforcement learning device 120. Further, the drive control device 115 acquires sensor signals detected by various sensors (not shown) arranged in the gripping mechanism section 111 and the main body section 113 of the manipulator 110, and transmits them to the reinforcement learning device 120.

Further, the drive control device 115 acquires information for controlling the operation of the gripping mechanism section 111 from the reinforcement learning device 120 in response to transmitting the captured image and the sensor signal. The information for controlling the operation of the gripping mechanism section 111 here includes, for example,
- Information indicating the state of the gripping mechanism section 111 after operation (target value),
- Specific operation amount and control amount for controlling the position and posture of the gripping mechanism section 111 and opening and closing of the gripping mechanism section 111,
Any command regarding the operation of the gripping mechanism section 111 may be included. Further, the information for controlling the operation of the gripping mechanism section 111 may include information for controlling the operation of the manipulator 110. In the following, the drive control device 115 will be described as acquiring information indicating the state of the gripping mechanism 111 after its operation, as an example of information for controlling the operation of the gripping mechanism 111.

Furthermore, upon acquiring the information indicating the state of the gripping mechanism 111 after operation, the drive control device 115 performs the following operations based on various sensor signals (information indicating the state of the gripping mechanism 111 before operation).
- Various actuators (not shown) in the gripping mechanism section 111 of the manipulator 110, and
- Various actuators (not shown) in the main body 113 of the manipulator 110,
control. As a result, the position and posture of the gripping mechanism section 111 and the opening and closing of the gripping mechanism section 111 are controlled.

The reinforcement learning device 120 inputs the photographed image transmitted from the drive control device 115 and a target object image indicating the object to be grasped to be grasped by the grasping mechanism section 111, and indicates the state after the operation of the grasping mechanism section 111. It has a reinforcement learning model (an example of a training model) that outputs information. For example, a neural network may be used as the reinforcement learning model.

Note that when inputting the information regarding the photographed image transmitted from the drive control device 115 to the reinforcement learning model, instead of inputting the photographed image itself, a feature amount extracted from the photographed image may be input. The feature amount extracted from the photographed image is, for example, the feature amount output from the intermediate layer by inputting the photographed image into a neural network.

Further, the information regarding the target object image input to the reinforcement learning model may be a captured image including each image of R value, G value, and B value, or each image of R value, G value, and B value and the object The captured image may also include distance information to each position on the surface. Alternatively, the target object image may be a distance image including distance information to each position on the object surface. Alternatively, the target object image may be a moving image. In addition, instead of inputting the target object image itself, the reinforcement learning model uses features extracted from the target object image (for example, features output from the intermediate layer by inputting the target object image into a neural network). You may also enter In the following description, in order to simplify the explanation, it will be assumed that a captured image including R value, G value, and B value images is input as an example of a target object image.

Furthermore, the operation of the grasping mechanism section 111 is controlled based on the information indicating the state after the operation of the grasping mechanism section 111 outputted by the reinforcement learning model, so that the reinforcement learning device 120 can perform operations on the object to be grasped. A result (for example, a determination result as to whether or not the gripping operation was successful) is obtained. Then, the reinforcement learning device 120 updates the model parameters of the reinforcement learning model based on the obtained operation results.

In this way, in the reinforcement learning system 100, in order to increase the success probability of a grasping operation when grasping a specified type of object from among the object group 130 in which a plurality of types of objects are placed together,
- Reinforcement learning is performed using information regarding images taken by the imaging device 112 whose position and orientation change as the position and orientation of the gripping mechanism section 111 change.

As a result, for example, even if the object to be gripped is not placed in a position where it can be photographed, the gripping mechanism unit 111 can be operated so that the object to be gripped can be photographed during the reinforcement learning process. can. In other words, according to the present embodiment, it is possible to provide the reinforcement learning system 100 that can increase the success probability of a grasping operation regardless of the placement state of the object to be grasped.

In this embodiment, as shown by reference numeral 140, the vertical direction of the paper in FIG. 1 is defined as the Z-axis direction, the horizontal direction in the paper of FIG. 1 is defined as the Y-axis direction, and the depth direction of the paper in FIG. 1 is defined as the X-axis direction. shall be.

<Hardware configuration of each device configuring the reinforcement learning system>
Next, using FIG. 2, we will explain the hardware configuration of the manipulator 110 (here, the mechanism system is omitted and the hardware configuration related to the control system is shown) and the reinforcement learning device 120, which constitute the reinforcement learning system 100. I will explain. FIG. 2 is a diagram illustrating an example of the hardware configuration of each device configuring the reinforcement learning system.

(1) Hardware Configuration of Manipulator As shown in FIG. 2, the manipulator 110 includes a sensor group 211 and an actuator group 212 in addition to an imaging device 112 and a drive control device 115.

The sensor group 211 includes n sensors. In this embodiment, the n sensors include at least:
- A sensor for calculating the position and orientation of the gripping mechanism section 111 (a sensor for measuring each joint angle of the main body section 113),
- A sensor that detects opening and closing of the gripping mechanism section 111;
is included.

Furthermore, the actuator group 212 includes m actuators. In this embodiment, the m actuators include at least the following:
- An actuator for controlling the position and posture of the gripping mechanism section 111 (an actuator for controlling each joint angle of the main body section 113),
- An actuator for controlling opening and closing of the gripping mechanism section 111;
is included.

Further, the drive control device 115 includes a sensor signal processing device 201, an actuator drive device 202, and a controller 203. The sensor signal processing device 201 receives sensor signals transmitted from the sensor group 211 and notifies the controller 203 of the sensor signal data. Further, the actuator drive device 202 acquires control signal data from the controller 203 and transmits the control signal to the actuator group 212.

The controller 203 acquires the captured image transmitted from the imaging device 112 and transmits it to the reinforcement learning device 120. Further, the controller 203 transmits the sensor signal data notified from the sensor signal processing device 201 to the reinforcement learning device 120.

Further, the controller 203 acquires information indicating the state of the gripping mechanism section 111 after operation from the reinforcement learning device 120 in response to transmitting the captured image and sensor signal data. Furthermore, upon acquiring information indicating the state of the gripping mechanism section 111 after operation, the controller 203 generates control signal data for operating the actuator group 212 based on the sensor signal data, and notifies the actuator drive device 202 of the control signal data.

(2) Hardware configuration of reinforcement learning device Next, the hardware configuration of the reinforcement learning device 120 will be explained. As shown in FIG. 2, the reinforcement learning device 120 includes a processor 221, a main storage device (memory) 222, an auxiliary storage device 223, a network interface 224, and a device interface 225 as components. Reinforcement learning device 120 is realized as a computer to which these components are connected via bus 226.

In the example of FIG. 2, the reinforcement learning device 120 is shown as having one of each component, but the reinforcement learning device 120 may include a plurality of the same components. In addition, in the example of FIG. 2, one reinforcement learning device 120 is shown, but the reinforcement learning program is installed in a plurality of reinforcement learning devices, and each of the plurality of reinforcement learning devices has a reinforcement learning program. They may be configured to perform some of the same or different processing. In this case, a form of distributed computing may be adopted in which each reinforcement learning device executes the entire process by communicating via the network interface 224 or the like. In other words, the reinforcement learning device 120 may be configured as a system that realizes functions by having one or more computers execute instructions stored in one or more storage devices. Alternatively, the configuration may be such that various data transmitted from the drive control device 115 are processed by one or more reinforcement learning devices provided on the cloud, and the processing results are transmitted to the drive control device 115.

Various operations of the reinforcement learning device 120 may be executed in parallel using one or more processors or using multiple reinforcement learning devices communicating via the communication network 240. Furthermore, various calculations may be distributed to a plurality of calculation cores within the processor 221 and executed in parallel. Further, some or all of the processing, means, etc. of the present disclosure are executed by an external device 230 (at least one of a processor and a storage device) provided on the cloud that can communicate with the reinforcement learning device 120 via the communication network 240. may be done. In this way, the reinforcement learning device 120 may take the form of parallel computing using one or more computers.

The processor 221 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.). Further, the processor 221 may be a semiconductor device or the like including a dedicated processing circuit. Note that the processor 221 is not limited to an electronic circuit using an electronic logic element, but may be realized by an optical circuit using an optical logic element. Furthermore, the processor 221 may include an arithmetic function based on quantum computing.

The processor 221 performs various calculations based on various data and instructions input from each device in the internal configuration of the reinforcement learning device 120, and outputs calculation results and control signals to each device. The processor 221 controls each component included in the reinforcement learning device 120 by executing an OS (Operating System), applications, and the like.

Further, the processor 221 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or devices. When using multiple electronic circuits, each electronic circuit may communicate by wire or wirelessly.

The main storage device 222 is a storage device that stores instructions and various data to be executed by the processor 221, and the various data stored in the main storage device 222 is read out by the processor 221. The auxiliary storage device 223 is a storage device other than the main storage device 222. Note that these storage devices are any electronic components capable of storing various data, and may be semiconductor memories. Semiconductor memory may be either volatile memory or nonvolatile memory. A storage device for storing various data in the reinforcement learning device 120 may be implemented by the main storage device 222 or the auxiliary storage device 223, or may be implemented by a built-in memory built into the processor 221.

Further, a plurality of processors 221 may be connected (combined) to one main storage device 222, or a single processor 221 may be connected to one main storage device 222. Alternatively, a plurality of main storage devices 222 may be connected (combined) to one processor 221. When the reinforcement learning device 120 is configured with at least one main storage device 222 and a plurality of processors 221 connected (coupled) to this at least one main storage device 222, at least one of the plurality of processors 221 The processor may include a configuration in which the processor is connected to (coupled with) at least one main memory device 222 . Further, this configuration may be realized by the main storage device 222 and processor 221 included in a plurality of reinforcement learning devices 120. Furthermore, a configuration in which the main storage device 222 is integrated with the processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

Network interface 224 is an interface for connecting to communication network 240 wirelessly or by wire. As the network interface 224, an appropriate interface such as one that complies with existing communication standards is used. The network interface 224 may exchange various data with the drive control device 115 and other external devices 230 connected via the communication network 240. Note that the communication network 240 may be a WAN (Wide Area Network), a LAN (Local Area Network), a PAN (Personal Area Network), or a combination thereof, and may include a computer, the drive control device 115, and the like. Any device that can exchange information with other external devices 230 may be used. Examples of WAN include the Internet, examples of LAN include IEEE802.11 and Ethernet, and examples of PAN include Bluetooth (registered trademark) and NFC (Near Field Communication).

The device interface 225 is an interface such as a USB that is directly connected to the external device 250.

External device 250 is a device connected to the computer. External device 250 may be an input device, for example. The input device is, for example, a device such as a camera, microphone, motion capture, various sensors, keyboard, mouse, or touch panel, and provides the acquired information to the computer. Alternatively, the device may be a personal computer, a tablet terminal, a smartphone, or other device including an input section, a memory, and a processor.

Moreover, the external device 250 may be an output device, for example. The output device may be a display device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence) panel, or output audio or the like. It may also be a speaker or the like. Further, it may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

Further, the external device 250 may be a storage device (memory). For example, the external device 250 may be a network storage or the like, and the external device 250 may be a storage such as an HDD.

Further, the external device 250 may be a device that has some functions of the components of the reinforcement learning device 120. That is, the computer may transmit or receive part or all of the processing results of the external device 250.

<Functional configuration of reinforcement learning device>
Next, as the functional configuration of the reinforcement learning device 120, two types of functional configuration examples will be described here. FIG. 3 is a first diagram showing an example of the functional configuration of the reinforcement learning device. As shown in FIG. 3A, the reinforcement learning device 120 includes an update section 310, a state input section 320, and a reinforcement learning model 330.

The update unit 310 includes a reward calculation unit 311 and updates model parameters of the reinforcement learning model 330. Specifically, the updating unit 310 acquires a determination result as to whether or not the gripping operation on the object to be gripped has been successful, and information indicating a change in the state due to the controlled operation of the gripping mechanism unit 111. . Further, the remuneration calculating unit 311 calculates remuneration based on the determination result obtained by the updating unit 310. Then, the updating unit 310 updates the model parameters of the reinforcement learning model 330 based on various types of information (information indicating a change in state, reward, etc.) acquired or calculated so far.

Note that the determination as to whether or not the gripping operation on the object to be gripped has been successful may be automatically performed based on the photographed image, for example. Alternatively, the user of the reinforcement learning system 100 may determine whether the grasping operation on the object to be grasped is successful.

Further, the method for calculating the reward described above is only an example, and the updating unit 310 may calculate the reward based on information other than the determination result as to whether or not the grasping operation was successful. For example, the updating unit 310 calculates the reward based on various information such as the operation time and number of operations required for a successful gripping operation, and the size of the entire operation of the manipulator 110 during the gripping operation (energy efficiency). You may.

The state input unit 320 acquires the captured image transmitted from the drive control device 115 and the target object image input by the user, and notifies the reinforcement learning model 330 of the acquired image.

The model parameters of the reinforcement learning model 330 are updated by the updating unit 310. Further, the reinforcement learning model 330 after the model parameters have been updated receives the captured image and target object image notified from the state input unit 320 and outputs information indicating the state of the gripping mechanism unit 111 after the operation. . In this embodiment, the reinforcement learning model 330 uses, for example, information indicating the state of the gripping mechanism section 111 after operation.
- Information indicating the position and posture of the gripping mechanism section 111 after operation;
- Information indicating opening/closing after operation of the gripping mechanism section 111,
Output.

On the other hand, FIG. 3(b) shows another functional configuration example. As shown in FIG. 3(b), the state input unit 320 of the reinforcement learning device 120 acquires information indicating the state of the gripping mechanism unit 111 before operation (currently) in addition to the captured image and the target object image, The reinforcement learning model 330 is configured to notify the reinforcement learning model 330. Here, the information indicating the state of the gripping mechanism section 111 before operation (currently) includes, for example,
- Information indicating the position and orientation of the gripping mechanism section 111 before operation (currently);
- Information indicating the opening/closing of the gripping mechanism section 111 before operation (currently);
is included.

In this case, the reinforcement learning model 330 receives the captured image, the target object image, and the information indicating the state before the operation of the gripping mechanism section 111 (currently) notified from the state input section 320 as input, and after the operation of the gripping mechanism section 111 Outputs information indicating the status.

<Flow of reinforcement learning processing>
Next, the flow of reinforcement learning processing by the reinforcement learning device 120 will be explained. FIG. 4 is a first flowchart showing the flow of reinforcement learning processing. The flow of reinforcement learning processing will be described below with reference to FIG. 4. Note that the reinforcement learning process shown in FIG. 4 is just an example, and a reinforced learning model may be generated by executing the reinforcement learning process using another model generation method.

In step S401, the state input unit 320 of the reinforcement learning device 120 acquires a target object image.

In step S402, the state input unit 320 of the reinforcement learning device 120 acquires a captured image.

In step S403, if the state input unit 320 of the reinforcement learning device 120 is configured to acquire information indicating the state of the gripping mechanism 111 before operation (currently), the state input unit 320 of the reinforcement learning device 120 Obtain information indicating the pre-operation (current) state.

In step S404, the reinforcement learning model 330 of the reinforcement learning device 120 inputs the target object image, the photographed image (and information indicating the state of the gripping mechanism 111 before operation), and calculates the state of the gripping mechanism 111 after the operation. Outputs information indicating. It is assumed that the reinforcement learning model 330 is configured to comprehensively output various information as information indicating the state of the gripping mechanism section 111 after operation. As a result, the operation of the gripping mechanism unit 111 during the reinforcement learning process includes an optimal operation selected from a set of possible operations and an operation randomly selected from a set of possible operations. It will be.

In step S405, the reinforcement learning device 120 transmits information output by the reinforcement learning model 330 indicating the state of the gripping mechanism section 111 after operation to the drive control device 115.

In step S406, the updating unit 310 of the reinforcement learning device 120 acquires information indicating a change in state due to the controlled operation of the gripping mechanism unit 111.

In step S407, the updating unit 310 of the reinforcement learning device 120 acquires a determination result as to whether or not the grasping operation on the object to be grasped is successful, and the reward calculation unit 311 of the reinforcement learning device 120 uses the acquired determination result to Compensation will be calculated based on the following.

In step S408, the updating unit 310 of the reinforcement learning device 120 updates the model parameters of the reinforcement learning model 330 based on various information (information indicating a change in state, reward, etc.) acquired or calculated so far.

In step S409, the state input unit 320 of the reinforcement learning device 120 determines whether to switch from the current target object image to a different target object image.

If it is determined in step S409 not to switch to a different target object image (No in step S409), the process returns to step S402.

On the other hand, if it is determined in step S409 to switch to a different target object image (in the case of Yes in step S409), the process advances to step S410.

In step S410, the update unit 310 of the reinforcement learning device 120 determines whether the conditions for ending the reinforcement learning process are satisfied. Note that the termination condition for the reinforcement learning process is, for example, a condition defined by the user of the reinforcement learning system 100, and includes, for example, the target success probability of a grasping operation on a predetermined object.

In step S410, if it is determined that the termination condition for the reinforcement learning process is not satisfied (in the case of No in step S410), the process returns to step S401.

On the other hand, if it is determined in step S410 that the conditions for ending the reinforcement learning process are satisfied (in the case of Yes in step S410), the reinforcement learning process is ended. Note that the reinforcement learning model 330 after the reinforcement learning process is a device (object manipulation device) that outputs information for controlling the operation of the gripping mechanism section 111 to the drive control device 115 as a reinforcement learning model. ).

The reinforcement learning model applied to the object manipulation device executes the processes of steps S401 to S405 in FIG. 4 (that is, it does not acquire information indicating changes in state, calculate rewards, update model parameters, etc.) . Further, in step S404, the configuration is such that optimal information is output as information indicating the state of the gripping mechanism section 111 after the operation. That is, unlike during reinforcement learning processing, the gripping mechanism unit 111 performs an optimal motion selected from a set of possible motions instead of exhaustively performing various motions.

<Execution example of reinforcement learning processing>
Next, an example of execution of reinforcement learning processing by the reinforcement learning system 100 will be described. 5 and 6 are first and second diagrams showing an example of execution of reinforcement learning processing. When the target object image 510 shown in FIG. 5A is input by the user, the reinforcement learning device 120 recognizes the object 511 included in the target object image 510 as an object to be grasped.

In this way, by specifying the type of the object to be grasped by inputting the target object image 510, the reinforcement learning device 120 allows the user to select any object included in the object group 130 as the object to be grasped. can be specified as an object.

In FIG. 5B, an arrow 500 indicates the position and orientation (photographing position and photographing direction) of the imaging device 112 at the time when the object 511 is recognized as the object to be grasped. Further, a photographed image 521 shows a photographed image when the object group 130 is photographed under the position and orientation shown by the arrow 500.

As shown in the photographed image 521, when the object group 130 is photographed from above (Z-axis direction), the object 511 to be grasped is blocked by another object 512, and the imaging device 112 cannot photograph the object 511. I can't. That is, in this state, the object 511 cannot be gripped.

Therefore, the reinforcement learning device 120 outputs information indicating the state of the gripping mechanism 111 after operation in order to change the position and posture of the gripping mechanism 111 so that the object 511 to be gripped can be photographed. Thereby, the drive control device 115 controls the operation of the gripping mechanism 111 based on information indicating the state of the gripping mechanism 111 after the operation.

In FIG. 5C, an arrow 501 indicates the changed position and orientation (photographing position and photographing direction) of the imaging device 112, which are changed by controlling the operation of the gripping mechanism section 111. Further, a photographed image 522 shows a photographed image when the object group 130 is photographed under the position and orientation shown by the arrow 501.

As shown in the photographed image 522, by photographing the object group 130 from the lateral direction (X-axis direction), the object 511 to be grasped can be photographed.

For this reason, the reinforcement learning device 120 outputs information indicating the state of the gripping mechanism 111 after the operation in order to further change the position and orientation of the gripping mechanism 111 so that the object 511 to be gripped can be gripped. . Thereby, the drive control device 115 controls the operation of the gripping mechanism 111 based on information indicating the state of the gripping mechanism 111 after the operation.

In FIG. 6A, an arrow 601 indicates the changed position and orientation (photographing position and photographing direction) of the imaging device 112, which are changed by controlling the operation of the manipulator 110. Further, a photographed image 611 shows a photographed image when the object group 130 is photographed under the position and orientation shown by the arrow 601.

As shown in the photographed image 611, the object 511 to be gripped can be gripped by approaching the object 511 to be gripped.

For this reason, the reinforcement learning device 120 outputs information indicating the state of the gripping mechanism 111 after the operation in order to cause the gripping mechanism 111 to grip the object 511 to be gripped. Thereby, the drive control device 115 controls the operation of the gripping mechanism 111 based on information indicating the state of the gripping mechanism 111 after the operation.

In FIG. 6(b), an arrow 602 indicates the position and orientation (photographing position and photographing direction) of the imaging device 112 after the change, which is changed by controlling the operation of the gripping mechanism section 111 (the object 511 is (It is shown being grasped and lifted to a predetermined height). Further, a photographed image 612 shows a photographed image when the object 511 is photographed under the position and orientation shown by the arrow 602.

In this way, the position and orientation of the imaging device 112 are configured to change in accordance with changes in the position and orientation of the gripping mechanism section 111, and reinforcement learning is performed using images taken by the imaging device. By doing
・It is possible to evaluate from a long-term perspective, such as changes in the appearance from the imaging device.
- In the process of attempting the action of grasping the object to be grasped, it is possible to attempt the action of searching for the object to be grasped.

That is, in the process of reinforcement learning, the operation of the gripping mechanism can be controlled so that the object to be gripped can be photographed. As a result, the success probability of the grasping operation can be increased regardless of the placement state of the object to be grasped.

<Summary>
As is clear from the above description, the reinforcement learning system 100 according to the first embodiment is
- A captured image taken by an imaging device whose position and orientation change as the position and orientation of the gripping mechanism changes, and a target object image showing the object to be gripped by the gripping mechanism, are transferred to the gripping mechanism. This information is input to a reinforcement learning model that outputs information indicating the state of the part after its operation.
・When the operation of the gripping mechanism is controlled based on the information indicating the state after the operation of the gripping mechanism, the result of the operation on the object to be gripped (the result of determining whether the gripping operation by the end effector was successful) Update the model parameters of the reinforcement learning model based on the information.

As a result, according to the reinforcement learning system 100, even if the object to be gripped is placed so as to be shielded, the gripping mechanism unit is configured such that the object to be gripped can be photographed during the reinforcement learning process. The movement can be controlled.

In other words, according to the first embodiment, there is a reinforcement learning device, a reinforcement learning system, and an object manipulation device that can increase the probability of success in grasping a specified type of object regardless of the placement state. , a model generation method and a reinforcement learning program can be provided.

[Second embodiment]
In the second embodiment, a case will be described in which reinforcement learning is performed using Q learning. The second embodiment will be described below, focusing on the differences from the first embodiment.

<Functional configuration of reinforcement learning device>
First, an example of the functional configuration of the reinforcement learning device 120 according to the second embodiment will be described. FIG. 7 is a second diagram showing an example of the functional configuration of the reinforcement learning device. As shown in FIG. 7, the reinforcement learning device 120 according to the second embodiment includes an update section 710 and a reinforcement learning model 720. For example, a neural network may be used as the reinforcement learning model 720.

The update unit 710 includes a reward calculation unit 711 and a parameter update unit 712, and updates model parameters of the reinforcement learning model 720.

Specifically, the updating unit 710 acquires a determination result as to whether or not the gripping operation on the object to be gripped has been successful, and information indicating a change in the state due to the controlled operation of the gripping mechanism unit 111. .

Further, the reward calculation unit 711 calculates a reward based on the determination result of whether or not the gripping operation on the object to be gripped is successful. Note that the method for determining whether or not the gripping operation on the object to be gripped has been successful and the method for calculating the reward have already been explained in the first embodiment, and therefore will not be described here.

Furthermore, the parameter update unit 712 updates each model parameter of the image analysis unit 721, state and action input unit 722, and expected value calculation unit 724 included in the reinforcement learning model 720. Note that the parameter update unit 712
- Information indicating a change in state acquired by the update unit 710;
・Remuneration calculated by the remuneration calculation unit 711 (immediate remuneration),
- A predicted value of the expected value (Q value) of the discount cumulative reward calculated by the expected value calculation unit 724, which will be described later;
Update model parameters based on.

The model parameters of the reinforcement learning model 720 are updated by the updating unit 710. In addition, the reinforcement learning model 720 after the model parameters have been updated receives the captured image, the target object image, and the information indicating the state (s) before the operation of the gripping mechanism 111 as input, and uses the information after the operation of the gripping mechanism 111 as input. Outputs information indicating the status.

Specifically, as shown in FIG. 7, the reinforcement learning model 720 includes an image analysis section 721, a state and action input section 722, an addition section 723, an expected value calculation section 724, and an adjustment section 725.

The image analysis unit 721 executes processing by acquiring the captured image transmitted from the drive control device 115 and the target object (g) image input by the user, and outputs the execution result to the addition unit 723. Note that the image analysis unit 721 is configured using, for example, a neural network. More specifically, the image analysis unit 721 includes, for example, a first convolution layer, a first MaxPooling layer, a second convolution layer, a second MaxPooling layer, and the like.

The state and operation input unit 722 executes processing by acquiring information indicating the state (s) of the gripping mechanism 111 before operation and information indicating the operation (a) of the gripping mechanism 111, and outputs the execution result. is output to the adding section 723. Note that the state and operation input unit 722 is configured using, for example, a neural network. More specifically, the state and operation input section 722 is configured of a first linear layer, a second linear layer, a shape conversion layer, and the like. In addition, the state and operation input section 722 inputs the operation (a) of the gripping mechanism section 111 adjusted by the adjustment section 725 in order to search for the maximum Q value calculated by the expected value calculation section 724, which will be described later. The information shown is input a predetermined number of times (for example, 20 times).

The adding unit 723 adds the execution result output from the image analysis unit 721 and the execution result output from the state and action input unit 722 and inputs the result to the expected value calculation unit 724.

The expected value calculation unit 724 executes processing by inputting the execution result of the image analysis unit 721 and the execution result of the state and action input unit 722, which are added in the addition unit 723, and calculates the Q value (Q( s, a, g)). The expected value calculation unit 724 calculates a number of Q values corresponding to the number of pieces of information indicating the operation (a) of the gripping mechanism unit 111 adjusted by the adjustment unit 725. Note that the expected value calculation unit 724 is configured using, for example, a neural network. More specifically, the expected value calculation unit 724 includes a first convolution layer, a first MaxPooling layer, a second convolution layer, a second MaxPooling layer, and the like.

The adjustment unit 725 adjusts information indicating the operation (a) of the gripping mechanism unit 111 every time the expected value calculation unit 724 calculates the Q value, and inputs the information to the state and operation input unit 722. The adjustment unit 725 adjusts the information indicating the operation (a) of the gripping mechanism unit 111 a predetermined number of times (for example, 20 times), and extracts the maximum Q value from among the Q values calculated during that time. Note that the adjustment unit 725 specifies information indicating one of the movements (a) from a set of possible movements of the gripping mechanism unit 111, for example, based on the ε-greedy method.

According to the ε-greedy method, information indicating the action (a) corresponding to the maximum Q value may be identified, or information indicating a randomly selected action (a) may be identified. .

Further, the adjustment unit 725 adjusts the state after the operation of the gripping mechanism 111 based on the information indicating the specified operation (a) of the gripping mechanism 111 and the information indicating the state (s) before the operation of the gripping mechanism 111. The information indicating the state of is derived and transmitted to the drive control device 115.

In this way, the reinforcement learning device 120 according to the second embodiment uses the ε-greedy method to comprehensively output various information as information indicating the state of the gripping mechanism section 111 after operation. I can do it. As a result, the operation of the gripping mechanism unit 111 during the reinforcement learning process includes an optimal operation (operation with the maximum Q value) selected from a set of possible operations, and an operation selected from a set of possible operations. Randomly selected actions will be included.

Note that the functional configuration shown in FIG. 7 is merely an example of the configuration of the reinforcement learning model 720 that implements this function, and the reinforcement learning model 720 may be configured with other functional configurations. For example, in the above description, the image analysis section 721, the state and action input section 722, and the expected value calculation section 724 are each configured using a neural network, but the entire reinforcement learning model 720 is configured using a neural network. may be configured.

Further, in the above description, the functions during the reinforcement learning process have been mentioned, but the functions after the reinforcement learning process is the same as those in the first embodiment. That is, after the reinforcement learning process is completed, the updating unit 710 does not acquire information indicating a change in state, calculate a reward, update model parameters, etc. The adjustment unit 725 also uses optimal information (the operation of the gripping mechanism 111 derived based on information indicating the operation (a) with the maximum Q value) as information indicating the state of the gripping mechanism 111 after the operation. information indicating the subsequent state) is output. Thereby, according to the model that has undergone reinforcement learning, it is possible to obtain a behavior rule that maximizes the expected value (Q value) of the discount cumulative reward.

<Flow of reinforcement learning processing>
Next, the flow of reinforcement learning processing by the reinforcement learning device 120 according to the second embodiment will be described. FIG. 8 is a second flowchart showing the flow of reinforcement learning processing. The flow of reinforcement learning processing will be described below with reference to FIG. 8. Note that the reinforcement learning process shown in FIG. 8 is just an example, and a reinforcement learning model may be generated by executing the reinforcement learning process using another model generation method.

In step S801, the reinforcement learning model 720 of the reinforcement learning device 120 acquires a target object image.

In step S802, the reinforcement learning model 720 of the reinforcement learning device 120 acquires a photographed image.

In step S803, the reinforcement learning model 720 of the reinforcement learning device 120 acquires information indicating the pre-operation (current) state (s) of the gripping mechanism section 111.

Steps S804 to S807 identify information indicating one of the motions (a) from a set of possible motions based on the ε-greedy method, and indicate the state of the gripping mechanism 111 after the motion. Output information comprehensively.

Specifically, when identifying information indicating the action (a) corresponding to the optimal Q value from a set of possible actions, steps S804 to S806 are executed, and then step S807 is performed. move on. Further, in the case of specifying information indicating a randomly selected action (a) from a set of possible actions, the process directly advances to step S807.

In step S804, the reinforcement learning model 720 of the reinforcement learning device 120 calculates a Q value.

In step S805, the reinforcement learning model 720 of the reinforcement learning device 120 determines whether the Q value has been calculated a predetermined number of times. If it is determined in step S805 that the Q value has not been calculated a predetermined number of times (No in step S805), the process advances to step S806.

In step S806, the reinforcement learning model 720 of the reinforcement learning device 120 adjusts the information indicating the operation (a) of the gripping mechanism section 111, and returns to step S804.

On the other hand, if it is determined in step S805 that the Q value has been calculated a predetermined number of times (in the case of Yes in step S805), the process advances to step S807.

In step S807, if steps S804 to S807 have been executed, the reinforcement learning model 720 of the reinforcement learning device 120 specifies information indicating the motion (a) corresponding to the maximum Q value, and After deriving information indicating the state after the operation, the information is transmitted to the drive control device 115. Furthermore, in the case where steps S804 to S807 are not executed, the reinforcement learning model 720 of the reinforcement learning device 120 specifies information indicating the randomly selected motion (a), and after the operation of the gripping mechanism section 111 After deriving information indicating the state, the information is transmitted to the drive control device 115.

In step S808, the updating unit 710 of the reinforcement learning device 120 acquires information indicating a change in state due to the controlled operation of the gripping mechanism unit 111.

In step S809, the updating unit 710 of the reinforcement learning device 120 obtains a determination result as to whether or not the grasping operation on the object to be grasped is successful, and calculates an immediate reward. Further, the updating unit 710 of the reinforcement learning device 120 obtains the predicted value of the expected value (Q value) of the discount cumulative reward calculated by the expected value calculating unit 724.

In step S810, the updating unit 710 of the reinforcement learning device 120 uses the acquired information indicating the state change, the calculated immediate reward, and the predicted value of the expected value (Q value) of the acquired discounted cumulative reward to update the reinforcement learning model. 720 model parameters are updated.

In step S811, the reinforcement learning device 120 determines whether to switch from the current target object image to a different target object image.

If it is determined in step S811 not to switch to a different target object image (No in step S811), the process returns to step S802.

On the other hand, if it is determined in step S811 to switch to a different target object image (in the case of Yes in step S811), the process advances to step S812.

In step S812, the update unit 310 of the reinforcement learning device 120 determines whether the termination condition for the reinforcement learning process is satisfied. Note that the termination condition for the reinforcement learning process is, for example, a condition defined by the user of the reinforcement learning system 100, and includes, for example, the target success probability of a grasping operation on a predetermined object.

In step S812, if it is determined that the termination condition for the reinforcement learning process is not satisfied (in the case of No in step S812), the process returns to step S801.

On the other hand, if it is determined in step S812 that the conditions for ending the reinforcement learning process are satisfied (in the case of Yes in step S812), the reinforcement learning process is ended. Note that the reinforcement learning model 720 after the reinforcement learning process is applied to the object manipulation device as a reinforcement learning completed model.

The reinforcement learning model applied to the object manipulation device executes the processes of steps S801 to S807 in FIG. 8 (that is, it does not acquire information indicating changes in state, calculate rewards, update model parameters, etc.) . Further, in step S807, the configuration is such that optimal information is output as information indicating the state of the gripping mechanism section 111 after its operation. In other words, unlike during reinforcement learning processing, the gripping mechanism unit 111 performs an optimal motion (motion with the maximum Q value) selected from a set of possible motions, instead of exhaustively performing various motions. I do.

<Summary>
As is clear from the above description, the reinforcement learning system 100 according to the second embodiment provides the same effects as the first embodiment.

[Third embodiment]
In the first and second embodiments described above, a case has been described in which a gripping operation is performed on a specified type of object. However, the predetermined operation performed on the specified type of object is not limited to the grasping operation, and may be any other operation. That is, the end effector attached to the distal end portion of the main body 113 of the manipulator 110 is not limited to the gripping mechanism 111, but may be any other operating mechanism. The arbitrary operations mentioned here include, for example, a pressing operation to press a specified type of object, a suction operation to attract a specified type of object, a suction operation to attract a specified type of object with an electromagnet, etc. is included.

Furthermore, in the first and second embodiments described above, the imaging device is attached to the tip of the manipulator, but the mounting position of the imaging device is not limited to the tip of the manipulator. Any other position may be used as long as the position and orientation of the imaging device change in accordance with changes in the position and orientation of the gripping mechanism.

Note that the gripping mechanism section and the imaging device may be attached to different manipulators, for example, and the above-described reinforcement learning model can be applied in that case as well. The reinforcement learning model in this case may be configured to output information for controlling at least one of the position and orientation of the imaging device in addition to information for controlling the operation of the gripping mechanism.

In addition, in the first and second embodiments described above, information indicating the position and orientation of the grasping mechanism, information indicating the opening and closing of the grasping mechanism, and information indicating the state before operation of the grasping mechanism are input to the reinforcement learning model. The explanation has been made assuming that the information shown is included. However, the information indicating the state of the gripping mechanism before operation is not limited to these, and other information may be input.

Furthermore, in the first and second embodiments described above, the manipulator 110 and the reinforcement learning device 120 (or the object manipulation device) are configured as separate bodies, but the manipulator 110 and the reinforcement learning device 120 (or the object manipulation device) are different from each other. It may also be configured as one piece. Alternatively, the drive control device 115 and the reinforcement learning device 120 (or object manipulation device) may be configured as one unit.

Furthermore, in the first and second embodiments described above, the operation of the gripping mechanism section 111 is actually controlled based on the information output from the reinforcement learning device 120 indicating the state after the operation of the gripping mechanism section 111. This was explained as performing reinforcement learning processing. However, it is not necessary to actually control the operation of the gripping mechanism section 111, and the reinforcement learning process may be performed using a simulator that simulates a real environment. In this case, the imaging device may also be configured to change its position and orientation, or to take images on a simulator that simulates a real environment. Further, the predetermined operation on the object to be operated and the generation of the operation result may also be configured to be performed on a simulator that simulates the real environment.

Furthermore, in the first and second embodiments, the reinforcement learning device 120 performs reinforcement learning processing in the case where the end effector is attached to the tip of the main body 113 of the manipulator 110. However, the reinforcement learning device 120 may perform reinforcement learning processing in a case where the manipulator 110, in which the end effector is not attached to the tip portion, operates the object to be manipulated using the main body portion 113. In this case, the reinforcement learning device 120 may output information for controlling the operation of the tip portion of the main body 113 of the manipulator 110.

Furthermore, in the first and second embodiments, the position and orientation of the tip portion of the main body 113 of the manipulator 110 are configured to change, but at least one of the position and orientation changes. It may be configured to change. In other words, the gripping mechanism section 111 may be configured to change at least one of its position and orientation. Further, the imaging device 112 may be configured such that at least one of the position and orientation of the gripping mechanism section 111 changes as at least one of the position and orientation of the gripping mechanism 111 changes. In this case, in the reinforcement learning device 120, the information for controlling the operation of the grasping mechanism section 111 includes information for controlling at least one of the position and orientation of the grasping mechanism section 111, and information for controlling the operation of the grasping mechanism section 111. Information for controlling opening/closing may be output.

[Other embodiments]
In this specification (including claims), the expression "at least one (one) of a, b, and c" or "at least one (one) of a, b, or c" (including similar expressions) When used, it includes any of a, b, c, a-b, a-c, b-c, or a-b-c. Further, each element may include multiple instances, such as aa, abb, aaabbbcc, etc. Furthermore, it also includes adding other elements other than the listed elements (a, b, and c), such as having d as in abcd.

In addition, in this specification (including claims), when expressions such as "as input data/based on data/according to/according to" (including similar expressions) are used, there is no specific notice. This includes cases in which various data itself is used as input, and cases in which various data subjected to some processing (for example, noise added, normalized, intermediate representation of various data, etc.) are used as input. In addition, if it is stated that a certain result is obtained "based on/according to/according to data", this includes cases where the result is obtained only based on the data, and other data other than the data, It may also include cases where the results are obtained under the influence of factors, conditions, and/or states. In addition, if it is stated that "data will be output", if there is no special notice, various data itself may be used as output, or data that has been processed in some way (for example, data with added noise, normal This also includes cases in which the output is digitized data, intermediate representations of various data, etc.).

In addition, in this specification (including claims), when the terms "connected" and "coupled" are used, direct connection/coupling, indirect connection /coupling, electrically connected/coupled, communicatively connected/coupled, functionally connected/coupled, physically connected/coupled, etc., but not limited to intended as a descriptive term. The term should be interpreted as appropriate depending on the context in which the term is used, but forms of connection/coupling that are not intentionally or naturally excluded are not included in the term. Should be construed in a limited manner.

In addition, in this specification (including the claims), when the expression "A configured to B" is used, it means that the physical structure of element A performs operation B. possible configuration and that the permanent or temporary setting/configuration of element A is configured/set to actually perform action B. may be included. For example, if element A is a general-purpose processor, the processor has a hardware configuration that can execute operation B, and can perform operation B by setting a permanent or temporary program (instruction). It only needs to be configured to actually execute. In addition, if element A is a dedicated processor or a dedicated arithmetic circuit, the circuit structure of the processor is configured to actually execute operation B, regardless of whether control instructions and data are actually attached. It is sufficient if it has been implemented.

In addition, in this specification (including claims), when terms meaning inclusion or ownership (for example, "comprising/including" and "having", etc.) are used, the purpose of the term is It is intended as an open-ended term, including the case of containing or possessing something other than the object indicated by the word. If the object of a term meaning inclusion or possession is an expression that does not specify a quantity or suggests a singular number (an expression with a or an as an article), the expression shall be interpreted as not being limited to a specific number. It should be.

In addition, in this specification (including the claims), expressions such as "one or more" or "at least one" are used in some places, and in other places, quantities are used. Even if an expression is used that does not specify or suggests the singular (an expression with a or an as an article), it is not intended that the latter expression means "one". In general, expressions that do not specify a quantity or imply a singular number (expressions with the article a or an) should be construed as not necessarily being limited to a particular number.

In addition, in this specification, if it is stated that a specific effect (advantage/result) can be obtained with a specific configuration of a certain embodiment, unless there is a reason to the contrary, another example having the configuration It should be understood that the same effect can also be obtained in a plurality of embodiments. However, it should be understood that the presence or absence of the said effect generally depends on various factors, conditions, and/or states, and that the said effect is not necessarily obtained by the said configuration. The effect is only obtained by the configuration described in the Examples when various factors, conditions, and/or states, etc. are satisfied, and in the claimed invention that specifies the configuration or a similar configuration. However, this effect is not necessarily obtained.

In addition, in this specification (including claims), when multiple pieces of hardware perform a predetermined process, each piece of hardware may cooperate to perform the predetermined process, or some of the hardware may perform the predetermined process. You may perform all of the above processing. Further, some hardware may perform part of a predetermined process, and another piece of hardware may perform the rest of the predetermined process. In this specification (including claims), when expressions such as "one or more hardware performs the first process, and the one or more hardware performs the second process" are used , the hardware that performs the first processing and the hardware that performs the second processing may be the same or different. In other words, the hardware that performs the first processing and the hardware that performs the second processing may be included in the one or more pieces of hardware. Note that the hardware may include an electronic circuit, a device including an electronic circuit, or the like.

In addition, in this specification (including claims), when multiple storage devices (memories) store data, each storage device (memory) among the multiple storage devices (memories) stores a portion of the data. Only the data may be stored, or the entire data may be stored.

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, changes, substitutions, and partial deletions are possible without departing from the conceptual idea and gist of the present invention derived from the content defined in the claims and equivalents thereof. For example, in all the embodiments described above, the numerical values used in the explanation are shown as examples, and the present invention is not limited to these. Further, the order of each operation in the embodiment is shown as an example, and is not limited to this order.

100: Reinforcement learning system 110: Manipulator 111: Gripping mechanism section 112: Imaging device 113: Main body section 115: Drive control device 120: Reinforcement learning device 310: Updating section 311: Reward calculation section 320: State input section 330: Reinforcement learning model 510: Target object image 511: Objects 521, 522: Captured images 611, 612: Captured image 710: Update unit 711: Reward calculation unit 712: Parameter update unit 720: Reinforcement learning model 721: Image analysis unit 722: State and motion input Section 723: Addition section 724: Expected value calculation section 725: Adjustment section

Claims (13)

at least one memory;
at least one processor;
The at least one processor includes:
The operation of the end effector is controlled by using information regarding a photographed image photographed by an imaging device whose position or orientation changes at least, and information regarding a target object image indicating an object to be operated by the end effector. inputting information into a training model that outputs information for
and updating the parameters of the training model based on the operation result on the object when the operation of the end effector is controlled based on the information output by the training model.
Reinforcement learning device. The at least one of the position and orientation of the imaging device changes depending on at least one of the position and orientation of the end effector.
The reinforcement learning device according to claim 1. the imaging device is attached to the end effector;
The reinforcement learning device according to claim 2. at least one of the position and orientation of the imaging device is controlled based on an output from the training model;
The reinforcement learning device according to claim 1. The end effector is a gripping mechanism that grips the object,
5. The at least one processor updates parameters of the training model based on a determination result of whether or not the gripping mechanism unit has successfully gripped the object. Reinforcement learning device described. 5. The at least one processor inputs into the training model information regarding at least one of the position and orientation of the gripping mechanism before operation, and information regarding opening and closing of the gripping mechanism before operation. Reinforcement learning device described in. 7. The reinforcement learning device according to claim 6, wherein the training model outputs information regarding at least one of the position and orientation of the gripping mechanism after the operation, and information regarding opening/closing of the gripping mechanism after the operation. The reinforcement learning device according to any one of claims 1 to 7, wherein a predetermined operation on the object by the end effector and a change in at least one of the position and orientation of the imaging device are executed on a simulator. A reinforcement learning device according to any one of claims 1 to 8,
a manipulator to which the end effector and the imaging device are attached;
A reinforcement learning system with at least one memory that stores a training model whose parameters have been updated by the reinforcement learning device according to any one of claims 1 to 8;
at least one processor;
The at least one processor includes:
Inputting into the training model information regarding a photographed image photographed by an imaging device whose position or orientation changes, and information regarding a target object image indicating an object to be operated by the end effector. and,
controlling the operation of the end effector based on information output by the training model;
configured to be executable,
Object manipulation device. the end effector;
the imaging device;
The object manipulation device according to claim 10, further comprising: A model generation method executed by at least one processor, the method comprising:
The operation of the end effector is controlled by using information regarding a photographed image photographed by an imaging device whose position or orientation changes at least, and information regarding a target object image indicating an object to be operated by the end effector. inputting information into a training model that outputs information for the purpose;
a step of updating parameters of the training model based on an operation result on the object when the operation of the end effector is controlled based on information output by the training model. The operation of the end effector is controlled by using information regarding a photographed image photographed by an imaging device whose position or orientation changes at least, and information regarding a target object image indicating an object to be operated by the end effector. inputting information into a training model that outputs information for the purpose;
and causing at least one computer to execute a step of updating the parameters of the training model based on the operation result on the object when the operation of the end effector is controlled based on the information output by the training model. Reinforcement learning program for.

Images