JP2021091067A - Information processing device, method and program - Google Patents

Abstract

To improve both the accuracy of generating an orbit to be traced by a robot arm and the accuracy of generating processing operation to an object of the robot arm.SOLUTION: An information processing device (40) includes: an orbit generation unit (41) for generating orbit information showing an orbit to be traced by a robot arm (10) by using first image data generated by a first camera (20), state information on a state of the robot arm, and a first learned model (43); and a processing operation generation unit (42) for generating processing operation information showing processing operation to be performed by the robot arm (10) by using second image data generated by a second camera (30), state information and a second learned model (44).SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, a method and a program for controlling a robot arm.

Robot arms that perform various processing operations on an object such as gripping an article or painting are known. As the control of such a robot arm, the control of moving the robot arm to the vicinity of the object and the control of the processing operation of the robot arm with respect to the object are performed. In addition, a technique for generating a trajectory related to the movement of a robot arm by machine learning is being researched. For example, Non-Patent Document 1 discloses an algorithm called MPNet (Motion Planning Networks) for generating a trajectory of a robot arm by machine learning. In MPNet, the trajectory of the robot arm is generated using an image of the working space of the robot arm taken by a camera.

Ahmed H. Qureshi, Anthony Simeonov, Mayur J. Bency and Michael C. Yip, Motion Planning Networks, 13 Jul, 2019

Although the technique described in Patent Document 1 can calculate the trajectory of the robot arm from the current position to the target position, the accuracy of the processing operation of the robot arm with respect to the object (grasping an article, painting, etc.) can be obtained. It was difficult to improve.

One aspect of the present invention realizes an information processing apparatus, method and program capable of improving both the accuracy of generating a trajectory to be traced by the robot arm and the accuracy of generating a processing motion with respect to an object of the robot arm. The purpose is.

In order to solve the above problems, the information processing apparatus according to one aspect of the present invention uses the first image data generated by the first camera, the state information regarding the state of the robot arm, and the first trained model. Using the orbit generation unit that generates orbit information indicating the orbit to be traced by the robot arm, the second image data generated by the second camera, the state information, and the second trained model, the robot It includes a processing operation generation unit that generates processing operation information indicating the processing operation to be performed by the arm.

According to one aspect of the present invention, both the accuracy of generating the trajectory to be traced by the robot arm and the accuracy of generating the processing motion with respect to the object of the robot arm can be improved.

It is a block diagram which shows the schematic structure of the control system which concerns on one Embodiment of this invention. It is a figure which shows typically the appearance of the control system which concerns on one Embodiment of this invention. It is a flowchart which shows an example of the control flow of the robot arm by the control system which concerns on one Embodiment of this invention. It is a figure which illustrates the trajectory of a robot arm. It is a block diagram which shows the schematic structure of the control system which concerns on one Embodiment of this invention. It is a block diagram which illustrates the physical structure of the information processing apparatus which concerns on one Embodiment of this invention.

[Embodiment 1]
FIG. 1 is a block diagram showing a schematic configuration of a control system 1 according to an embodiment of the present invention. The control system 1 is a system for controlling the robot arm 10. The control system 1 includes a robot arm 10, a first camera 20, a second camera 30, and an information processing device 40. The robot arm 10 is a device that performs processing operations related to an object. Processing operations relating to the object are, for example, gripping, painting, polishing, cutting, drilling, or assembling an article.

The first camera 20 and the second camera 30 are cameras that photograph the work space of the robot arm 10.
The first camera 20 can capture, for example, a work space of the robot arm 10 and the robot arm 10 so that the robot arm 10 can take an image suitable for generating information indicating the trajectory to be traced (hereinafter referred to as “orbit information”). Is fixed at a position included in the shooting range. The first camera 20 outputs the first image data representing the captured image.

The second camera 30, for example, at least a part of the periphery of the robot arm 10 so that the second camera 30 can take an image suitable for generating information indicating the processing operation to be performed by the robot arm 10 (hereinafter referred to as “processing operation information”). (For example, the front) is fixed at a position included in the shooting range. The second camera 30 outputs the second image data representing the captured image.

The information processing device 40 includes an orbit generation unit 41, a processing operation generation unit 42, a first trained model 43, and a second trained model 44.

The trajectory generation unit 41 generates trajectory information using the first image data generated by the first camera 20, information on the state of the robot arm 10 (hereinafter referred to as “state information”), and the first trained model 43. To do. The state information is arbitrary information regarding the state of the robot arm 10. The state information includes, but is not limited to, information indicating the joint angles of the joint 12 and the end effector 11 and the coordinates of each part of the robot arm 10 including the end effector 11. The trajectory information is information indicating the trajectory (change in state) traced by each part of the robot arm 10 until the robot arm 10 moves from the current position (start point of the trajectory) to the target position (end point of the trajectory). The trajectory information can be time-series data of the state information of the robot arm 10 from the current position to the target position.

Further, in the following description, the current position of the robot arm 10 refers to the position of the robot arm 10 and the state of the robot arm 10 at each processing time. The target position refers to the state of the robot arm 10 when the robot arm 10 moves to the periphery of the object to be processed. The target position is, for example, the state of the robot arm 10 when the robot arm 10 moves to a position where the processing operation can be started.

The processing motion generation unit 42 generates processing motion information using the second image data generated by the second camera 30, the state information of the robot arm 10, and the second trained model.

The first trained model 43 is learned by machine learning the correlation between the state information of the robot arm 10 at the current position and the target position, the information about the work space of the robot arm 10 (hereinafter referred to as “spatial information”), and the trajectory information. It is a finished model. In this embodiment, the spatial information includes, for example, the coordinates of an obstacle included in the work space of the robot arm 10 and information indicating the coordinates of an object to be processed by the robot arm 10. Spatial information is specified, for example, from a captured image of the work space of the robot arm 10. The first trained model 43 is, for example, MPNet, but is not limited thereto. The first trained model 43 can be any machine learning model capable of generating orbital information based on state information and spatial information. For example, the first trained model 43 can be realized by CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), RSTM (Long Short-Term Memory), DNN (Deep Neural Network), or a combination thereof.

The second trained model 44 is a model different from the first trained model 43, and is a trained model in which the correlation between the second image data and the state information and the processing operation information is machine-learned. The second trained model 44 can be any machine learning model capable of generating processing operation information from the second image data and state information. For example, the second trained model 44 can be realized by CNN, RNN, LSTM, DNN, or a combination thereof.

The information processing device 40 configured as described above generates trajectory information of the robot arm 10 using the first trained model 43, and also uses the second trained model 44 to generate processing operation information of the robot arm 10. Generate. In this way, by using a trained model suitable for each of the generation of the trajectory information and the generation of the processing motion information, it is possible to improve the accuracy of both the generation of the trajectory information and the generation of the processing motion information. Further, in this embodiment, it is possible to improve the accuracy of both the generation of the trajectory information and the generation of the processing operation information by using the captured images suitable for each of the generation of the trajectory information and the generation of the processing operation information. it can.

[Embodiment 2]
(Constitution)
FIG. 2 is a diagram schematically showing the appearance of the control system 1 according to the embodiment of the present invention. For convenience of explanation, the same reference numerals will be added to the members having the same functions as the members described in the first embodiment, and the description will not be repeated.

The control system 1 is a system that controls a series of operations in which the robot arm 10 is moved to the vicinity of the object 50 to grip the object 50. In the present embodiment, an end effector 11 is provided at the end of the robot arm 10. The end effector 11 is, for example, a multi-finger hand or a gripper. In this embodiment, the end effector 11 has a mechanism for gripping an article. The processing operation performed by the robot arm 10 is not limited to gripping the article, and may be configured so that another processing operation can be performed by exchanging the end effector 11.

The robot arm 10 includes one or more joints 12, and operates by driving each joint 12. The joint 12 may be a joint of an arm or a joint of an end effector 11 (for example, a multi-fingered hand). The robot arm 10 also includes one or more sensors 13 (see FIG. 1). Each sensor 13 may include, for example, an angle sensor that detects the joint angle of each joint 12, a force sensor that detects a force sense at a specific position of the robot arm 10, and the like. The detection result of each sensor 13 is output to the information processing device 40.

The first camera 20 photographs the work space of the robot arm 10 and generates photographing data (first image data) representing the photographed image. The first camera 20 is fixed at a position where the robot arm 10 and the work space of the robot arm 10 are included in the photographing range. In the present embodiment, the first camera 20 includes a depth sensor and outputs a captured image represented by an RGB color space (hereinafter referred to as “RGB image”) and captured data representing a depth image having distance data for each pixel. .. The captured image is not limited to a color image and may be a monochrome image. Although one first camera 20 is shown in FIG. 2, a plurality of cameras may be provided as the first camera 20 for photographing the work space of the robot arm 10.

The second camera 30 is a camera fixed to the robot arm 10 and outputs shooting data (second image data) representing a shot image. The second camera 30 is provided at a position (for example, the fifth axis of the robot arm 10) in which at least a part of the end effector 11 and the periphery of the end effector 11 are included in the photographing range. For example, the second camera 30 is fixed in a direction in which the axial direction of the fifth axis of the robot arm 10 and the photographing direction are parallel to each other. The second camera 30 may include a depth sensor. In this case, the shooting data output by the second camera 30 may include data representing a depth image. Further, although one second camera 30 is shown in FIG. 2, a plurality of cameras may be provided as the second camera 30. Further, the second camera 30 may be fixed to the fourth axis or the sixth axis of the robot arm 10.

The object 50 is an object of a processing operation performed by the robot arm 10, and is, for example, an article held by the robot arm 10. The obstacles 61 and 62 are arbitrary objects other than the object 50 existing in the work space of the robot arm 10. Obstacles 61 and 62 can be obstacles in the movement of the robot arm 10 or the processing operation of the robot arm. Obstacles 61 and 62 are, for example, articles. In the example of FIG. 2, two obstacles 61 and 62 are shown, but the number of obstacles included in the work space of the robot arm 10 is not limited to 2, and may be more or less than this.

The information processing device 40 includes an orbit generation unit 41, a processing operation generation unit 42, a first learned model 43, a second learned model 44, a robot control unit 45, and an orbit storage unit 46.

The trajectory generation unit 41 uses the first image data, the state information, and the first trained model 43 to generate trajectory information indicating the trajectory of the robot arm 10 from the current position to the target position. When there is an obstacle in the work space, the trajectory generation unit 41 generates trajectory information indicating a route in which the robot arm 10 moves from the current position to the target position while avoiding the obstacles 61 and 62. Further, the trajectory generation unit 41 repeatedly executes the trajectory information generation process for a period from the current position to the target position of the robot arm 10. The generated orbit information is stored in the orbit storage unit 46.

The processing operation generation unit 42 generates processing operation information using the second image data, the state information, and the second trained model 44. The processing operation information can be, for example, time-series data of the state information of the robot arm 10. In this embodiment, the processing operation information generated by the processing operation generating unit 42 indicates the processing operation to be performed by the robot arm 10 after the robot arm 10 has moved to the target position along the trajectory generated by the trajectory generating unit 41. Information.

The first trained model 43 is a trained model in which the correlation between the state information and spatial information of the robot arm 10 at the current position and the target position and the trajectory information is machine-learned. In this embodiment, the spatial information includes the coordinate information of the obstacle and the coordinate information of the object. The input information of the first trained model 43 is not limited to the state information and the spatial information, and may include other information. For example, the input information may include the first image data. The first trained model 43 is pre-generated by the information processing device 40 or another device. The information processing device 40 or another device generates a first trained model 43 in which the correlation between the state information and the spatial information and the orbit information is machine-learned by using the pair of the state information and the spatial information and the orbit information. To do.

The second trained model 44 is pre-generated by the information processing device 40 or another device. In this embodiment, in the second trained model 44, the correlation between the second image data and the state information and the processing operation information when the end effector 11 performs the processing operation of gripping the object 50 is machine-learned. It is a trained model. The information processing device 40 or another device uses a pair of the second image data and the state information and the processing operation information to determine the correlation between the second image data and the state information and the processing operation information, for example, by machine learning. The trained second trained model 44 is generated.

The robot control unit 45 controls the robot arm 10 by using the trajectory information generated by the trajectory generation unit 41 and the processing operation information generated by the processing operation generation unit 42. That is, the robot control unit 45 changes the state of the robot arm 10 using the trajectory information, and controls the robot arm 10 so that the state of the robot arm 10 approaches the state indicated by the trajectory information. Further, the robot control unit 45 changes the state of the robot arm 10 by using the processing operation information.

The orbit storage unit 46 stores the orbit information generated by the orbit generation unit 41. In this embodiment, the orbit storage unit 46 stores the state information of the robot arm 10 as the orbit information in chronological order.

(motion)
FIG. 3 is a flowchart showing an example of the flow of control of the robot arm 10 by the control system 1. Note that some steps may be performed in parallel or in a different order.

In step S101, the work space is photographed by the first camera 20. The first camera 20 supplies the first image data representing the captured work space to the information processing device 40.

In step S102, the trajectory generation unit 41 recognizes an object (object and obstacle) existing in the work space by using the RGB image and the depth image included in the first image data. Various methods can be used as the object recognition method. For example, MMSS (Multi-model Sharable and Specific Feature Learning for RGB-D Object Recognition), which is a three-dimensional object recognition method using CNN, is used. .. In this case, by inputting the 3D data into the trained model in which the correlation between the 3D data (RGB image and depth image) and the type (label) of the object is machine-learned, the coordinates of the object existing in the work space can be obtained. The type (spatial information) of each object is estimated. The estimated spatial information can be output as a point cloud with the type (label) of each object. In the example of FIG. 2, the object 50, the obstacle 61, and the obstacle 62 are recognized as the objects existing in the work space. The type of obstacle does not have to be specified.

In step S103, the trajectory generation unit 41 determines the target position for movement of the robot arm 10. The target position is determined, for example, as follows. The trajectory generation unit 41 identifies an object from the objects existing in the work space based on the type of the object recognized in step S102. In this case, the type of the object may be preset by the user. In the example of FIG. 2, the object 50 is specified.

Next, the trajectory generation unit 41 determines the state (position) of the robot arm 10 in which the positional relationship between the object 50 and the end effector 11 of the robot arm 10 satisfies a predetermined condition as a movement target position. The predetermined condition may be, for example, a condition that the distance from the end effector 11 to the object 50 is equal to or less than a predetermined threshold value. The distance from the end effector 11 to the object 50 is, for example, the distance from a predetermined portion of the end effector 11 to the center of gravity of the object 50, or the shortest distance from a predetermined portion of the end effector 11 to the surface of the object 50. Is. That is, the target position of the robot arm 10 means, for example, a state of the robot arm 10 in which the shortest distance from a predetermined portion of the end effector 11 to the surface of the object 50 is equal to or less than a predetermined threshold value.

In step S104, the trajectory generating unit 41 includes state information and spatial information at the current position and target position of the robot arm 10 (coordinate information of the object 50 and obstacles 61 and 62, state information at the target position of the robot arm 10). , And the first trained model 43 is used to generate orbital information. In this embodiment, the trajectory generation unit 41 inputs the state information and the spatial information at the current position and the target position of the robot arm 10 into the first trained model 43, and the trajectory information output from the first trained model 43. To get. In this embodiment, the trajectory generation unit 41 may acquire the detection result of the sensor 13 as the current state information of the robot arm 10, or may calculate the state information from the detection result, or the first. The state information may be specified by performing image analysis on the image data. The trajectory generation unit 41 stores the acquired trajectory information in the trajectory storage unit 46, and supplies the trajectory information to the robot control unit 45.

In step S105, the robot control unit 45 controls the robot arm 10 according to the supplied trajectory information and moves the robot arm 10. That is, the robot control unit 45 controls the robot arm 10 so that the robot arm 10 moves along the trajectory indicated by the trajectory information. At this time, the detection result of the sensor 13 may be used for controlling the robot arm 10.

In step S106, the robot control unit 45 determines whether the robot arm 10 has moved to the target position by comparing the current position of the robot arm 10 with the target position. When the current position and the target position match, or the difference between the current position and the target position is less than the threshold value, the robot control unit 45 determines that the robot arm 10 has moved to the target position. On the other hand, when the difference between the current position and the target position is equal to or greater than the threshold value, the robot control unit 45 determines that the robot arm 10 has not moved to the target position.

When the robot arm 10 moves to the target position (step S106; YES), the information processing device 40 proceeds to the process of step S107. On the other hand, when it has not moved to the target position (step S106; NO), the information processing apparatus 40 returns to the process of step S101 and acquires the first image data from the first camera 20. That is, the information processing device 40 repeats the processes of steps S101 to S106 at predetermined time intervals until the robot arm 10 reaches the target position.

As described above, the processes of steps S101 to S106 are repeatedly executed until the robot arm 10 moves to the target position. That is, the trajectory generation unit 41 repeatedly executes the trajectory information generation process until the robot arm 10 moves to the target position. By repeatedly executing these processes, the robot arm 10 gradually moves in a trajectory that does not collide with the obstacle 61 and the obstacle 62, and approaches the object 50.

FIG. 4 is a diagram illustrating the trajectory of the end effector 11 of the robot arm 10. In the example of FIG. 4, the end effector 11 gradually moves along the trajectory generated so as not to collide with the obstacle 61 and the obstacle 62 under the control of the robot control unit 45, and approaches the object 50. .. The orbits q11 to q19 indicate the orbits of the end effector 11 generated by the orbit generation unit 41. In FIG. 4, only the trajectory of the end effector 11 is shown, and the trajectory of the entire robot arm 10 is omitted in order to prevent the drawing from becoming complicated.

When the robot arm 10 reaches the target position, the captured image used for control is switched from the captured image of the first camera 20 to the captured image of the second camera 30, and the processes of steps S107 to S110 are executed. In step S107, shooting is performed by the second camera 30, and the second image data of the second camera 30 is supplied to the information processing device 40.

In step S108, the processing motion generating unit 42 generates processing motion information using the state information at the current position of the robot arm 10, the second image data, and the second learned model 44. In this embodiment, the processing motion generation unit 42 inputs the state information and the second image data at the current position of the robot arm 10 into the second trained model 44, and the processing motion information output from the second trained model 44. To get. The acquired processing operation information is supplied to the robot control unit 45.

In step S109, the robot control unit 45 controls the robot arm 10, particularly the end effector 11, according to the processing operation information supplied, and causes the robot arm 10 to perform a gripping operation.

In step S110, the information processing device 40 determines whether the robot arm 10 has completed the processing operation. When the processing operation is completed (step S110; YES), the information processing apparatus 40 ends the processing of FIG. On the other hand, when the processing operation is not completed (step S110; NO), the information processing apparatus 40 returns to the processing in step S107 and continues the processing operation control using the second image data. That is, the information processing apparatus 40 repeatedly executes the processes of steps S107 to S109 at predetermined time intervals until the processing operation of the robot arm 10 is completed. By repeatedly executing these processes, the robot arm 10 performs a gripping operation of the object 50.

The control system 1 executes the process shown in FIG. 3 to control the robot arm 10, so that the robot arm 10 grips the object 50. After the robot arm 10 completes the gripping motion of the object 50, the robot arm 10 may perform another motion.

For example, the robot arm 10 may perform an operation of moving the object 50 to another place. In this case, the robot control unit 45 of the information processing device 40 controls the robot arm 10 by using the orbit information stored in the orbit storage unit 46, and the object 50 is held by the robot arm 10 while holding the object 50. To move. For example, the robot control unit 45 reads out the time-series state information stored in the trajectory storage unit 46 in the reverse order, and controls the robot arm 10 according to the read state information to change the state of the robot arm 10 from the target position to the initial position. It is possible to control the return to.

Further, the information processing device 40 sets a new target position and uses the current position of the robot arm 10 and the state information at the new target position, the spatial information acquired from the first image data, and the first trained model 43. Then, the trajectory information to the new target position may be generated, and the robot arm 10 may be moved along the trajectory information.

In the present embodiment, the information processing apparatus 40 generates the trajectory information of the robot arm 10 by using the captured image of the first camera 20 and the first trained model 43, and also generates the captured image of the second camera 30 and the second. The processing operation information of the robot arm 10 is generated by using the trained model 44. In this way, by using trained models and captured images suitable for each of the generation of orbit information and the generation of processing operation information, the accuracy of orbit information generation and the accuracy of processing operation information generation can be improved. It can be compatible.

Further, in general, the processing operation control of the robot arm 10 requires finer control than the movement control of the robot arm 10. In the processing operation control, even if a slight deviation of about several millimeters occurs, the robot arm 10 collides with the object 50, the object 50 is scratched, or the object 50 is damaged. This is because it may be damaged. In the present embodiment, the position of the robot arm 10 in which the positional relationship between the object 50 and the end effector 11 satisfies a predetermined condition is set as the target position, and when the robot arm 10 moves to the target position, the robot arm 10 is controlled. Is switched from movement control to processing operation control. In this way, by terminating the movement control, which is coarser than the processing operation control, at the target position, it is possible to suppress the contact between the end effector 11 and the object 50. Further, by starting the processing operation control from the target position, the object 50 can be reliably photographed by the second camera 30 and the processing operation control can be executed.

Further, conventionally, the trajectory information of the robot arm 10 may be generated by a mathematical search. However, in this case, it is difficult to generate orbital information in real time because the amount of calculation increases exponentially according to the dimension of the workspace and the search takes time. On the other hand, in the present embodiment, the information processing apparatus 40 can generate the orbit information in a short time by generating the orbit information using the first trained model 43. Therefore, real-time trajectory information of the robot arm 10 can be generated. That is, even when the obstacle 61 or the object 50 moves while the robot arm 10 is moving, the information processing device 40 can update the trajectory information of the robot arm 10 in real time.

Further, in this embodiment, since the first camera 20 is provided with the depth sensor, the trajectory generating unit 41 can accurately generate trajectory information that does not collide with the obstacles 61 and 62.

Further, when the trajectory information and the processing operation information are generated from the images captured by one camera, it is necessary to install a high-resolution camera equipped with a depth sensor at a position where the work space can be photographed. Such cameras are expensive. On the other hand, in this embodiment, it is possible to generate trajectory information and processing operation information by combining two inexpensive cameras without using an expensive camera. As a result, the control system 1 can be constructed at low cost.

[Embodiment 3]
Other embodiments of the present invention will be described. The difference between this embodiment and the above-described second embodiment is that the content of the process of determining the target position of the robot arm 10 (step S103 in FIG. 3) performed by the information processing apparatus 40 is different. For convenience of explanation, the same reference numerals will be added to the members having the same functions as the members described in the second embodiment, and the description will not be repeated.

In this embodiment, the trajectory generating unit 41 determines the position of the robot arm 10 in which the positional relationship between the object 50 and the shooting direction of the second camera 30 satisfies a predetermined condition as a target position, and the robot up to this target position. Generates trajectory information indicating the trajectory of the arm 10. In this case, the orbit generation unit 41 determines, for example, a position in which a part or all of the object 50 is included in the shooting range of the second camera 30 as a target position. More specifically, for example, the trajectory generation unit 41 determines the position of the robot arm 10 in which the entire object 50 is included in the angle of view of the second camera 30 as the target position.

In this case, the robot control unit 45 of the information processing device 40 does not perform the determination process of whether the robot arm 10 has moved to the target position (the process of step S106 of FIG. 3), but instead of comparing the current position with the target position, for example. You may do it as follows. The robot control unit 45 recognizes an object by analyzing the image captured by the first camera 20, and identifies the positional relationship between the object 50 included in the image captured by the first camera 20 and the photographing direction of the second camera 30. To do. The robot control unit 45 determines whether the specified positional relationship satisfies a predetermined condition. When the specified positional relationship satisfies a predetermined condition, the robot control unit 45 determines that the robot arm 10 has moved to the target position. On the other hand, when the positional relationship does not satisfy the predetermined condition, the robot control unit 45 determines that the robot arm 10 has not moved to the target position.

Further, the method for determining the target position of the robot arm 10 is not limited to that shown in the above-described embodiment, and other methods may be used. For example, the trajectory generation unit 41 may determine a position where the ratio of the region of the object 50 in the image captured by the second camera 30 satisfies a predetermined condition as a target position. For example, the trajectory generation unit 41 determines a position at which the ratio of the object 50 occupies 50% or more in the image captured by the second camera as the target position.

In this case, the robot control unit 45 of the information processing device 40 does not perform the determination process of whether the robot arm 10 has moved to the target position (the process of step S106 of FIG. 3), but instead of comparing the current position with the target position, for example. You may do it as follows. The robot control unit 45 analyzes the captured image of the first camera 20 and recognizes the object 50 and the second camera 30 included in the captured image of the first camera 20. The robot control unit 45 uses information such as the positional relationship between the recognized object 50 and the second camera 30, the shooting direction of the second camera, the angle of view of the second camera, and the like to capture the target in the image captured by the second camera. The ratio occupied by the object 50 is calculated. When the calculated ratio is 50% or more, the robot control unit 45 determines that the robot arm 10 has moved to the target position. On the other hand, when the calculated ratio is less than 50%, the robot control unit 45 determines that the robot arm 10 has not moved to the target position.

In this embodiment, the target position can be determined according to the desired shooting state of the object 50 by the second camera 30 at the start of the processing operation control. As a result, processing operation control can be smoothly performed.

[Embodiment 4]
Next, other embodiments of the present invention will be described. The difference between this embodiment and the above-described second embodiment is that the content of the process of determining the target position of the robot arm 10 (step S103 in FIG. 3) performed by the information processing apparatus 40 is different. For convenience of explanation, the same reference numerals will be added to the members having the same functions as the members described in the second embodiment, and the description will not be repeated.

In this embodiment, the trajectory generating unit 41 has at least one of the characteristics of the object 50, the type of processing operation performed by the robot arm 10 on the object 50, and the positions of the obstacle 61 and the obstacle 62. Use one to change the method of determining the target position. The characteristic of the object 50 is a feature or property of the object 50, for example, the size or shape of the object 50. The processing operation performed by the robot arm 10 is, for example, gripping, painting, polishing, cutting, drilling, or assembling an article. In this case, for example, the trajectory generating unit 41 may determine the target position so that the larger the size of the object 50, the larger the threshold value of the distance between the end effector 11 and the object 50. Further, as another example, the trajectory generating unit 41 may make the angle of the robot arm 10 with respect to the object 50 at the threshold value or the target position different depending on the type of processing operation performed by the robot arm 10. Further, as another example, when the obstacle 61 or the obstacle 62 is near the object 50 (the distance between the object 50 and the obstacle 61 or 62 is equal to or less than a predetermined threshold value, the orbit generating unit 41 may be used. ), The target position may be determined from the positional relationship between the object 50 and the second camera 30, while the target position may be determined from the positional relationship between the object 50 and the end effector 11 in other cases.

According to this embodiment, the information processing apparatus 40 can determine a target position suitable for, for example, the size of the object 50. As a result, the information processing device 40 can smoothly start controlling the processing operation of the robot arm 10 with respect to the object 50.

[Embodiment 5]
FIG. 5 is a block diagram showing a schematic configuration of the control system 1A according to the present embodiment. For convenience of explanation, the same reference numerals will be added to the members having the same functions as the members described in the second embodiment, and the description will not be repeated.

The control system 1A shown in FIG. 5 is different from the control system 1 according to the second embodiment in that the camera 70 is provided instead of the first camera 20 and the second camera 30, and the information processing device 40 is replaced. The point is that the information processing device 40A is provided. In this embodiment, the camera 70 photographs the working space of the robot arm 10. The camera 70 outputs shooting data representing an RGB image and a depth image.

The information processing device 40A includes an orbit generation unit 41A and a processing operation generation unit 42A in place of the orbit generation unit 41 and the processing operation generation unit 42. The trajectory generation unit 41A generates trajectory information using the state information, the shooting data of the camera 70, and the first trained model 43. In addition, the processing operation generation unit 42A generates processing operation information using the state information, the shooting data of the camera 70, and the second trained model 44. As described above, in this embodiment, the shooting data of the common camera 70 is used in the generation of the trajectory information of the robot arm 10 and the generation of the processing operation information.

In this embodiment, a trained model suitable for each of the generation of trajectory information and the generation of processing operation information is used. Thereby, the accuracy of both the generation of the trajectory information and the generation of the processing operation information can be improved.

(Modification example)
In the above-described second embodiment, the second camera 30 is installed on the robot arm 10, but the position where the second camera 30 is installed is not limited to that shown in the above-described embodiment. For example, the second camera 30 may be fixed to something other than the robot arm 10. Further, the second camera may be fixed to the end portion of the end effector 11. In this case, the robot arm 10 (or the end effector 11) may not be included in the shooting range of the second camera 30.

In the second embodiment described above, the trajectory generating unit 41 repeatedly executes the trajectory information generation process for a period from the current position to the target position of the robot arm 10 (step S104 in FIG. 3). The trajectory generation unit 41 may perform the trajectory information generation process only once at the initial position of the robot arm 10 without repeatedly executing the trajectory information generation process. In this case, the robot control unit 45 controls the robot arm 10 according to the trajectory information generated at the initial position of the robot arm 10, and controls the robot arm 10 to be gradually moved to the target position.

The functions realized by the information processing device 40 according to each of the above-described embodiments may be shared and realized by a plurality of devices. For example, the first device including the trajectory generation unit 41 and the processing motion generation unit 42 and the second device including the robot control unit 45 may be configured as separate devices. Further, in this case, a part of the plurality of devices may be a so-called cloud server connected to another device in a communication network.

[Example of realization by software]
The control block of the information processing device 40 (particularly, the trajectory generation unit 41, the processing operation generation unit 42, and the robot control unit 45) is realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like. It may be realized by software. In the latter case, the information processing device 40 is configured by using, for example, a computer (electronic computer).

FIG. 6 is a block diagram illustrating the physical configuration of a computer used as the information processing device 40. As shown in FIG. 6, the information processing device 40 can be configured by a computer including a bus 410, a processor 401, a main memory 402, an auxiliary memory 403, and an input / output interface 404. The processor 401, the main memory 402, the auxiliary memory 403, and the input / output interface 404 are connected to each other via the bus 410.

As the processor 401, for example, a microprocessor, a digital signal processor, a microcontroller, or a combination thereof or the like is used. As the main memory 402, for example, a semiconductor RAM (random access memory) or the like is used.

As the auxiliary memory 403, for example, a flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a combination thereof or the like is used. The auxiliary memory 403 stores a program for causing the processor 401 to execute the operation of the information processing device 40 described above. The processor 401 expands the program stored in the auxiliary memory 403 on the main memory 402, and executes each instruction included in the expanded program. As the input / output interface 404, for example, a USB interface, a short-range communication interface such as infrared rays or Bluetooth (registered trademark), or a combination thereof is used.

(Summary)
The information processing apparatus according to the first aspect of the present invention uses the first image data generated by the first camera, the state information regarding the state of the robot arm, and the first trained model, and the trajectory to be traced by the robot arm. Processing indicating the processing operation to be performed by the robot arm using the trajectory generating unit that generates the trajectory information indicating the above, the second image data generated by the second camera, the state information, and the second trained model. It includes a processing operation generation unit that generates operation information.

In the information processing apparatus according to the second aspect of the present invention, in the first aspect, the orbit generation unit may repeatedly execute the orbit information generation process until the robot arm moves to a predetermined target position.

In the information processing apparatus according to the third aspect of the present invention, in the first or second aspect, the processing operation information generated by the processing operation generating unit is performed by the robot arm after moving to a predetermined target position along the trajectory. It may be information indicating the processing operation to be performed.

In the information processing apparatus according to the fourth aspect of the present invention, in the third aspect, the orbit generating unit satisfies a predetermined condition in the positional relationship between the object to be processed by the robot arm and the end effector of the robot arm. The position of the robot arm may be determined as the target position, and trajectory information indicating the trajectory of the robot arm up to the target position may be generated.

In the information processing apparatus according to the fifth aspect of the present invention, in the fourth aspect, the orbit generating unit sets the position of the robot arm at which the distance from the end effector of the robot arm to the object satisfies the condition. It may be determined as a position.

In the information processing apparatus according to the sixth aspect of the present invention, in the third aspect, the second camera is provided on the robot arm.
The trajectory generating unit determines the position of the robot arm whose positional relationship between the object to be processed by the robot arm and the shooting direction of the second camera satisfies a predetermined condition as a target position, and reaches the target position. It may be possible to generate trajectory information indicating the trajectory of the robot arm.

In the information processing apparatus according to the seventh aspect of the present invention, in the sixth aspect, the orbit generating unit determines a position in which a part or all of the object is included in the photographing range of the second camera as the target position. It may be that.

In the information processing apparatus according to the eighth aspect of the present invention, in the seventh aspect, the orbit generating unit sets a position where the ratio of the region of the object in the image captured by the second camera satisfies a predetermined condition. It may be decided as.

In the information processing apparatus according to the ninth aspect of the present invention, in the fourth to eighth aspects, the trajectory generating unit has the characteristics of the object, the type of processing operation performed by the robot arm on the object, and obstacles. The above condition may be changed by using at least one of the positions of.

In the information processing apparatus according to the tenth aspect of the present invention, in the first to fifth aspects, the second camera may be provided on the robot arm.

In the information processing apparatus according to the eleventh aspect of the present invention, in the first to tenth aspects, the first camera may include a depth sensor, and the first image data may include image data representing the depth.

The method according to aspect 12 of the present invention uses the first image data generated by the first camera, the state information regarding the state of the robot arm, and the first trained model to indicate the trajectory to be traced by the robot arm. Using the step of generating trajectory information, the second image data generated by the second camera, the state information, and the second trained model, processing operation information indicating the processing operation to be performed by the robot arm is generated. This is a method in which the information processing apparatus executes the steps to be performed.

The program according to the thirteenth aspect of the present invention is a program for operating a computer as the above-mentioned information processing device, and is a program for operating the computer as each of the above-mentioned parts.

The information processing device according to each aspect of the present invention may be realized by a computer. In this case, the information processing device is made into a computer by operating the computer as each part (software element) included in the information processing device. The learning program of the information processing device to be realized and the computer-readable recording medium on which the information processing device is recorded are also included in the scope of the present invention.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

1, 1A Control system 10 Robot arm 40, 40A Information processing device 11 End effector 12 Joint 13 Sensor 20 First camera 30 Second camera 41, 41A Orbit generation unit 42, 42A Processing motion generation unit 43 First trained model 44 2 Learned model 45 Robot control unit 46 Orbit storage unit 50 Objects 61, 62 Obstacles 70 Camera 401 Processor 402 Main memory 403 Auxiliary memory 404 I / O interface 410 Bus

Claims (13)

A trajectory generator that generates trajectory information indicating the trajectory to be traced by the robot arm using the first image data generated by the first camera, state information regarding the state of the robot arm, and the first trained model.
Using the second image data generated by the second camera, the state information, and the second trained model, a processing motion generation unit that generates processing motion information indicating the processing motion to be performed by the robot arm, and a processing motion generating unit.
Information processing device equipped with. The trajectory generating unit repeatedly executes the trajectory information generation process until the robot arm moves to a predetermined target position.
The information processing device according to claim 1. The processing operation information generated by the processing operation generating unit is information indicating the processing operation to be performed by the robot arm after moving to a predetermined target position along the trajectory.
The information processing device according to claim 1 or 2. The trajectory generating unit determines the position of the robot arm as the target position in which the positional relationship between the object to be processed by the robot arm and the end effector of the robot arm satisfies a predetermined condition, and reaches the target position. Generates trajectory information indicating the trajectory of the robot arm of
The information processing device according to claim 3. The trajectory generating unit determines the position of the robot arm whose distance from the end effector of the robot arm to the object satisfies the condition as the target position.
The information processing device according to claim 4. The second camera is provided on the robot arm.
The trajectory generating unit determines the position of the robot arm whose positional relationship between the object to be processed by the robot arm and the shooting direction of the second camera satisfies a predetermined condition as a target position, and reaches the target position. Generates trajectory information indicating the trajectory of the robot arm of
The information processing device according to claim 3. The orbit generating unit determines a position in which a part or all of the object is included in the shooting range of the second camera as the target position.
The information processing device according to claim 6. The trajectory generating unit determines a position where the ratio of the region of the object in the image captured by the second camera satisfies a predetermined condition as the target position.
The information processing device according to claim 7. The trajectory generator changes the conditions by using at least one of the characteristics of the object, the type of processing operation performed by the robot arm on the object, and the position of the obstacle. ,
The information processing device according to any one of claims 4 to 8. The second camera is provided on the robot arm.
The information processing device according to any one of claims 1 to 5. The first camera includes a depth sensor, and the first image data includes image data representing the depth.
The information processing device according to any one of claims 1 to 10. Using the first image data generated by the first camera, the state information regarding the state of the robot arm, and the first trained model, a step of generating trajectory information indicating the trajectory to be traced by the robot arm, and
Using the second image data generated by the second camera, the state information, and the second trained model, a step of generating processing operation information indicating the processing operation to be performed by the robot arm, and a step of generating processing operation information.
How the information processing device executes. A program for operating a computer as the information processing device according to any one of claims 1 to 11, and for operating the computer as each part.

Images