JP6771744B2 - Handling system and controller - Google Patents

Description

Disclosure embodiments relate to handling systems and controllers.

Patent Document 1 describes a method of extracting a gripping surface on which a robot hand grips a part and calculating a gripping position based on a three-dimensional model of the part.

Japanese Unexamined Patent Publication No. 2016-03879

However, in the above-mentioned conventional technique, since the gripping surface and the gripping position are artificially designed by the user, it is difficult to perform a reliable gripping plan for parts having various shapes and postures, and the gripping performance is low.

The present invention has been made in view of such problems, and an object of the present invention is to provide a handling system and a controller capable of improving the handling performance by a machine.

In order to solve the above problems, according to one viewpoint of the present invention, a shape acquisition unit that acquires shape information of an object, a plurality of types of end effectors that directly contact the object and perform a handling operation, and a machine. A handling setting unit that estimates the handling position, which is the contact position on the object side, and selects the end effector to be operated with respect to the object, based on the learning content in the learning process. A handling system with and is applied.

Further, according to another aspect of the present invention, based on the learning content in the machine learning process, the handling position, which is the contact position on the object side, is estimated according to the shape of the object, and the object is subjected to the estimation. On the other hand, a controller having a handling setting unit for selecting an end effector to be operated is applied.

According to the present invention, the handling performance by a machine can be improved.

It is a figure which shows an example of the schematic system block composition of the handling system of embodiment. It is a front view which shows the structure of three kinds of end effectors. It is a figure which shows the example of the handling position with respect to the industrial part of a substantially egg shape. It is a figure which shows the example of the handling position with respect to the bolt type industrial part. It is a figure which shows the example of the handling position with respect to the flat plate-shaped industrial part. It is a figure which shows the example of the handling position with respect to the industrial part of 2 cone type shape. It is a figure which shows an example of the neural network model outline structure of the handling position estimation part. It is a figure which shows an example of the neural network model outline structure of the end effector selection part. It is a flowchart which shows the control procedure which CPU of a robot controller executes in order to realize data learning processing. It is a flowchart which shows the control procedure which the CPU of a robot controller executes in order to realize the reinforcement learning process. It is a figure which shows an example of the system block composition of the robot controller which exchanged the order of a handling position estimation part and an end effector selection part. It is a figure which shows an example of the neural network model outline composition which configured the handling position estimation part and the end effector selection part in parallel and the same. It is a figure which shows the structural example which acquires the shape information of food with a 3D camera. It is a figure which shows an example of the model schematic structure of the neural network at the time of outputting the handling position information and end effector selection information based on two shape information. It is a system block diagram which shows the hardware configuration of a robot controller.

Hereinafter, embodiments will be described with reference to the drawings.

<Outline configuration of handling system>
FIG. 1 shows an example of a schematic system block configuration of the handling system of the present embodiment. This handling system is a system for handling and transporting a large number of industrial parts of a plurality of types randomly arranged in a container 7 so as to classify them according to the type and divide them into a plurality of target containers. In FIG. 1, the handling system 1 includes a camera 2, a robot controller 3, a servo amplifier 4, a handling robot 5, and an end effector exchange unit 6. In the example of this embodiment, the case where the industrial part 10 is the object of handling is shown, but if it is handled on the production line, it may be food, cosmetics, stationery, etc. instead of the industrial part 10. You may.

In this example, the camera 2 is an imaging device that optically captures image data of a two-dimensional pixel sequence. The camera 2 is arranged above the container 7, and is capable of capturing the entire image of a large number of industrial parts 10 of a plurality of types arranged in a messy posture inside the container 7.

The robot controller 3 (controller) performs processing related to the handling work and the transfer work of the handling robot 5 based on the image data captured by the camera 2, and outputs a drive command to the servo amplifier 4. The robot controller 3 has an image recognition unit 11, a handling setting unit 12, a work planning unit 13, and an inverse kinematics calculation unit 14.

The image recognition unit 11 recognizes the individual of each of the plurality of industrial parts 10 arranged in the container 7 at that time by recognizing the image data captured by the camera 2 in the image data. The appearance partial image (see FIGS. 3 to 6 described later) of each individual in the above is extracted and output as shape information. The image recognition unit 11 also detects the arrangement position information in the entire image data of each shape information (appearance partial image) and outputs it separately to the work planning unit 13 described later, but it is omitted in the drawing. .. For such internal processing of the image recognition unit 11, a known image recognition method using machine learning in a neural network or the like may be used, and detailed description thereof will be omitted here. The camera 2 and the image recognition unit 11 correspond to the shape acquisition unit described in each claim.

The handling setting unit 12 is based on the shape information input from the image recognition unit 11, and is information for appropriately handling the individual of the industrial part 10 corresponding to the shape information, that is, handling position information (in the figure, "" HDL position information (abbreviated as "HDL position information") and end effector selection information (abbreviated as "EE selection information" in the figure) are set and output. Here, the handling in the present embodiment refers to the operation itself of contacting and fixing the industrial part 10 with the end effector described later attached to the handling robot 5 so that the individual target industrial part 10 can be stably transferred. Means. In the example of the present embodiment, specifically, the handling operation includes a gripping operation by a multi-finger hand or a gripper described later and a suction operation by a suction nozzle described later. The handling setting unit 12 has a handling position estimation unit 21 and an end effector selection unit 22.

The handling position estimation unit 21 is estimated to be appropriate when performing a handling operation on an individual of the industrial parts 10 corresponding to the shape information based on the shape information input from the image recognition unit 11. The contact position on the side is output as handling position information.

The end effector selection unit 22 selects an appropriate type of end effector when handling the industrial component 10 at the contact position indicated by the handling position information, based on the handling position information input from the handling position estimation unit 21. Then, this type is output as end effector selection information.

The internal processing of the handling setting unit 12 in the example of the present embodiment is performed by the neural network learned in the machine learning process, and the processing content and method will be described in detail later.

The work planning unit 13 causes the handling robot 5 to perform the operation based on the handling position information and the end effector selection information input from the handling setting unit 12, and the arrangement position information input from the image recognition unit 11 (not shown). The work contents of the specific handling operation and the transfer operation (such as the trajectory and operation of the end effector described later) are planned, and the work command generated thereby is output to the inverse kinematics calculation unit 14.

The inverse kinematics calculation unit 14 is necessary to realize the work content (movement and operation of the end effector on the planned trajectory, etc.) of the work command based on the work command input from the work planning unit 13. The target rotation angle of each drive shaft motor (not shown) of the handling robot 5 and the operation value of the end effector drive auxiliary device (not shown) are calculated, and the corresponding drive command is output. The work planning unit 13 and the inverse kinematics calculation unit 14 correspond to the handling control unit described in each claim.

The processing and the like in the image recognition unit 11, the handling setting unit 12 (handling position estimation unit 21, end effector selection unit 22), work planning unit 13, inverse kinematics calculation unit 14, and the like described above are shared among these processes. The present invention is not limited to the examples, and for example, the processing may be performed by a smaller number of processing units (for example, one processing unit), or may be processed by a further subdivided processing unit. Further, the robot controller 3 may be implemented in software by a program executed by the CPU 901 (see FIG. 15) described later, and a part or all of the robot controller 3 may be an ASIC, an FPGA, other electric circuits, or the like (neuromorphic chip). Etc.) may be implemented in hardware by the actual device.

The servo amplifier 4 uses each drive shaft motor (not shown) and end effector drive auxiliary device (not shown) of the handling robot 5 based on the drive command input from the inverse kinematics calculation unit 14 of the robot controller 3. Drive control Performs power supply control of drive power.

The handling robot 5 is a manipulator arm (six-axis robot) having six joint axes in the illustrated example of the present embodiment. Various end effectors 8 capable of handling the industrial parts 10 can be attached to and detached from the arm tip portion 5a, and the attached end effectors 8 handle the industrial parts 10 in the container 7 one by one and are in the vicinity. It has a function of being able to be transferred to the target container 9. Further, in the illustrated example, an end effector replacement unit 6 that functions as a holder capable of individually holding various end effectors 8 is provided at one end of the table, and the handling robot 5 is a desired end in the end effector replacement unit 6. The effector 8 can be attached to and detached from the arm tip 5a.

In the example of the handling system 1 of the present embodiment, three types of end effectors, a multi-finger hand 8A, a gripper 8B, and a suction nozzle 8C, as shown in FIG. 2, can be used. The handling reference point P and the gripping direction vector V shown in FIG. 2 will be described in detail later.

The multi-finger hand 8A of the example shown in FIG. 2A faces two finger portions 8Ab and 8Ac arranged in parallel in the base portion 8Aa that can be directly attached to the arm tip portion 5a. A total of three finger portions 8Ab to 8Ad are provided, one finger portion 8Ad arranged in such a manner. Each finger portion 8Ab to 8Ad has a joint at a root position with respect to the respective base portion 8Aa and an intermediate position in the longitudinal direction, and these joints switch between a flexion motion and an extension motion in the interfinger center direction. It is possible to switch between gripping and releasing the object (industrial part 10 of the example of the present embodiment) located at the center between the fingers.

The gripper 8B of the example shown in FIG. 2B is provided with two claw portions 8Bb and 8Bc arranged in parallel in the base portion 8Ba that can be directly attached to the arm tip portion 5a. By switching between the proximity operation and the separation operation in the direction in which these two claw portions 8Bb and 8Bc face each other, it is possible to switch between gripping and releasing the object located in the center between the claw portions. ing. In addition to the two-claw type shown in the figure, three or more claws may be provided around the central axis between the claws at equal intervals (not shown).

In the suction nozzle 8C shown in FIG. 2C, a hollow tube 8Cb protrudes from a base portion 8Ca that can be directly attached to the arm tip portion 5a, and an elastic material such as resin is used at the tip of the hollow tube 8Cb. A suction port 8Cc is provided. By applying an air suction force by a suction pump (not shown) to the opening of the suction port 8Cc via the base 8Ca and the hollow tube 8Cb, the suction port 8Cc can suck and fix the surface of the object.

<Characteristics of this embodiment>
For example, when an object having a specified shape and size (industrial part 10 in the example of the present embodiment) is arranged in a specified position and posture, the user may determine the contact handling position and the end effector 8 to be used. Appropriate handling operation by the machine can be easily performed by artificially setting and selecting.

However, when a plurality of types of objects having different shapes are mixed, the user needs to artificially set the handling position and select the end effector 8 for each type, which increases the setting work. Resulting in. Further, such artificial setting and selection work largely depends on the user's experience and skill, and there is no guarantee that it is optimal, so that the certainty of the handling operation cannot be ensured. Furthermore, even if the object has a specified shape, if the posture to be placed is indefinite, the handling position to be set and the end effector to be selected in order to reliably perform the handling operation corresponding to the posture. Since 8 changes in various ways, it was difficult to set those combinations one by one. As described above, even if an object having an indefinite shape and an indefinite posture is imaged by the camera 2, it is further difficult for the machine to reliably handle the object based on the captured image.

On the other hand, in the handling system 1 of the present embodiment, based on the learning content in the machine learning process, the handling position, which is the contact position on the object side, is estimated according to the shape information of the object, and the object is It has a handling setting unit 12 for selecting an end effector 8 to be operated. Then, in the machine learning process, the result of repeatedly trying the handling operation with various combinations of handling positions and end effectors 8 is learned for the shape information of the objects having various shapes and arrangement postures. This machine learning process can be automatically performed by the machine itself, and the greater the number of trials (success / failure data), the higher the accuracy and accuracy of the handling operation. As a result, the handling setting unit 12 estimates the optimum handling position and the end effector 8 for the handling operation for the object in response to the shape information indicating the combination of the shape and the posture of the object, regardless of the artificial setting work. Can be selected.

<About the machine learning process of handling operation>
3 to 6 show examples of shape information showing the appearance shapes of industrial parts 10 having various shapes in various arrangement postures. These shape information are uniformly acquired by fixed-point photography from the same direction with reference to the work coordinates (XYZ coordinates in the figure) set in the work space of the handling robot 5 (note that the illustration is easy to understand). As described above, the imaging direction of the camera 2 shown in FIG. 1 is different). That is, this shape information, which is an external partial image, includes two pieces of information, that is, the arrangement posture of one individual industrial part 10 and the appearance shape of the industrial part 10 corresponding to the arrangement posture. Further, the shape information of a large number of industrial parts 10 in a state of being randomly arranged (or stacked separately) in the container 7 is acquired in various arrangement postures other than those shown in the drawing.

Then, in the machine learning process of the handling setting unit 12, the handling operation is repeatedly tried with various combinations of handling positions and end effectors 8 for the appearance shape and the arrangement posture included in the certain shape information. For example, as shown in FIG. 3A, the shape information of the industrial part 10A having an oval shape as a whole and its long axis being substantially parallel to the Y-axis direction is diversified by the multi-finger hand 8A. The handling operation is repeatedly tried by gripping at various gripping positions, gripping at various gripping positions with the gripper 8B, sucking at various suction positions with the suction nozzle 8C, and the like.

Then, in the machine learning process in the example of the present embodiment, the appropriate handling position and the end effector 8 corresponding to the shape information are learned by data learning based on the trial of these handling operations. For example, in a trial of a handling operation performed in response to a certain shape information, the success or failure of the handling (for example, whether or not the industrial part 10 could be lifted) and the accuracy (for example, stability and reliability) are detected as trial results. Then, these are recorded and saved as handling success / failure data in association with the original shape information, the tried handling position, and the end effector 8. Then, by performing machine learning (see FIG. 9 described later) using a large amount of such handling success / failure data, the handling setting unit 12 can estimate an appropriate handling position corresponding to the shape information and select the end effector 8. Become.

For example, in the case of the multi-finger hand 8A having the above configuration, it is particularly effective for gripping an object having a substantially spherical shape as a whole so as to wrap the outer peripheral side surface with each finger portion 8Ab to 8Ad. Therefore, according to the machine learning process, the handling setting unit 12 sets three points sandwiching the short axis direction with respect to the industrial part 10A having a substantially egg-shaped external shape as shown in FIG. It is assumed that it is appropriate to learn that it is appropriate to grip with the multi-finger hand 8A as the position of the broken line portion in the figure).

Further, for example, in the case of the two-claw gripper 8B having the above configuration, the two claw portions 8Bb and 8Bc sandwich the object having two planes (or curved surfaces) arranged so as to face substantially parallel to each other in the opposite direction. It is effective for such gripping. Therefore, according to the machine learning process, the handling setting unit 12 refers to the industrial part 10B having a bolt-shaped external shape as shown in FIG. 4 in the axial radial direction of the bolt shaft and the bolt head according to the arrangement posture. It is assumed that it is appropriate to learn that it is appropriate to grip the two points sandwiching (opposing side surfaces) with the two-claw gripper 8B as the handling position (contact position; the position of the broken line portion in the drawing).

Further, for example, in the case of the suction nozzle 8C having the above configuration, it is particularly effective to suck the flat surface of the object having a flat surface upward by the suction port 8Cc and pull it up. Therefore, according to the machine learning process, the handling setting unit 12 sets one point on the plane as the handling position (adsorption position; in the drawing) for the industrial part 10C having a substantially flat plate-shaped appearance shape as shown in FIG. It is assumed that it is learned that it is appropriate to suck with the suction nozzle 8C as the position of the broken line portion of.

Further, even if the objects have the same shape, the effective handling position and the end effector 8 may change depending on the arrangement posture. For example, as shown in FIG. 6, with respect to the industrial part 10D having a shape in which the apex portions of the two conical shapes are connected, the handling setting unit 12 is set so that the axial direction of the industrial part 10D is substantially horizontal by the machine learning process. In a certain arrangement posture (see FIGS. 6A and 6B), the two-claw gripper 8B uses two points sandwiching the joint portion in the axial direction as a handling position (contact position; the position of the broken line portion in the drawing). It is assumed that learning is appropriate to grasp. Further, in the arrangement posture (see FIG. 6C) in which the axial direction of the same industrial component 10D is substantially vertical, one point on the bottom surface (plane located above) of the conical portion located above is the handling position (see FIG. 6C). It is assumed that it is learned that it is appropriate to suck with the suction nozzle 8C as the suction position (the position of the broken line portion in the figure).

In any of the above cases, considering the accuracy of the handling operation, the handling setting unit 12 learns that it is appropriate to set the handling position close to the center of gravity of each industrial part 10 by the machine learning process. There is expected. Further, the combination of the handling position and the end effector 8 in each of the above-mentioned learning examples is merely an assumed example, and depending on the detailed configuration of the industrial parts 10 and the end effector 8 and other factors, other handling positions and the end There is also the possibility of learning that the combination of effectors 8 is appropriate.

<About the configuration of the neural network in the handling setting section>
As described above, the handling setting unit 12 estimates the handling position and selects the end effector 8 by machine learning of deep learning using a neural network. Then, in the example of the present embodiment, the handling position estimation unit 21 first estimates the handling position based on the shape information, and then the end effector selection unit 22 selects the end effector 8 based on the handling position information. .. From the above, the handling setting unit 12 combines, for example, the handling position estimation unit 21 as shown in the schematic diagram of the neural network model of FIG. 7 and the end effector selection unit 22 as shown in the schematic diagram of the neural network model of FIG. Can be configured with.

In the model schematic diagram of the handling position estimation unit 21 shown in FIG. 7, markings (broken lines in the figure) indicating the estimated handling position with respect to the shape information which is the image data of the two-dimensional pixel sequence input from the image recognition unit 11 It is designed to process the image so that the part) is entered in the image data and output it as handling position information. The entry position of the marking is based on the learning content in the machine learning process described above, and is appropriately corresponding to the appearance (shape, posture) of the industrial part 10 included in the image data of the shape information. This is the contact position on the industrial component 10 side estimated to be handleable. That is, the neural network of the handling position estimation unit 21 learns the feature amount representing the correlation between the appearance shape pattern of the industrial component 10 included in the shape information and the handling position.

Further, in the model schematic diagram of the end effector selection unit 22 shown in FIG. 8, the types of the selected end effector 8 are clustered based on the handling position information which is the image data of the two-dimensional pixel sequence input from the handling position estimation unit 21. It is designed to output. The type of the selected end effector 8 is based on the learning content in the machine learning process described above, and corresponds to the number of markings included in the image data of the handling position information and the arrangement relationship with the industrial parts 10. This is the type of end effector 8 selected so that it can be handled appropriately. That is, the neural network of the end effector selection unit 22 learns the feature amount representing the correlation between the appearance shape pattern of the industrial component 10 and the marking arrangement pattern included in the handling position information and the type of the end effector 8. ..

In the neural networks in each of the above basic specifications, for example, flexible pattern recognition is possible by configuring the nearest input layer with a convolutional neural network (not shown in particular), which is a combination of a so-called convolutional layer and a pooling layer. Become. Further, for example, it is also preferable to configure the immediate vicinity of the output layer with a fully connected layer (not particularly shown) suitable for so-called pattern recognition and optimum value calculation.

<About data learning of machine learning process>
The specific procedure of the machine learning process for each neural network of the handling position estimation unit 21 and the end effector selection unit 22 will be described below. FIG. 9 shows a flowchart of a processing procedure when the CPU 901 of the robot controller 3 (see FIG. 15 described later) executes a machine learning process by data learning in the example of the present embodiment. The data learning process shown in this flow starts, for example, when a command is input from an operation unit (not shown) to execute a machine learning process.

First, in step S5, the CPU 901 acquires image data from the camera 2.

Next, the process proceeds to step S10, and the CPU 901 recognizes an individual individual industrial component 10 independently by the image recognition process of the image recognition unit 11 based on the image data acquired in step S5, and shapes the appearance partial image thereof. Extract and acquire as information.

Next, in step S15, the CPU 901 randomly sets a handling position, which is a contact position, with respect to the appearance pattern of an individual of a predetermined industrial part 10 included in the shape information acquired in step S10, and the corresponding handling position. Generate information (image information with markings).

Next, the process proceeds to step S20, and the CPU 901 randomly selects the type of end effector 8 to be handled with respect to the handling position set in step S15. At this time, the handling robot 5 performs attachment / detachment / replacement work at the end effector replacement section 6 so that the selected end effector 8 is actually mounted on the arm tip portion 5a.

Next, the process proceeds to step S25, in which the CPU 901 uses a combination of the handling position set in step S15 and the type of end effector 8 selected in step S20 to handle and transfer the predetermined industrial component 10 to an individual. Let the handling robot 5 perform the sorting to the target container). This handling operation is controlled by processing in the work planning unit 13 and the inverse kinematics calculation unit 14.

Next, the process proceeds to step S30, and the CPU 901 determines whether or not the handling operation in step S25 is successful, that is, the success or failure as a trial result of gripping or suction of the industrial component 10, by a separate detection device (not shown). .. As the detection device, for example, an image recognition device using an optical device such as a camera or a laser scanner, a weight sensor for measuring the weight of the industrial component 10, or the like may be separately used. Further, here, not only the success or failure of the handling operation is determined, but also a high-speed transfer operation after handling, a rotation operation on a specific drive shaft, a swing operation of a specific arm node, and the like are executed. The accuracy (stability, reliability) of the handling may be detected by detecting the change in the handling state of the industrial component 10 in the end effector 8 before and after.

Next, the process proceeds to step S35, and the CPU 901 determines the shape information acquired in step S10, the handling position information set in step S15, the type of end effector 8 selected in step S20, and the type of end effector 8 selected in step S30. One handling success / failure data is generated and recorded in association with the trial result of the handling operation.

Next, the process proceeds to step S40, and the CPU 901 determines whether or not the handling operation has been tried for all the individual industrial parts 10 arranged in the container 7, for example. If there is an industrial part 10 in the container 7 that has not been handled (sorted to the target container), the determination is not satisfied, and the process returns to step S5 and the same procedure is repeated.

On the other hand, when all the industrial parts 10 in the container 7 have been handled, the determination is satisfied and the process proceeds to step S45.

In step S45, the CPU 901 learns data for each neural network of the handling position estimation unit 21 and the end effector selection unit 22 based on the handling success / failure data generated in the trials of steps S5 to S35. For example, in this data learning, between the input layer and the output layer of each neural network, the acquired shape information is used as input data, and the set handling position information and the teacher data of the combination using the selected end effector 8 as output data are used. Learning is performed by so-called backpropagation processing that adjusts the weighting coefficient of each edge connecting each node so that the relationship of is established. In this backpropagation processing, only the data having a particularly successful handling operation and high accuracy may be extracted from a large number of handling success / failure data, and only this may be used as the teacher data to adjust the weighting coefficient of each edge. .. Alternatively, all handling success / failure data may be used as teacher data, and the weighting coefficient of each edge may be adjusted to be strengthened or weakened according to the success / failure and accuracy of each. In addition to such backpropagation, various known learning methods such as so-called autoencoder, dropout, noise addition, and sparse regularization may be used in combination to improve processing accuracy. After that, this flow ends.

The machine learning process by the above data learning process may be performed on the industrial part 10 to be handled before the operation of the handling system 1 starts, or when the industrial part 10 is changed or the operation accuracy is improved. It may be done when necessary during operation as a purpose.

<Effect of this embodiment>
As described above, the handling system 1 of the present embodiment includes a handling position estimation unit 21 that estimates the handling position according to the shape information of the individual of the industrial part 10 based on the learning content in the machine learning process. It has a handling setting unit 12 including an end effector selection unit 22 that selects an end effector 8 to be operated. Then, in the machine learning process, the results of repeated trials of handling operations with various combinations of handling positions and end effectors 8 are learned for the shape information of individual industrial parts 10 having various shapes and arrangement postures. This machine learning process can be automatically performed by the machine itself, and the greater the number of trials (handling success / failure data), the higher the accuracy and accuracy of the handling operation. As a result, the handling setting unit 12 estimates the optimum handling position for the handling operation of the industrial part 10 in response to the shape information representing the combination of the individual shape and the posture of the industrial part 10, regardless of the artificial setting work. And the end effector 8 can be selected. As a result, the handling performance by the machine can be improved.

Further, in the present embodiment, in particular, it has a handling robot 5, a work planning unit 13 for controlling the handling operation of the handling robot 5, and an inverse kinematics calculation unit 14. As a result, it is possible to realize a reliable handling operation and transfer operation using the handling robot 5 for the industrial parts 10 arranged in various positions and postures. It should be noted that the configuration is not limited to the configuration in which a plurality of types of end effectors 8 can be attached to and detached from one handling robot 5 as in the example of the present embodiment. Alternatively, one handling robot may be equipped with one type of end effector 8 and the same number of handling robots as the types of end effectors 8 may be provided side by side, or one handling robot may be equipped with a plurality of types of end effectors. 8 may be attached and used by switching between them (above, not shown). Further, the handling robot itself to be used is not limited to the manipulator arm of the 6-axis robot of the example of this embodiment. In addition, so-called SCARA robots, parallel robots, orthogonal axis robots and the like (above, not shown) may be applied.

Further, in the present embodiment, in particular, the handling setting unit 12 estimates the handling position and then selects the end effector 8 corresponding to the handling position. As a result, the end effector 8 can be selected based on the handling position, and a flexible handling operation corresponding to the posture of the industrial component 10 becomes possible.

Further, in the present embodiment, particularly in the machine learning process, the machine learning process is executed by data learning based on the handling success / failure data obtained when the handling operation is repeated. As a result, in the machine learning process, it is possible to learn highly versatile handling operations corresponding to the industrial parts 10 having various shapes and postures.

Further, in this embodiment, in particular, the machine learning process is executed for the limited industrial parts 10 having a predetermined shape arranged in the container 7. As a result, in the machine learning process, it is possible to quickly learn a highly accurate handling operation limited to the industrial component 10 having a predetermined shape. It should be noted that the machine learning process may be executed not only for the industrial parts 10 having such a limited predetermined shape but also for many industrial parts 10 having a wide variety of sizes and shapes. In that case, the content learned by the same neural network can improve the versatility that can flexibly deal with objects of various shapes, and unknown shapes that have not been learned until then. It is possible to learn highly versatile handling operations even for objects in (indefinite shape).

Further, in the present embodiment, in particular, the type of the end effector 8 includes at least one of the multi-finger hand 8A, the gripper 8B, and the suction nozzle 8C. As described above, the multi-finger hand 8A is particularly effective for gripping the outer peripheral side surface of the industrial part 10, and the gripper 8B is particularly effective for gripping in two opposite directions (or three or more directions). The suction nozzle 8C is particularly effective for pulling up a flat surface. In the present embodiment, by making it possible to select from these, highly versatile handling operation can be performed corresponding to the national parts having various shapes and postures. The end effector 8 used in the handling system 1 is not limited to these three types, and other types of end effectors (not shown) may be applied.

<Modification example>
It should be noted that the embodiments described above can be variously modified within a range that does not deviate from the purpose and technical idea.

<Transformation example 1: When the machine learning process is performed by reinforcement learning>
In the above embodiment, the appropriate handling position corresponding to the shape information and the machine learning process of the end effector 8 are performed by data learning, but the present invention is not limited to this. In addition, the neural networks of the handling position estimation unit 21 and the end effector selection unit 22 may be learned by the machine learning process of the reinforcement learning process as illustrated in FIG. The reinforcement learning process of this modified example shown in the figure is performed based on the so-called Q-learning method.

First, in step S105, the CPU 901 acquires image data from the camera 2.

Next, the process proceeds to step S110, and the CPU 901 recognizes an individual individual industrial component 10 independently by the image recognition process of the image recognition unit 11 based on the image data acquired in step S105, and shapes the appearance partial image thereof. Extract and acquire as information.

Next, the process proceeds to step S115, and the CPU 901 searches for a new contact position with respect to the appearance pattern of the individual of the predetermined industrial component 10 included in the shape information acquired in step S10. Regarding the search for this new contact position, a contact position corrected in a random direction with a small probability is set from the contact position set as appropriate (the so-called Q value described later is the maximum) up to that point, or In most other cases, the contact position that is considered appropriate is left as it is. The newly searched contact position is set as the current handling position, and the corresponding handling position information (image information with markings) is generated.

Next, the process proceeds to step S120, and the CPU 901 searches for a new end effector 8 with respect to the handling position set in step S115. Regarding the search for this new end effector 8, the end effector 8 randomly changed with a small probability is selected from the end effectors 8 selected as being appropriate (the so-called Q value described later is the maximum) up to that point. Or, in most other cases, the end effector 8 which is considered to be appropriate is left as it is. The end effector 8 newly searched in this way is selected as the end effector 8 to be used this time, and the corresponding end effector selection information is generated. At this time, the handling robot 5 performs attachment / detachment / replacement work at the end effector replacement section 6 so that the selected end effector 8 is actually mounted on the arm tip portion 5a.

Next, the process proceeds to step S125, and the CPU 901 handles and transfers the predetermined industrial component 10 to an individual by combining the handling position searched and set in step S115 and the type of end effector 8 searched and selected in step S120. Let the handling robot 5 perform the operation (partitioning to the target container). This handling operation is controlled by processing in the work planning unit 13 and the inverse kinematics calculation unit 14.

Next, the process proceeds to step S130, and the CPU 901 determines whether or not the handling operation in step S25 is successful, that is, the success or failure as a trial result of gripping or suction of the industrial component 10, by a separate detection device (not shown). .. As the detection device, for example, an image recognition device using an optical device such as a camera or a laser scanner, a weight sensor for measuring the weight of the industrial component 10, or the like may be separately used. Further, here, not only the success or failure of the handling operation is determined, but also a high-speed transfer operation after handling, a rotation operation on a specific drive shaft, a swing operation of a specific arm node, and the like are executed. The accuracy (stability, reliability) of the handling may be detected by detecting the change in the handling state of the industrial component 10 in the end effector 8 before and after. The handling operation this time is evaluated by the success / failure judgment and the detection of the accuracy as described above, and the reward according to this evaluation is calculated.

Next, the process proceeds to step S135, where the CPU 901 includes the shape information acquired in step S110, the handling position information searched and set in step S115, the type of end effector 8 searched and selected in step S120, and the end effector 8 searched and selected in step S130. Reinforcement learning for each neural network of the handling position estimation unit 21 and the end effector selection unit 22 is performed by so-called Q-learning based on the calculated handling operation reward. The handling position and the end effector 8 are learned so that the Q value indicating the effectiveness of the handling operation is further maximized with respect to the trial of the handling operation in steps S105 to S135. For the reinforcement learning by Q-learning, a known method may be used, and detailed description thereof will be omitted here.

Next, the process proceeds to step S140, and the CPU 901 determines whether or not machine learning has been performed on the neural networks of the handling position estimation unit 21 and the end effector selection unit 22 with a sufficient number of trials. If the machine learning is insufficient and the reinforcement learning process is not completed, the determination is not satisfied, and the process returns to step S105 to repeat the same procedure.

On the other hand, when the machine learning is sufficient and the reinforcement learning process is terminated, the determination is satisfied and this flow is terminated.

As described above, the handling system 1 of this modified example is executed by reinforcement learning based on the respective handling results when the machine learning process is repeated. As a result, in the machine learning process, it is possible to quickly learn a highly accurate handling operation limited to the industrial component 10 having a specific shape.

<Modification example 2: When estimating the handling position after selecting the end effector>
In the above embodiment, in the handling setting unit 12, the handling position estimation unit 21 estimates the handling position information corresponding to the shape information, and then the end effector selection unit 22 selects the end effector 8 based on the handling position information. It was, but it is not limited to this. In addition, as shown in FIG. 11, in the handling setting unit 12A in the robot controller 3A, the end effector selection unit 22A selects the end effector 8 corresponding to the shape information, and then the handling position estimation unit 21A selects the end effector. The handling position information corresponding to the selection information and the shape information may be estimated.

In this case, the neural network of the end effector selection unit 22A is a feature quantity that represents a correlation between the appearance shape pattern of the industrial component 10 included in the shape information and the type of end effector 8 corresponding to the appropriate handling operation. To learn. Further, the neural network of the handling position estimation unit 21A provides a feature amount that represents the correlation between the appearance shape pattern and end effector selection information of the industrial component 10 included in the shape information and the corresponding handling position appropriate for the handling operation. learn.

As described above, in the handling system 1 of the present modification, after the handling setting unit 12 selects the end effector 8, the handling position corresponding to the end effector 8 is estimated. As a result, the handling position can be estimated with reference to the end effector 8, and a flexible handling operation corresponding to the shape of the industrial part 10 becomes possible.

Further, the handling position estimation unit 21 and the end effector selection unit 22 may not be individually configured, but may be configured to be processed in parallel by the same neural network as shown in FIG. In this illustrated configuration, the types of end effectors 8 to be selected are clustered to output end effector selection information, and the handling reference target points P'(Xp', Yp', Zp') and the gripping direction target vector V'are set. It is designed to be output as handling position information. For the output of handling position information, a neural network is designed so that the coordinate position of an integer value and the vector value are output by a so-called regression problem.

Here, the handling reference target point P'is a target position for positioning the handling reference point P set on each end effector 8 during the handling operation. Further, the gripping direction target vector V'is a target direction in which the gripping direction vector V set in each end effector 8 (only for the multi-finger hand 8A and the gripper 8B) is matched during the handling operation (the above are FIGS. 3 to 6). reference). As shown in FIG. 2, the handling reference point P of each end effector 8 is set at the center position between the finger portions 8Ab to 8Ad or between the claw portions 8Bb and 8Bc in the case of the multi-finger hand 8A and the gripper 8B, for example. In the case of the suction nozzle 8C, it is set at the center position of the end of the suction port 8Cc. Further, the gripping direction vector V is a direction vector indicating the gripping direction of each finger portion 8Ab to 8Ad or each claw portion 8Bb, 8Bc in the case of the multi-finger hand 8A and the gripper 8B. In this way, even if the position and orientation of the end effector 8 estimated to be appropriate during the handling operation are indicated by the handling reference target point P'and the gripping direction target vector V', and these numerical information are generated as the handling position information. Good.

<Modification 3: When recognizing an individual with multiple image data captured by multiple cameras>
In the example shown in FIG. 13, one food F (fried chicken in the illustrated example) is imaged differently by using a so-called three-dimensional camera 43 composed of two cameras 41 and 42 having overlapping imaging regions. The image is taken from the direction. As a result, as shown on the left side of FIG. 14, it is possible to acquire two image data (two shape information; each two-dimensional pixel sequence) having parallax with respect to one food F.

In the example shown in FIG. 14, the handling position estimation unit 21 and the end effector selection unit 22 configured in parallel by the same neural network input the above two shape information as they are, and end based on the two shape directions. It is designed to output effector selection information and handling position information. By using the three-dimensional image data in this way, the spatial recognition accuracy of the three-dimensional shape is improved especially for the food F whose shape and size are indefinite. Three or more cameras may be used as long as the imaging regions overlap, and the image data of each shape information is processed by an individual convolutional neural network in the neural network, and then pattern recognition is performed by the same fully connected layer. May be good.

<Other variants>
Further, although not particularly shown, a laser scanner or the like may be used instead of the camera 2 (and the three-dimensional sensor 8) as an optical sensing means for acquiring image data of an object. In this case, for example, the distance to each point on the surface of the object is measured by a scanning line projected from a laser scanner, and image data is acquired as a set of these distance data. Further, the image recognition unit 11 may acquire, for example, a 3D stereoscopic model as shape information instead of the image data of the appearance partial image.

In addition to the fixed arrangement of the camera 2, the camera may be installed on, for example, the arm tip 5a of the handling robot 5 or the end effector 8.

Further, the processing algorithms of the handling position estimation unit 21 and the end effector selection unit 22 are not only those based on deep learning using the illustrated neural network, but also other algorithms using, for example, a support vector machine or a Bayesian network. A processing algorithm (not shown) may be applied.

Further, when the handling robot 5 mounts and handles only a single robot without switching the end effector 8, the handling position estimation unit 21 corresponds to the end effector 8 by omitting the end effector selection unit 22. Only the handling position information may be learned and output. Further, as the type of the end effector 8 to be mounted, other types of end effectors (not particularly shown) other than the three types illustrated in FIG. 2 may be mounted, and the handling position corresponds to their handling mode. The estimation unit 21 may learn the handling position information.

<Hardware configuration example of robot controller>
Next, with reference to FIG. 15, the image recognition unit 11, the handling setting unit 12 (handling position estimation unit 21, end effector selection unit 22), and the work, which are implemented by software by the program executed by the CPU 901 described above. A hardware configuration example of the robot controller 3 that realizes processing by the planning unit 13, the inverse kinematics calculation unit 14, and the like will be described.

As shown in FIG. 15, the robot controller 3 includes, for example, a CPU 901, a ROM 903, a RAM 905, a dedicated integrated circuit 907 constructed for a specific application such as an ASIC or an FPGA, an input device 913, and an output device 915. It has a recording device 917, a drive 919, a connection port 921, and a communication device 923. These configurations are connected so that signals can be transmitted to each other via the bus 909 and the input / output interface 911.

The program can be recorded in, for example, a ROM 903, a RAM 905, a recording device 917, or the like.

Further, the program can be temporarily or permanently recorded on, for example, a magnetic disk such as a flexible disk, an optical disk such as various CDs, MO disks, or DVDs, or a removable recording medium 925 such as a semiconductor memory. .. Such a recording medium 925 can also be provided as so-called package software. In this case, the program recorded on these recording media 925 may be read by the drive 919 and recorded on the recording device 917 via the input / output interface 911, the bus 909, or the like.

The program can also be recorded on, for example, a download site, another computer, another recording device, or the like (not shown). In this case, the program is transferred via a network NW such as LAN or the Internet, and the communication device 923 receives the program. Then, the program received by the communication device 923 may be recorded in the recording device 917 via the input / output interface 911, the bus 909, or the like.

Further, the program can be recorded in an appropriate externally connected device 927, for example. In this case, the program may be transferred via the appropriate connection port 921 and recorded in the recording device 917 via the input / output interface 911, the bus 909, or the like.

Then, the CPU 901 executes various processes according to the program recorded in the recording device 917, so that the image recognition unit 11, the handling setting unit 12 (handling position estimation unit 21, end effector selection unit 22), and the work Processing by the planning unit 13, the inverse kinematics calculation unit 14, and the like is realized. At this time, for example, the CPU 901 may directly read the program from the recording device 917 and execute it, or may execute it after loading it into the RAM 905 once. Further, for example, when the CPU 901 receives the program via the communication device 923, the drive 919, or the connection port 921, the CPU 901 may directly execute the received program without recording it in the recording device 917.

Further, the CPU 901 may perform various processes based on signals and information input from an input device 913 such as a mouse, keyboard, microphone (not shown), if necessary.

Then, the CPU 901 may output the result of executing the above processing from an output device 915 such as a display device or an audio output device, and the CPU 901 may connect the processing result to the communication device 923 or the communication device 923 as needed. It may be transmitted via the port 921, or may be recorded on the recording device 917 or the recording medium 925.

In the above description, when there is a description such as "vertical", "parallel", "plane", etc., the description does not have a strict meaning. That is, these "vertical", "parallel", and "planar" mean "substantially vertical", "substantially parallel", and "substantially flat", with design and manufacturing tolerances and errors allowed. ..

Further, in the above description, when there is a description such as "same", "same", "equal", "different", etc. in the external dimensions, size, shape, position, etc., the description is not a strict meaning. That is, those "same", "equal", and "different" are allowed for design and manufacturing tolerances and errors, and are "substantially the same", "substantially the same", "substantially equal", and "substantially equal". It means "different".

In addition to the above, the methods according to the above-described embodiment and each modification may be appropriately combined and used. In addition, although not illustrated one by one, the above-described embodiment and each modification are implemented with various modifications within a range that does not deviate from the purpose.

1 Handling system 2 Camera (shape acquisition unit)
3,3A robot controller (controller)
4 Servo amplifier 5 Handling robot 6 End effector replacement part 8 End effector 8A Multi-finger hand 8B Gripper 8C Suction nozzle 10 Industrial parts (object)
11 Image recognition unit (shape acquisition unit)
12, 12A Handling setting unit 13 Work planning unit (Handling control unit)
14 Inverse kinematics calculation unit (handling control unit)
21,21A Handling position estimation unit 22,22A End effector selection unit

Claims (8)

A shape acquisition unit that acquires shape information of an object,
Multiple types of end effectors that perform handling operations in direct contact with the object,
Based on the learning content in the machine learning process, the handling position, which is the contact position on the object side, is estimated in response to the shape information, and then the end effector that performs the handling operation on the object is set to the handling position. The handling setting unit to be selected correspondingly,
A handling system characterized by having. A handling robot that mounts the end effector and transfers the end effector and the object,
A handling control unit that controls the handling operation of the handling robot so that the selected end effector is brought into contact with the estimated handling position.
The handling system according to claim 1, wherein the handling system comprises. The handling system according to claim 1 or 2 , wherein the machine learning process is executed by reinforcement learning based on each handling result when the handling operation is repeated. The handling system according to any one of claims 1 to 3 , wherein the machine learning process is executed by data learning based on handling success / failure data obtained when the handling operation is repeated. The handling system according to any one of claims 1 to 4 , wherein the machine learning process is performed on an object having a predetermined shape. The handling system according to any one of claims 1 to 4 , wherein the machine learning process is performed on objects having various shapes. The handling system according to any one of claims 1 to 6 , wherein the type of the end effector includes at least one of a multi-finger hand, a gripper, and a suction nozzle. Based on the learning content in the machine learning process, the handling position, which is the contact position on the object side, is estimated according to the shape of the object, and then the end effector that handles the object is moved to the handling position. Handling setting section to select according to
A controller characterized by having.

Images