JP2021013996A - Control method of robot system, manufacturing method of articles, control program, recording medium, and robot system - Google Patents

Abstract

To provide a control method of a robot system capable of enhancing, in a short time, a success rate of part take-out operations, reducing a number of retry operations, and preventing extension of cycle time in a part take-out device or the like.SOLUTION: A control method of a robot system comprises: generating a bulk state of a plurality of virtual objects where a virtual environment is used (101); generating a holding position where a virtual robot device holds each of the virtual objects (102); executing operations of taking out the virtual objects in the bulk state in the virtual robot device (104); generating a trained model capable of outputting a priority of multiple possible holding positions from a specific position and orientation of the object by machine learning resulted by operations of taking out the virtual objects that the virtual robot device is caused to execute (200); then, causing an actual robot manipulator to take out objects, using this trained model; and performing re-learning to update the priority of multiple holding positions of the trained model (203).SELECTED DRAWING: Figure 1

Description

The present invention relates to a control method of a robot system for taking out objects in a loosely stacked state, a method of manufacturing an article, a production system, a control program, a recording medium, and a robot system.

Conventionally, there has been known a device for taking out so-called loosely stacked parts in which the parts are not positioned and are randomly placed on a tray or a flat plate (for example, Patent Document 1 below). ). As shown in FIG. 17, for example, this type of extraction device includes an image input device that acquires a wide range of images, a robot manipulator that extracts parts and conveys them to a predetermined placement position, and the like.

In this type of extraction device, in order to ensure successful extraction of parts, for example, it is necessary to determine in advance which part of the parts the hand extraction hand of the robot manipulator should hold. This is because, when the hand holds the part, if the hand does not hold an appropriate position according to the shape of the part, the part may fall out of the hand or interfere with other parts. It is conceivable that the optimum holding position of the component is determined in advance by performing a simulation in the virtual space.

However, in a real robot manipulator, when the take-out hand holds the part, an external force for holding the part is applied to the part. As a result, the posture of the part to be taken out may be lost, and the robot manipulator may fail to hold it. Regarding this point, for example, as shown in Patent Document 2 below, a function for learning the operating conditions when the parts cannot be taken out during the period of actually operating the parts is implemented in the robot manipulator. Is possible.

Japanese Unexamined Patent Publication No. 5-127722 JP-A-2017-64910

However, in the above-mentioned Patent Document 2, in order to comprehensively search for a component holding position that can hold the target component without breaking its posture, if it fails, the retry operation is repeated. As a result, a series of processing time (= cycle time) required for component operation, for example, taking out one component may become long.

This is a problem peculiar to machine learning in which the robot manipulator is actually operated, and when the extraction fails, learning that avoids the failed holding position for a pattern in a similar loose stacking state from the next time onward. This is because it is only done. The control of Patent Document 2 is merely learning to randomly avoid the failed position. For example, in the control of Patent Document 2, information is not given as to which holding position should be avoided so that the component can be held without losing its posture regardless of the external force.

Therefore, an object of the present invention is to increase the success rate of extraction in a short time after the actual operation in a production device such as a parts extraction device, reduce the frequency of retry operations, and shorten the extension of the cycle time. To be.

In order to solve the above problems, in the control method of the robot system of the present invention, a virtual bulk stacking state generation step of generating a bulk stacking state of a plurality of virtual objects in a virtual environment and a virtual robot device are used for the virtual target. A component holding position candidate generation process for generating a holding position for holding a robot, and a virtual component holding operation execution process for causing a virtual robot device to take out a virtual object in a virtual environment for a virtual object in a separately stacked state. , The virtual object in the loosely stacked state is subjected to the operation of taking out the virtual object by the virtual robot device at different holding positions multiple times in the virtual component holding operation execution step. A priority learning process that generates a trained model that can output the priority of multiple holding positions that the robot device can take for that position / orientation from a specific position / orientation, and the position of the object in the actual loose stacking state. The robot control step of causing the actual robot device to execute the take-out operation of taking out the object and the robot control step based on the holding position determined by using the posture and the learned model generated by the priority learning step. A configuration including a priority update step of re-learning to update the priority of a plurality of holding positions of the trained model by machine learning of the result of the retrieval operation executed by the robot device is adopted.

According to the above configuration, machine learning is not performed only by the actual operation, but a trained model used for determining the holding position candidate of the actual extraction operation is generated using the virtual environment. In addition, this trained model is retrained according to the success or failure of taking out the parts in the actual operation. Therefore, according to the above configuration, in a production device such as a parts take-out device, it is possible to increase the take-out success rate in a short time after the actual operation, reduce the occurrence frequency of the retry operation, and shorten the extension of the cycle time.

It is a block diagram which showed the functional structure of the component taking-out apparatus which adopted this invention. It is a flowchart which showed the calculation process of the component taking-out apparatus which adopted this invention. (A) is an explanatory diagram showing an example of loosely stacked parts, (B) is an explanatory diagram of an arrangement example of loosely stacked parts, (C) is an explanatory diagram showing an example of the outer shape of parts and a division method, and (D) is an explanatory diagram. An explanatory diagram showing the arrangement of loosely stacked parts when N = 3, and (E) an explanatory diagram showing the setting of holding position candidates. (A) is an explanatory diagram of an example of a loose stacking state in a virtual space, (B) is an explanatory diagram showing an example of point cloud data in the virtual space, and (C) is an explanatory diagram showing an example of trimming image data. (D) is an explanatory diagram showing an example of applying a virtual external force. (A), (B), and (C) are explanatory views showing the mode of applying an external force in the suction type, the outer diameter gripping type, and the inner diameter gripping type. (A) is an explanatory diagram showing a data list of holding position candidates before and after ascending sort for one pattern of image data, and (B) is an explanatory diagram showing a data list of holding position candidates before and after ascending sorting for all patterns of image data. It is a figure. It is explanatory drawing which showed an example of the learning model of CNN (scratch neural network). (A) is a flowchart showing an image data acquisition process, and (B) is an explanatory diagram showing a method of determining a part to be taken out. It is explanatory drawing which showed an example of the prediction model of CNN (scratch neural network). It is explanatory drawing which showed an example of the change process of the priority order of the holding position candidate with respect to one pattern image data. It is a block diagram which showed the functional structure of the different component taking-out apparatus which adopted this invention. It is a flowchart which showed the calculation process in the structure of FIG. (A) is an explanatory diagram showing a calculation method of a determination value in the configuration of FIG. 11, and (B) is an explanatory diagram showing an image data having a determination value in the configuration of FIG. 11 and a data list of holding position candidates. .. (A) is an explanatory diagram showing an example of a learning model of CNN (convolutional neural network) in the configuration of FIG. 11, and (B) is an example of a prediction model of CNN (convolutional neural network) in the configuration of FIG. It is explanatory drawing which showed. (A) is a flowchart showing the image data acquisition process in the configuration of FIG. 11, and (B) is an explanatory diagram showing an example of trimming of the image data in the configuration of FIG. It is explanatory drawing which showed an example of the update method of the determination value in the configuration of FIG. It is explanatory drawing which showed the structure of the production line which manufactures an article by using the parts take-out apparatus which adopted this invention. It is a block diagram which showed an example of the control system which comprises the calculation part, the learning part, etc. of FIG. 1 or FIG.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings. The configuration shown below is merely an example. For example, a person skilled in the art can appropriately change the detailed configuration without departing from the spirit of the present invention. Further, the numerical values taken up in the present embodiment are reference numerical values and do not limit the present invention.

FIG. 1 is a schematic view showing an embodiment of a robot system according to the present embodiment, particularly a robot system configured as a component extraction device 1. The parts retrieval device 1 is composed of a calculation unit 100 and a learning unit 200, a robot manipulator 11 that retrieves parts with an actual device, and a retrieval result detection unit 15 that detects the component retrieval results of the robot manipulator 11. The calculation unit 100 has a function of determining the priority of the holding position candidates for the parts in the virtual loosely stacked state in the virtual space. The learning unit 200 has a function of determining the priority of the component holding position when the robot manipulator 11 takes out the components from the components in the actual loosely stacked state based on the calculation results tried a plurality of times by the calculation unit 100.

The robot manipulator 11 of the robot system of FIG. 1 can be arranged on a production line (production system) for producing an article, for example, in the form shown in FIG. The robot manipulator 11 of FIG. 17 is delivered by a transport means (not shown) such as another conveyor. Then, the parts 13 randomly stacked on the flat plate (for example, the parts supply tray) 14 are taken out one by one, placed on the Yatoi 150 (jig), and the manufacturing process executed by other production equipment. Hand over to.

Here, FIG. 18 shows an example of a specific hardware configuration of the control system constituting the calculation unit 100, the learning unit 200, and the like in FIG. The control system as shown in the figure can be realized, for example, by implementing a so-called PC form.

The control system of FIG. 18 can be configured by a CPU 1601 as a main control means, a ROM 1602 as a storage device, and PC hardware including a RAM 1603. The ROM 1602 can store a control program of the CPU 1601 and constant information for realizing the control procedure of the present embodiment. Further, the RAM 1603 is used as a work area of the CPU 1601 when executing the control procedure. Further, an external storage device 1606 is connected to the control system of FIG. The external storage device 1606 is not necessarily necessary for the practice of the present invention, but can be configured from an HDD, an SSD, an external storage device of another network-mounted system, or the like.

The control program of the CPU 1601 for realizing the arithmetic processing of the present embodiment, for example, the processing of the calculation unit 100 and the learning unit 200, is stored in a storage unit such as the external storage device 1606 or ROM 1602 (for example, EEPROM area). deep. In that case, the control program of the CPU 1601 for realizing the control procedure of the present embodiment is supplied to each of the above-mentioned storage units via the network interface 1607 (NIF), and is updated to a new (another) program. be able to. Alternatively, the control program of the CPU 1601 for realizing the control procedure described later is supplied to each of the above storage units via storage means such as various magnetic disks, optical disks, and flash memories, and a drive device thereof. In addition, the contents can be updated. Various storage means, storage units, or storage devices in a state in which the control program of the CPU 1601 for realizing the control procedure of the present embodiment is stored constitutes a computer-readable recording medium in which the control procedure of the present invention is stored. It will be.

The network interface 1607 can be configured using, for example, a wired communication standard such as IEEE 802.3 or a wireless communication standard such as IEEE 802.11, 802.11. The CPU 1601 can perform data communication with another device 1104 via the network interface 1607. The other device 1104 corresponds to, for example, a controller or a control control device for manufacturing an article, or another management server. Communication with the robot manipulator 11 shown in FIG. 1 and other production equipment arranged on the production line can also be performed via the network interface 1607.

Further, the control device of FIG. 18 includes a user interface device 400 (UI device). As the user interface device 400, a GUI device including an LCD display, a keyboard, a pointing device (mouse, joystick, etc.) and the like may be arranged. Through such a user interface device 400, it is possible to notify the user of the progress of various arithmetic processes, or to set various parameters for controlling the processes described later. Further, the user interface device 400 can be used as a user designation of a holding position candidate described later, various user notifications, an input means of a user command regarding the progress of processing, and the like.

(Calculation department)
FIG. 2 shows a flow of processing executed by the calculation unit 100 in the virtual space (virtual environment) to determine a specific bulk stacking mode and a priority order of holding positions of corresponding parts. In the process of FIG. 2, first, in step S101, the virtual bulk stacking state generation unit 101 generates a loose stacking state of parts (virtual objects) in the virtual space (virtual bulk stacking state generation step).

The parts generated as virtual objects in this virtual space naturally correspond to the parts of the objects handled by the robot manipulator 11 on the actual production line. For example, a part as a virtual object generated in a virtual space and a real part have the same shape, size, and weight. This virtual component can be generated from the design data of the component handled by the robot manipulator 11 in the actual production process by using, for example, CAD (Computer Aided Design). The number N of parts generated for the virtual bulk stacking state can be freely set. In the following, for ease of explanation, N = 2, that is, a state in which two parts are stacked separately will be described as an example.

In the generation of the loosely stacked state in step S101, as shown in FIG. 3A, parts having the same shape, weight, and material as the parts to be actually taken out are generated in the virtual space, and the fixed parts 130 and the moving parts are generated, respectively. Let it be part 131. The moving component 131 is in a state of being placed on top of the fixed component 130.

FIG. 3B shows the arrangement relationship between the fixed component 130 and the moving component 131. The center of gravity position G0 of the fixed component 130 is arranged at the origin O of the virtual space, and the center of gravity position G1 of the moving component 131 is arranged at a position shifted by Δx and Δy from the center of gravity position G0 of the fixed component 130. At this time, the constraint conditions for Δx and Δy are shown in the following equation (1).

Δｘ、Δｙについては図３（Ｃ）に示すように、部品の外形に対しｎ分割、ｍ分割した幅に基づき、下式（２）、（３）のように決定する。

As shown in FIG. 3C, Δx and Δy are determined as shown in the following equations (2) and (3) based on the width divided into n and m with respect to the outer shape of the component.

Based on the above expression, if the center of gravity position G1 of the moving component 131 is arranged as (Δx, Δy), (−Δx, Δy), (Δx, −Δy), (−Δx, −Δy), the total n × It is possible to obtain a loose stacking arrangement pattern of m patterns. Further, in each arrangement, when the moving component 131 is rotated on the xy plane shown in FIG. 3 (B) by Δθ obtained by dividing 360 degrees by k as shown in the following equation (4), a total of n × It is possible to obtain an m × k pattern of loose stacking arrangement pattern. For example, in the case of n = 10, m = 10, k = 12, a total of 1200 patterns of loose stacking arrangement patterns can be obtained.

By the above arithmetic processing, the virtual bulk stacking state generation unit 101 can generate the bulk stacking state by the same method even when the number of parts is N = 3 or more. For example, when N = 3, the loose stacking arrangement state (pattern) is formed by further arranging the moving parts 132 on the moving parts 131 as shown in FIG. 3 (D). At this time, if the moving part 131 is regarded as a fixed part and the moving part 132 is moved by the same method as described above, n × m × k can be obtained and combined with the arrangement pattern of the fixed part 130 and the moving part 131. And, a total (n × m × k) of ^two patterns can be obtained. If the number of parts is N, the calculation is recursively repeated, so the number of arrangement patterns is the number of parts / patterns to the (N-1) power as shown by the following equation (5).

Next, in step S102 of FIG. 2, the component holding position candidate generation unit 102 outputs a holding position candidate when the hand 12 of the robot manipulator 11 (virtual robot) holds the component. As shown in FIG. 3 (E), the holding position candidates H1 to H4 are provided at a plurality of points on the surface of the component so that the contact surface between the hand 12 and the component has a certain area or more. For example, when a suction pad (suction head) is used for the hand 12, a surface where the suction pad and the surface of the component can be in close contact with each other is selected as a holding position candidate. In the holding by the suction pad (suction head), the position of the uneven surface or the through hole of the component is not preferable as a holding position candidate because a gap is formed between the suction pad and the component.

These holding position candidates are determined by the designer when, for example, designing the hand 12, and the component holding position candidate generation unit 102 sequentially outputs the holding position candidates based on the determination. The component holding position candidate generation unit 102, for example, arranges a plurality of candidates for holding positions with respect to the object around the center of gravity position based on the design information of the component which is an actual object. The details of the generation of the candidate holding position are arbitrary as follows.

In the component holding position candidate generation step executed by the component holding position candidate generation unit 102, a plurality of holding position candidates for the object to be prioritized in the trained model are generated based on the actual component design information. be able to. Alternatively, a plurality of holding position candidates for the object may be generated based on the holding position specified by the user via the user interface device 400. In that case, based on the design information of the actual object, the availability of the holding position specified by the user is determined via the user interface, and if the holding position specified by the user cannot be used, the user is notified to that effect. It may be configured. For example, in the case of using the suction pad, the position where air leakage is likely to occur, such as the position of the through hole or the step, can be collated with the design information of the actual object. You can then check if the user has specified such an inappropriate holding position.

Next, in step S103 of FIG. 2, only one holding position is selected from the holding position candidates generated in step S102. This selection process is performed until all the holding position candidates are selected in step S107, which will be described later.

Subsequently, in step S104, the virtual component state recognition unit 103 recognizes the parts in the loosely stacked state generated in step S101 on the virtual space. As shown in FIG. 4 (A), in the coordinate system on the virtual space, when the height information of the component surface is plotted on the xy plane along the illustrated Z axis, FIG. 4 (B) shows. The point cloud data as shown can be obtained. Such point group data is recorded in a storage device such as RAM 1603 (FIG. 18) using an image data format such as TIFF (Tagged Image File Form) format that can hold "pixel data + height information". Keep it.

Further, in order to make the image data consistent with the image data of the recognition unit 10 described later, the outer shape of the component is centered on the position of the center of gravity of the uppermost moving component (moving component 132 in the example of FIG. 3D). Trimming and recording so that it is included. At the time of trimming, as shown by Wx and Wy in FIG. 3C, for example, twice the size of the outer shape of the part is set as a fixed value such as the number of pixels that fits in the screen, and the aspect ratio is also set to an arbitrary constant. To do.

Next, in step S105, the virtual component retrieval operation execution unit 104 of FIG. 1 executes a component holding operation by a virtual robot that simulates an actual robot manipulator 11 in a virtual environment (virtual component holding operation execution step). The virtual robot that simulates the robot manipulator 11 can be modeled by using the design information of the robot manipulator 11 and the like. In the extraction operation in the virtual environment by the virtual robot, as shown in FIG. 4D, in the virtual component holding operation, the robot manipulator 11 actually performs the component for one of the plurality of holding candidates. Virtually applies an external force that is expected to be applied to the part when holding the. In this procedure, it is necessary to consider the frictional force and normal force between parts, and the frictional force and normal force between parts and the installation surface. For example, calculation is performed using a computer support tool such as CAE (Computer Aided Engineering). be able to.

Further, as shown in FIG. 5A, the method of applying the force applied when the hand attached to the tip of the robot manipulator holds the parts is in the shape of the hand 12 attached to the tip of the robot manipulator 11. Set as appropriate according to the situation. For example, when a suction type suction pad is adopted for the hand 12, the direction and magnitude of the force of the hand 12 is 5 N at maximum in the direction from the normal direction of the contact surface where the hand 12 and the component contact to the pressing direction of the component. And so on. Further, the area to which the hand 12 applies the force is equal to the area of the suction pad, and the pressing stroke amount can be set to 1 mm or the like. Similarly, in the case of a hand having an outer shape gripping method (121) or an inner diameter gripping method (122) other than the suction type as shown in FIGS. (B) and (C), the size and direction, area, and pressing stroke to apply force are also used. Set each amount.

Next, in step S106 of FIG. 2, similarly to step S104, the virtual component state recognition unit 103 recognizes the position of the component after applying the virtual external force in step S105, and the virtual component movement amount calculation unit 105. Calculates the amount of movement of parts before and after the take-out operation. The virtual component movement amount calculation unit 105 obtains the position of the center of gravity of the component from the position information of the component in the xyz coordinate system on the virtual space recognized in step S104 and step S106, respectively, and determines the position of the center of gravity of the component in the x direction and y. The transition Δ in the direction and the z direction is calculated and recorded as in the following equation (6), for example.

Next, in step S107 of FIG. 2, it is confirmed whether the trials of steps S103 to S106 have been completed for all the holding position candidates generated in step S102. If the trials of steps S103 to S106 for all the generated holding position candidates are not completed, the process returns from steps S107 to S103, the untried holding position candidates are selected, and the processes of steps S103 to S106 are repeated. .. If the trials of steps S103 to S106 for all the generated holding position candidates are completed, the process proceeds from steps S107 to S108.

In step S108, the priority order determination unit 106 determines the priority of all the holding position candidates based on the Δ calculated by the equation (6). When determining the priority order, as shown in FIG. 6A, for example, the holding position candidates are sorted in ascending order by Δ for the pattern 1 in the loosely stacked state in step S101, and Δ is small. It is defined as having a higher priority. At this time, if there are holding position candidates having the same value of Δ, the one with the smaller number of the holding position candidates has the higher priority. Such priority determination is based on the inference that the smaller the movement amount (Δ) calculated when the parts are virtually held, the more stable the holding and taking out can be performed.

In step S109 of FIG. 2, the sort calculation result of the priority calculated in step S108 is recorded in a storage device such as RAM 1603 (FIG. 18). Next, in step S110, it is confirmed whether the trials of S101 to S109 have been completed for all the arrangement patterns generated in step S101. If it is not completed, the process returns to step S101, a pattern in an untried bulk stacking state is generated, and the processes of S102 to S109 are executed. If completed, the processing flow of the calculation unit 100 ends. At the end of this processing flow, as shown in FIG. 6B, a list in which the priority order of the holding position candidates is determined is output for the pattern in the all loose stacking state. This list is stored in a storage device (not shown) of the learning unit 200.

(Learning Department)
With reference to FIG. 1, a configuration of a learning unit 200 that generates a trained model used for deriving a priority list of holding position candidates of the target component from an image recording the position and orientation of the target component will be described. The learning unit 200 is composed of a priority learning unit 201, a priority prediction unit 202, and a priority updating unit 203. First, the priority learning unit 201 statistically analyzes the relationship between the two from the combination of a huge amount of input data and output data. Then, it has a so-called machine learning function of modeling a certain rule to obtain a trained model (priority learning process).

Machine learning includes "supervised learning," which calculates a trained model from training data whose input / output data relationships are known, and models certain rules that are common to input data when what should be output has not been decided. There is "unsupervised learning". In addition, "reinforcement learning" that determines the content to be output next from the current input data is also known. The machine learning used in this embodiment is "supervised learning" in which a model is calculated from a data set (learning data) in which input / output data is known.

In the "supervised learning" of the priority learning unit 201, the image data obtained by the virtual component state recognition unit 103 in step 104 is used as the input data as the teacher data, and the holding position candidate obtained in step 108 is used as the output data. (Fig. 6 (B)). When performing "supervised learning" using such image data as an input, for example, a neural network such as CNN (convolutional neural network) known as one of deep learning can be used. This CNN (Neural Network) constitutes a trained model that correlates a specific position / orientation of a virtual object with a plurality of holding positions that can be prioritized by a virtual robot device. By using such a trained model, it is possible to output the priority of a plurality of holding positions that the robot device can take with respect to the specific position and orientation of the object.

In this embodiment, for example, a CNN (convolutional neural network) as shown in FIG. 7 is used, but other forms of CNN may also be used. The CNN of FIG. 7 constitutes a network that can reach the list of priority of the holding position candidates of the output unit from the image data of the input unit as a trained model. The network is composed of a plurality of stages of a squeeze layer, a pooling layer, and a dropout layer that process image data of an input unit. By using such a priority learning unit 201, it is possible to model the relationship between the image data and the priority of the holding position candidate.

When the priority list generation of FIG. 2 and the machine learning (priority learning process) of the trained model using the virtual environment by the priority learning unit 201 (learning unit 200) are completed, the user interface device 400 notifies that fact. Notify the user via. This notification can be performed by displaying characters or displaying voice. After generating the trained model using the virtual environment, it is possible to try the extraction operation under the actual robot control. Therefore, it is possible to further notify the user that the take-out operation by the robot control unit can be executed via the user interface device 400. In that case, the user interface device 400 can also be configured to input a user operation that commands a take-out operation by the robot control unit. The user interface device 400 can be configured so that the user operation for instructing the retrieval operation can be performed by keyboard operation, operation of the pointing device, voice input, or the like.

After the trained model is generated using the virtual environment, it is possible to try the extraction operation by the actual robot control (robot control process). In this actual retrieval operation, the priority prediction unit 202 uses the trained model calculated by the priority learning unit 201 to predict the priority of the holding position when the robot manipulator 11 extracts the parts in the actual device. Do. In this priority prediction, as the input data, the image data of the parts in the loosely stacked state in the actual device, which is acquired by the recognition unit 10, is used. The image data used here is, for example, point cloud data including three-dimensional information equivalent to that obtained by the virtual component state recognition unit 103.

Therefore, the recognition unit 10 (FIG. 1) needs to be configured to obtain three-dimensional information reflecting the position and orientation of the parts. For example, the recognition unit 10 can be configured by using a stereo camera as a three-dimensional measuring means for obtaining point cloud data including the above-mentioned three-dimensional information. Alternatively, the recognition unit 10 may use a three-dimensional measuring device that combines a single camera and a projector instead of the stereo camera.

FIG. 8A shows a flow of image data acquisition processing by the recognition unit 10 required for actual robot control. The process of FIG. 8A is executed when the robot is actually controlled by using the trained model which has been trained as shown in FIG. 7 to take out the parts.

In step S201 of FIG. 8 (A), an imaging process is performed by, for example, a stereo camera constituting the recognition unit 10, and a three-dimensional measurement process is executed in step S202, and the three-dimensional shape is the same as that of FIG. 4 (B). Generate point cloud data. In step S203, the component being imaged is recognized using the generated three-dimensional point cloud data. For this component recognition, a method such as template matching using a 3D CAD model or normalization cross-correlation matching based on shading can be used. As a result, the position and orientation of the part to be taken out are recognized.

In step S204, when a plurality of parts are recognized in step S203, the parts to be taken out are determined. As shown in FIG. 8B, when two target parts 134 and 135 (overlapping) are recognized as extraction targets, the overlap and interference relationship between the parts are checked, and the parts that are easiest for the robot manipulator 11 to extract. Is determined. For example, in the case of FIG. 8B, in order to preferentially take out the part having the smallest overlapping area between the parts, the target part 134 has a smaller overlapping area between the parts than the target part 135. Select 134. As another method for selecting the target part, the height information included in the three-dimensional information is used to give priority to the part at the top of the randomly arranged loosely stacked parts, and the recognition accuracy by matching. Control may be performed to select in descending order of.

Next, in step S205, the three-dimensional point cloud data is trimmed around the position of the center of gravity of the component to be taken out, and in step S206, the data is converted into a predetermined image file format, for example, TIFF format and recorded. When trimming, preferably, the same number of pixels and aspect ratio as when trimming by the virtual component state recognition unit 103 is used. However, if these same settings cannot be applied, the same number of pixels and aspect ratio can be obtained by performing pooling that resamples the resolution, interpolation between pixels, and interpolation processing on the trimmed image file. It is desirable to adjust as such.

As described above, the image data of the trimmed part is taken into the priority prediction unit 202, and the position of the part is held by using the trained model (CNN in FIG. 9) calculated by the priority learning unit 201. Predict the priority of what to do. The initial state of the network in FIG. 9 is the same as the trained model calculated by the priority learning unit 201 as shown in FIG. Using this trained model (CNN in FIG. 9), it is possible to reach the priority list of the holding position candidates of the component in the output section from the image of the component actually captured in the input section. Then, the robot control unit controls the robot manipulator 11 so as to hold the component holding position having the highest priority obtained by using the learned model (CNN in FIG. 9), and the robot manipulator 11 takes out the robot manipulator 11. To execute.

The robot manipulator 11 that has executed the take-out operation performs the component holding operation according to the instruction of the priority order prediction unit 202 described above, and transfers the component to a predetermined position while holding the component. Then, at the destination point, the extraction result detection unit 15 confirms the extraction result. The take-out result detection unit 15 confirms whether or not the robot manipulator 11 holds the part at the transfer destination, for example. If the part is held, the take-out is successful and the part must be held. If so, a binary signal of extraction failure is output.

As a unit for detecting the presence or absence of these parts, for example, a photoelectric sensor or a laser type object sensor, particularly when the hand 12 is a suction type, a pressure value at the time of suction can be used. In the present embodiment, when the retrieval result detection unit detects a retrieval failure, the priority updating unit 203 performs a re-learning process for updating the model created by the priority learning unit 201 according to the result of the failure ( Priority update process).

For example, when the priority prediction value (at the time of failure) shown in FIG. 10 is output by the priority prediction unit 202, the holding position candidate used for the failed retrieval, that is, the first candidate in the priority list is at the end. Drop in (191-> 192). Then, the list after the priority change is generated by moving up the other priorities one by one. This list is added to the list shown in FIG. 6 (B), re-learning is performed by the priority learning unit 201, and the trained model is updated.

As described above, according to the present embodiment, the trained model generated using the virtual environment is subjected to a pattern of stacking similar parts from the next time onward according to the result of the actual trial extraction operation using the model. Relearn for position and posture). Therefore, the trained model for reaching the priority of the holding position candidate can be updated according to the result of the retrieval operation tried in the real environment so as not to repeat the same failure. Therefore, after actually starting the take-out operation of the robot manipulator 11, the take-out success rate can be increased at high speed, and the extension of the cycle time due to the retry operation can be shortened.

Further, in the component retrieval using the trained model of the present embodiment, the original trained model can be easily generated at high speed and easily by using the virtual environment without actually operating the retrieval hardware. .. Moreover, after actually operating the retrieval hardware, re-learning is performed so as to update the trained model that controls component retrieval according to the actual component retrieval result. Therefore, it is possible to grow a trained model that controls component retrieval with high accuracy and controllable component retrieval by reflecting various conditions in the actual environment, and it is possible to perform retrieval operation with a high success rate in the real environment. ..

That is, according to the above configuration, it is possible to generate a trained model in which the priority order of the holding position candidates in the component extraction can be determined in advance in the virtual environment. Therefore, the robot manipulator can be provided with a function of predicting a holding position of one part to be taken out without losing the posture of the part even when an external force is applied. Therefore, it is possible to increase the success rate of taking out parts in a short time after the start of operation in the actual environment. Further, even if the parts cannot be taken out for a specific pattern in a loosely stacked state, the holding position of the parts is set to another holding position with a relatively high success rate of taking out the parts for a similar pattern in a loosely stacked state from the next time onward. Change to a candidate. By performing such re-learning, there is an excellent effect that the same retrieval failure is not repeated, the frequency of occurrence of the component retrieval retry operation can be reduced, and the extension of the cycle time can be shortened.

(Select a part with a high success rate of removal and a holding position)
It is also possible to control the selection of parts and holding positions with a high take-out success rate. In the above, an example of selecting the holding position having the highest take-out success rate for one part recognized by the recognition unit 10 has been shown, but here, the part having the highest take-out success rate among a plurality of parts has been shown. Select. In addition, the parts are taken out by selecting the holding position having the highest take-out success rate among the selected parts. This control will be described with reference to FIGS. 11 and 12. FIG. 11 shows the configuration of the system corresponding to the above FIG. 1, and FIG. 12 shows the control procedure corresponding to the above FIG. FIG. 11 is different from FIG. 1 in that the retention success / failure determination unit 106b, the retention success rate learning unit 201b, the retention success rate prediction unit 202b, and the retention success rate update unit 203b are provided.

In the calculation unit 100 of FIG. 11, the same processing as in FIG. 2 is performed until the virtual component movement amount calculation unit 105 calculates the movement amount of the parts in step S106. In the control of FIG. 12, after step S107 for determining whether or not all the holding position candidates have been processed, the holding success / failure determination unit 106b determines the holding success / failure in step S108b. As shown in FIG. 13A, for example, the retention success / failure determination unit 106b determines the retention success / failure based on the Δ calculated in (Equation 6) and an arbitrary threshold value Δth, as in the following equation (7). Is judged.

Then, when all the calculation processing flows of FIG. 12 are completed, a list of calculation results as shown in FIG. 13B is output and stored in the learning unit 200.

Next, the learning unit performs "supervised learning" by the retention success rate learning unit 201b with respect to the retention success rate based on the above-mentioned list of calculation results. At this time, as the teacher data, the image data recognized in step S104 is used as the input data, and the pattern number (the leftmost column in FIG. 13B) is used as the output data. By this learning process, the relationship between the two can be modeled into a CNN and trained model as shown in FIG. 14 (A).

When actually trying to take out parts, the holding success rate prediction unit 202b is a robot manipulator with an actual device based on the learned model calculated by the holding success / failure judgment unit 106b as shown in FIG. 14 (A). The parts taken out by 11 and the holding position to be held are predicted.

In order to make a prediction, as the input data, the recognition unit 10 mounted by a stereo camera or the like is used in the same manner as described above, and the image data of the parts in the actual loosely stacked state to be acquired is acquired, and this is used as described above. Used as input data for the trained model.

The image data acquisition process is shown in FIG. 15 (A). FIG. 15A is a process corresponding to FIG. 8A, and the difference from FIG. 8A is that step S204 in FIG. 8A is step S204b. In the above example, the target parts are narrowed down to one according to how the parts overlap (FIG. 8 (B)), whereas in step S204b of FIG. 15 (A), for example, a plurality of targets are used. Parts can be selected. When a plurality of target parts are determined, the three-dimensional data corresponding to the number of target parts is trimmed as shown by 136 and 137 in FIG. 15B, as in the case where the number of target parts is one. Record (steps S205, S206).

Next, as shown in FIG. 14B, the retention success rate prediction unit 202b uses the trained model learned by the retention success rate learning unit 201b to obtain the most pattern for the input image data. The number of the closest pattern is output by analogizing whether it is a close image. Then, according to the output pattern number, the judgment values in the list shown in FIG. 13B are collated, and the holding position candidate having the largest judgment value of the extraction success rate (minimum value 0, maximum value 1) is held. The robot manipulator 11 is controlled so as to do so. At this time, when a plurality of image data are input, the pattern numbers are inferred for all the image data, the holding position having the largest judgment value among all the judgment values is selected, and the robot manipulator 11 To try to hold the part.

The robot manipulator 11 executes the component extraction operation at the position instructed by the holding success rate prediction unit 202b, and detects the extraction success / failure by the extraction result detection unit 15 as in the above example. Then, when the removal of the parts fails, the holding success rate updating unit 203b relearns to update the model created by the holding success rate learning unit 201b. For example, when the retention success rate prediction unit 202b predicts the pattern number shown in FIG. 16 from the input data, the retention position candidate 1 is selected. According to this instruction, the robot manipulator 11 performs an extraction operation, and when the extraction result detection unit 15 detects an extraction failure, re-learning is performed to update the determination value as shown in the list of FIG. 16 (281-> 282). The determination value can be updated by, for example, an operation such as the following equation (8).

This newly updated pattern is added to the list of FIG. 16, and the retention success rate learning unit 201b retrains the model. As shown above, according to the configurations of FIGS. 11 to 16, the success of taking out the same failure is not repeated in consideration of the success rate of taking out the parts for the pattern in the similar piled state from the next time onward. / Can learn failure. Therefore, after actually starting the take-out operation of the robot manipulator 11, the take-out success rate can be increased at high speed, and the extension of the cycle time due to the retry operation can be shortened.

That is, according to the above configuration, the robot manipulator can take out the parts having the highest take-out success rate and the holding position thereof from the parts in the loosely stacked state. Therefore, in addition to the above-mentioned effects, there is an excellent effect that the parts of the object can be taken out with a higher take-out success rate.

1 ... Parts retrieval device, 10 ... Recognition unit, 11 ... Robot manipulator, 12 ... Hand, 13, 130, 131 ... Parts, 14 ... Flat plate, 15 ... Extraction result detection unit, 100 ... Calculation unit, 101 ... Virtual bulk stacking state Generation unit, 102 ... Part holding position candidate generation unit, 103 ... Virtual part state recognition unit, 104 ... Virtual part retrieval operation execution unit, 105 ... Virtual part movement amount calculation unit, 106 ... Priority determination unit, 106b ... Holding success / failure judgment Department, 200 ... Learning unit, 201 ... Priority learning unit, 201b ... Retention success rate learning unit, 202 ... Priority prediction unit, 202b ... Retention success rate prediction unit, 203 ... Priority update unit, 203b ... Retention success rate update Department.

Claims (19)

A virtual bulking state generation process that generates a bulking state of multiple virtual objects in a virtual environment,
A component holding position candidate generation step of generating a holding position in which the virtual robot device holds the virtual object, and
A virtual component holding operation execution process that causes a virtual robot device to execute an operation of taking out a virtual object in a virtual environment for a virtual object in a separately stacked state.
Specifying an object by machine learning as a result of having a virtual robot device execute an operation of taking out a virtual object at different holding positions multiple times in the virtual component holding operation execution process for a virtual object in a loosely stacked state. A priority learning process that generates a trained model that can output the priority of multiple holding positions that the robot device can take with respect to the position and orientation of
The actual robot device is made to execute the taking-out operation of taking out the object according to the holding position determined by using the position and orientation of the objects in the actual loose stacking state and the learned model generated by the priority learning step. Robot control process and
A robot system including a priority update step of performing re-learning to update the priority of a plurality of holding positions of the learned model by machine learning as a result of a take-out operation executed by the robot control step. Control method. In the robot system control method according to claim 1, the priority learning step calculates the amount of movement of the virtual object before and after the operation of taking out the virtual object, and the holding position of the object in ascending order of the amount of movement. A control method for a robot system that performs priority learning to determine the priority of. In the control method of the robot system according to claim 1 or 2, in the priority update step, when the take-out operation executed by the robot device fails, the holding position of the holding position where the take-out operation fails in the learned model. A method of controlling a robot system that performs re-learning to lower the priority. In the robot system control method according to any one of claims 1 to 3, the robot control step is a robot system control method for acquiring an actual position and orientation of an object by a three-dimensional measuring means. The method for controlling a robot system according to claim 4, wherein the three-dimensional measuring means is composed of a stereo camera. In the robot system control method according to any one of claims 1 to 5, the component holding position candidate generation step gives priority in the trained model based on the design information of the actual object. A control method for a robot system that generates multiple holding position candidates for an object. In the robot system control method according to any one of claims 1 to 5, the component holding position candidate generation step has a priority in the trained model based on a holding position specified by the user via the user interface. A method for controlling a robot system that generates candidates for a plurality of holding positions with respect to the object. In the robot system control method according to claim 7, the component holding position candidate generation step determines the availability of a holding position specified by the user via the user interface based on the design information of the actual object. , A method of controlling a robot system that notifies the user when the holding position specified by the user cannot be used. In the robot system control method according to any one of claims 1 to 8, the component holding position candidate generation step is performed around the center of gravity position of the object based on the design information of the actual object. A control method of a robot system for arranging a plurality of holding position candidates with respect to the object to be prioritized in the trained model. In the robot system control method according to any one of claims 1 to 9, the priority learning step is a priority of a specific position / orientation of a virtual object and a plurality of holding positions that the virtual robot device can take. A control method for a robot system composed of a squeeze neural network that generates the trained model. In the robot system control method according to any one of claims 1 to 10, the learned model generated by the priority learning step is a virtual component holding operation for a virtual object in a loosely stacked state. In the robot control process, the operation of extracting the object to the actual robot device is executed, further including the judgment value of the success or failure of the operation of extracting the virtual object executed by the virtual robot device at different holding positions multiple times in the execution process. A control method of a robot system for determining an object having a high extraction success rate and a holding position thereof by using the determination value included in the trained model. In the control method of the robot system according to any one of claims 1 to 11, after the priority learning step generates the trained model, the user interface indicates that the generation of the trained model is completed. How to control the robot system to notify the user. In the robot system control method according to claim 12, the user is notified via the user interface that the generation of the trained model has been completed, and the retrieval operation by the robot control step can be executed. A method of controlling a robot system that notifies a user via a user interface. The method for controlling a robot system according to claim 13, wherein the user interface includes an input means for instructing the take-out operation by the robot control step. A method for manufacturing an article by using a work taken out as an object by a robot device controlled by the control method of the robot system according to any one of claims 1 to 14. A production system for manufacturing an article using a work taken out as an object by a robot device controlled by the control method of the robot system according to any one of claims 1 to 14. A control program for causing a computer to execute each step of the robot system control method according to any one of claims 1 to 14. A computer-readable recording medium containing the control program according to claim 17. A virtual bulking state generator that generates bulking states of multiple virtual objects in a virtual environment,
A component holding position candidate generation unit that generates a holding position in which the virtual robot device holds the virtual object,
A virtual component retrieval operation execution unit that causes a virtual robot device to perform an operation of extracting a virtual object in a virtual environment for a virtual object in a separately stacked state.
The virtual object in a loosely stacked state is identified by machine learning as a result of having the virtual robot device execute the operation of extracting the virtual object multiple times at different holding positions in the virtual component extraction operation execution unit. A priority learning unit that generates a trained model that can output the priority of multiple holding positions that the robot device can take with respect to the position and orientation of
A robot that causes an actual robot device to execute an operation of taking out an object according to a holding position determined by using the position and orientation of the objects in an actual loose stacking state and the learned model generated by the priority learning unit. Control unit and
A robot system including a priority update unit for re-learning to update the priority of a plurality of holding positions of the learned model by machine learning of a take-out operation executed by the robot control unit.

Images