JP2018144158A - Robot simulation device, robot simulation method, robot simulation program, computer-readable recording medium and recording device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To bring simulation results closer to results acquired during actual operation.SOLUTION: A robot simulation device comprises: a bulk data generating part to generate virtual bulk data, where a plurality of workpiece models are piled up in a virtual work space in which work is virtually carried out; a region estimating part to compare a virtual inclination angle of each point constituting each workpiece model within the bulk data generated by the bulk data generating part with an angle threshold determined by an angle threshold determination part, and identify a point having a virtual inclination angle larger than the angle threshold as an estimation region where measurement is estimated to be difficult; a simulation data generating part to generate bulk data not including data in the estimation region identified by the region estimating part as simulation data; and a picking operation simulation part to execute simulation for verifying the bulk picking operation of the workpiece models within the work space by using the simulation data.SELECTED DRAWING: Figure 25

Description

  The present invention relates to a robot simulation apparatus, a robot simulation method, a robot simulation program, a computer-readable recording medium, and a recorded device.

  The robot vision is combined with the manipulator, the target workpiece is imaged to obtain height information, the appropriate position is grasped (pick or picked), and placed at the desired position (place or placement) Possible robotic devices have been developed. Using such a robotic device, a large number of workpieces placed in a returnable box, called bulk picking, are imaged with a camera or sensor unit to grasp the posture and grasp the appropriate gripping position. The robot arm is moved to a gripping position, gripped by an end effector such as a hand unit, and placed at a predetermined position outside the return box.

  In order to correctly operate the bulk picking, a robot simulation apparatus has been developed that three-dimensionally simulates picking and place motion of a robot that performs bulk picking in advance. In such a robot simulation apparatus, a CAD model having a three-dimensional shape of a work is used as a work model and is randomly arranged in a virtual work space to simulate a robot take-out process, that is, bulk picking.

  However, during actual operation, the work area that can be measured by the sensor unit is only the area located on the outermost surface when viewed from the sensor part installed above, and the dead area is determined by the back surface area of the work or other work or container. It is not possible to measure the area. Even in the region located on the outermost surface, the region having a steep inclination or the region having a large specular component may not be measured because the sensor unit cannot receive sufficient reflected light. For example, in the case of a metal workpiece, specular reflection is strong, and there are parts that cannot be measured depending on the posture of the workpiece.

  As an example, FIG. 79A shows a state of a work displayed on a simulation for a certain work, and FIG. 79B shows a state in which the actual work is actually measured. As shown in FIG. 79A, even if the workpiece has a shape that is clearly displayed in the simulation, the slope is steep when the image is taken from above with a camera or illumination, as shown in FIG. 79B, during actual measurement. The three-dimensional shape cannot be measured for the part.

  As described above, the bulk data obtained by stacking the CAD models used in the simulation is three-dimensional data having complete height information. Therefore, although a workpiece can be searched, an area that cannot be measured during actual operation occurs. As a result, the search may fail. In other words, in the conventional simulation, since a work model having a complete three-dimensional shape that cannot be obtained in actual operation is detected, even a work model that cannot be detected in actual operation on the simulation succeeds in gripping. . For this reason, even if the gripping is successful in the simulation, it fails during gripping during actual operation, which causes the simulation result to differ from the actual operation result.

Japanese Patent No. 4153528 JP 2015-215259 A JP 2002-331480 A

  The present invention has been made in view of such circumstances, and one object of the present invention is a robot simulation apparatus, a robot simulation method, a robot simulation program, and a computer that bring simulation results close to actual operation results. To provide a readable recording medium and a recorded device.

Means for Solving the Problems and Effects of the Invention

  According to the robot simulation apparatus of the first embodiment of the present invention, the robot for simulating the bulk picking operation in which the three-dimensional shapes of a plurality of workpieces stacked in the work space are measured by the sensor unit and sequentially taken out by the robot A simulation apparatus for measuring three-dimensionally measured workpieces in a plurality of different postures by means of a sensor unit, and storing three-dimensional measurement data having information on an inclination angle with respect to a measurement axis of the sensor unit at each position Whether or not three-dimensional measurement is possible is determined based on the measurement data storage unit and the inclination angle information of each point constituting the three-dimensional measurement data of the plurality of workpieces stored in the measurement data storage unit with respect to the measurement axis of the sensor unit. The angle threshold value determination unit for determining the angle threshold value to be used as a reference, and the three-dimensional shape of the workpiece A work model setting unit for setting a work model, and a bulk data generation unit for generating virtual bulk data in which a plurality of the work models are stacked in a virtual work space that is a virtual work space And the virtual inclination angle of each point constituting each work model in the bulk data generated by the bulk data generation unit and the angle threshold determined by the angle threshold determination unit, the angle threshold A region estimation unit for identifying a point having a larger virtual inclination angle as an estimation region estimated to be difficult to measure, and a simulation of unstacked data that does not include data of the estimation region specified by the region estimation unit The work model in the work space using a simulation data generation unit for generating as data for simulation and the simulation data It may include a picking operation simulating section for executing a simulation for verifying the bulk picking operation. With the above configuration, simulation data can be acquired after excluding data that is estimated to be difficult to measure with reference to an angle threshold value determined based on the tilt angle of the workpiece actually measured by the sensor unit. High picking motion simulation is possible.

  Further, according to the robot simulation apparatus of the second embodiment, in addition to the above configuration, the sensor unit of each point constituting the three-dimensional measurement data of the plurality of workpieces stored in the measurement data storage unit A display unit capable of displaying an angle distribution histogram indicating a distribution in which the inclination angles of the respective points are accumulated based on the inclination angle information with respect to the measurement axis can be provided.

  Furthermore, according to the robot simulation device of the third aspect, in addition to the above configuration, the angle threshold value determination unit manually refers to the angle distribution histogram displayed on the display unit, and is manually operated by the user. It can be configured to accept a threshold setting.

  Furthermore, according to the robot simulation apparatus of the fourth embodiment, in addition to any of the above-described configurations, the display unit can be configured to be able to display the simulation data generated by the simulation data generation unit.

  Furthermore, according to the robot simulation apparatus of the fifth aspect, in addition to any one of the above-described configurations, the sensor of each point constituting the three-dimensional measurement data of the plurality of workpieces stored in the measurement data storage unit On the basis of the tilt angle information with respect to the measurement axis of the part, it is possible to remove a point with low reliability of the three-dimensional measurement as an invalid pixel.

  Furthermore, according to the robot simulation apparatus of the sixth aspect, in addition to any of the above-described configurations, the angle threshold value determination unit stores the three-dimensional measurement data of a plurality of workpieces stored in the measurement data storage unit. An angle threshold value can be automatically calculated based on an angle distribution histogram indicating a distribution in which the inclination angle of each point is accumulated based on the inclination angle information of each point constituting the measurement axis of the sensor unit.

  Furthermore, according to the robot simulation apparatus of the seventh embodiment, the robot for simulating the bulk picking operation in which the three-dimensional shapes of a plurality of workpieces stacked in the work space are measured by the sensor unit and sequentially taken out by the robot A simulation apparatus, a work model setting unit for setting a work model obtained by modeling a three-dimensional shape of a work, and a virtual work space in which a plurality of the work models are stacked in a virtual work space that is a virtual work space A bulk data generation unit for generating typical bulk data, a measurement data storage unit for storing three-dimensional measurement data obtained by three-dimensionally measuring a plurality of stacked workpieces with different orientations, For the three-dimensional measurement data of a plurality of workpieces stored in the measurement data storage unit, A 3D search is performed with the work model set in the work model setting unit, and based on the position and orientation of the point where the work model and 3D measurement data match, it becomes a reference for determining whether or not 3D measurement is possible An angle threshold value determination unit for determining an angle threshold value, a virtual inclination angle of each point constituting each work model in the bulk data generated by the bulk data generation unit, and the angle threshold value determination unit A region estimation unit for comparing a point having a virtual inclination angle larger than the angle threshold as an estimation region estimated to be difficult to measure, and the estimation region identified by the region estimation unit A data generation unit for simulation for generating loose data that does not include the above data as simulation data, and using the simulation data Te, it is possible and a picking operation simulating section for the performing the simulation for verifying the bulk picking operation of the workpiece model of the working space.

  Furthermore, according to the robot simulation apparatus of the eighth embodiment, in addition to the above configuration, the angle threshold value determination unit can measure a point where the work model and the three-dimensional measurement data match as a result of the three-dimensional search. The workpiece can be measured by placing the normal of the measurable point in a three-dimensional space, calculating the slope of the normal, and counting the normal angle range of the measurable point. It can be configured to determine a threshold value for the surface tilt.

  Furthermore, according to the robot simulation method of the ninth embodiment, a robot simulation method for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially picked up by the robot, A step of three-dimensionally measuring the workpiece of the posture by the sensor unit, storing the information of the inclination angle with respect to the measurement axis of the sensor unit as three-dimensional measurement data for each position, and three-dimensional measurement of the plurality of stored workpieces A step of determining an angle threshold value serving as a reference for determining whether or not three-dimensional measurement is possible based on the inclination angle information of each point constituting the data with respect to the measurement axis of the sensor unit, and a workpiece obtained by modeling the three-dimensional shape of the workpiece A step of setting a model and the plurality of set work models are connected to a virtual work space. A step of generating virtual stacking data stacked in a virtual work space, a virtual tilt angle of each point constituting each work model in the generated bulk stacking data, and the determined angle threshold And specifying a point having a virtual tilt angle larger than the angle threshold as an estimation region estimated to be difficult to measure, and simulating unstacked data not including the data of the specified estimation region And generating a simulation for verifying a bulk picking operation of the work model in the work space using the generated simulation data. As a result, it is possible to obtain simulation data after excluding data estimated to be difficult to measure with reference to the angle threshold value determined based on the tilt angle of the workpiece actually measured by the sensor unit, so that the accuracy is higher. The picking motion can be simulated.

  Furthermore, the robot simulation program according to the tenth embodiment is a robot simulation program for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially picked up by the robot, A function of storing a workpiece in a posture as a three-dimensional measurement data obtained by three-dimensionally measuring a workpiece by a sensor unit and having information on an inclination angle with respect to a measurement axis of the sensor unit for each position, and three-dimensional measurement of the plurality of stored workpieces Based on the tilt angle information of each point constituting the data with respect to the measurement axis of the sensor unit, a function for determining an angle threshold value serving as a reference for determining whether or not three-dimensional measurement is possible, and a workpiece modeling the three-dimensional shape of the workpiece A function for setting a model and a plurality of the set work models are temporarily A function of generating virtual unloading data stacked in a virtual work space that is a typical work space, a virtual inclination angle of each point constituting each work model in the generated unloading data, and A function of comparing the determined angle threshold value and specifying a point having a virtual inclination angle larger than the angle threshold value as an estimated region estimated to be difficult to measure, and a variable not including the data of the specified estimated region The computer realizes a function of generating stacking data as simulation data and a function of executing simulation for verifying the bulk picking operation of the work model in the work space using the generated simulation data. Can be made. With the above configuration, simulation data can be acquired after excluding data that is estimated to be difficult to measure with reference to an angle threshold value determined based on the tilt angle of the workpiece actually measured by the sensor unit. High picking motion simulation is possible.

  Furthermore, a computer-readable recording medium or stored device according to the eleventh embodiment stores the robot simulation program. Recording media include CD-ROM, CD-R, CD-RW, flexible disk, magnetic tape, MO, DVD-ROM, DVD-RAM, DVD-R, DVD + R, DVD-RW, DVD + RW, Blu-ray, HD A medium that can store a program such as a magnetic disk such as a DVD (AOD), an optical disk, a magneto-optical disk, a semiconductor memory, or the like is included. The program includes a program distributed in a download manner through a network line such as the Internet, in addition to a program stored and distributed in the recording medium. Furthermore, the stored devices include general-purpose or dedicated devices in which the program is implemented in a state where it can be executed in the form of software, firmware, or the like. Furthermore, each process and function included in the program may be executed by computer-executable program software, or each part of the process or hardware may be executed by hardware such as a predetermined gate array (FPGA, ASIC), or program software. And a partial hardware module that realizes a part of hardware elements may be mixed.

It is a schematic diagram which shows a mode that the bulk picking operation | movement is performed using a robot system. It is a block diagram of a robot system. 1 is a block diagram illustrating a robot simulation apparatus according to a first embodiment. It is a block diagram which shows the picking operation | movement simulation system which concerns on Embodiment 1B. It is a block diagram which shows the picking operation | movement simulation system which concerns on Embodiment 2. FIG. It is a perspective view which shows an example of a sensor part. FIG. 7A is a schematic cross-sectional view showing an example in which workpieces are randomly placed in a storage container and stacked, FIG. 7B is a schematic cross-sectional view showing an example in which workpieces are stacked on a floor surface, and FIG. It is a perspective view which shows the state arranged on the top. 8A is a schematic diagram showing an example of gripping a workpiece with an end effector, FIG. 8B is a schematic diagram showing an example of gripping a workpiece having a cavity from the inner surface, and FIG. 8C shows an example of sucking and gripping a plate-like workpiece. It is a schematic diagram. It is a block diagram which shows a picking operation | movement simulation system. It is a block diagram which shows the picking operation | movement simulation system which concerns on Embodiment 4. FIG. It is a block diagram which shows the robot simulation apparatus which concerns on Embodiment 5. It is a block diagram which shows the robot simulation apparatus which concerns on Embodiment 6. FIG. It is a block diagram which shows the robot simulation apparatus which concerns on Embodiment 7. FIG. It is a flowchart which shows the setting procedure of a simulation environment. FIG. 15A is a perspective view showing a work model, and FIG. 15B is a perspective view showing a state where an area is designated on the work model of FIG. 15A. FIG. 16A is a perspective view showing the work model of FIG. IA in a wire frame, and FIG. 16B is a perspective view showing a state in which an area is designated on the work model of FIG. 16A. It is a perspective view which shows an example of a work model. It is an expanded view of the work model of FIG. FIG. 19A is a schematic diagram for explaining a Lambert reflection model, and FIG. 19B is a schematic diagram for explaining a Phong specular reflection model. It is a graph which shows the example of distribution of reflected light. FIG. 21A is a schematic diagram showing the relationship between the incident light, the observation point direction, and the regular reflection direction, and FIG. 21B is a schematic diagram showing how the angle of the workpiece is changed with respect to the incident light and the observation point direction. It is a graph which shows the ratio of incident light and reflected light when changing an inclination angle with the workpiece | work of FIG. 21B. 23A is an image diagram showing an example of a two-dimensional display of a height image having a high diffuse reflection ratio, FIG. 23B is a perspective view of FIG. 23A three-dimensionally displayed, and FIG. 23C is an example of a height image having a high specular reflection ratio. FIG. 23D is a perspective view of FIG. 23C. FIG. 10 is a functional block diagram of a robot simulation apparatus according to an eighth embodiment. It is a flowchart which shows the procedure of picking operation | movement simulation. It is a flowchart which shows the procedure which produces | generates the unpacking state of a work model by physical simulation. It is an image figure which shows the unloading data which unpacked the cylindrical work model. 28A is an initial state in which cylindrical workpiece models are stacked, FIG. 28B is a state retroactive 20 times, FIG. 28C is a state retrospectively 10 times, and FIG. 28D is an image diagram showing the end state. is there. FIG. 29A is a flowchart illustrating a part of a physical simulation procedure in consideration of data storage. FIG. 29B is a flowchart showing a physical simulation procedure in consideration of data storage following FIG. 29A. It is a schematic diagram which shows the state where the T-shaped workpiece | work exists in the unstable attitude | position. It is a flowchart which shows the procedure which adds a restriction | limiting to the attitude | position at the time of dropping of a workpiece | work model, and produces | generates a stacking state. It is a flowchart which shows the procedure which adds a restriction | limiting to the attitude | position after dropping of a workpiece | work model, and produces | generates a stacking state. It is a flowchart which shows the procedure which adds a restriction | limiting to the attitude | position at the time of dropping of a workpiece | work model, and after dropping, and produces | generates a stacking state. 34A is a perspective view showing the front surface of the workpiece, and FIG. 34B is a perspective view showing the back surface of the workpiece. It is a perspective view which shows the state where the workpiece | work of FIG. 34A and FIG. 34B was piled up. FIG. 36A is a perspective view illustrating a state in which workpieces are aligned and packed in the storage container, and FIG. 36B is a perspective view illustrating a state in which the workpieces are stacked in the storage container. It is a flowchart which shows the procedure which adds the restriction | limiting to the position and attitude | position of the workpiece | work model which concerns on Embodiment 9, and produces | generates a stacking state. It is an image figure which shows an example of the basic posture setting screen of a work model. It is an image figure which shows an example of a posture condition setting screen. It is an image figure which shows the example of the attitude | position of a workpiece | work model at the time of designating the inclination-angle of a workpiece | work model to 0 degree and rotation angle +/- 180 degrees. It is an image figure which shows the example of the attitude | position of a workpiece | work model at the time of designating the inclination-angle of a workpiece | work model to 15 degrees and rotation angle +/- 180 degrees. It is an image figure which shows an example of an arrangement condition setting screen. 43A is an image diagram showing an example in which work models are completely randomly arranged in the storage container, FIG. 43B is an example in which the work model is arranged in one direction, and FIG. 43C is an image diagram showing an example in which the work model is arranged in multiple directions and multiple stages. It is a flowchart which shows the calculation method of attitude | position stability. 45A is a perspective view showing a T-shaped workpiece, FIG. 45B is a perspective view showing a state in which a circumscribed cube is set with respect to FIG. 45A, and FIG. 45C is a plan view showing a surface taking the maximum area of FIG. 45B. 46A is a perspective view showing a state in which the workpiece of FIG. 45A is placed in an inclined posture, and FIG. 46B is a plan view showing a region of the contact portion of FIG. 46A. 47A is a perspective view showing a state in which the workpiece of FIG. 45A is placed in an upright posture, and FIG. 47B is a plan view showing a region of the contact portion of FIG. 47A. 48A is a perspective view showing a state in which the workpiece of FIG. 45A is placed in a posture leaning against the wall surface, and FIG. 48B is a plan view showing a region of the contact portion of FIG. 48A. 49A is a plan view of FIG. 46B, and FIG. 49B is a plan view showing a state in which a minimum area circumscribed rectangle is obtained from FIG. 49A. FIG. 50A is a plan view of FIG. 47B, and FIG. 50B is a plan view showing a state in which a minimum area circumscribed rectangle is obtained from FIG. 50A. 51A is a plan view of FIG. 48B, and FIG. 51B is a plan view showing a state in which a minimum area circumscribed rectangle is obtained from FIG. 51A. It is a block diagram which shows the robot simulation apparatus which concerns on Embodiment 10. FIG. It is a block diagram which shows the robot simulation apparatus concerning Embodiment 11. It is a schematic cross section which shows the example which arranged the workpiece | work in the storage container in multiple steps through the insole. It is a schematic cross section which shows the example which arranged the workpiece | work through the partition in the storage container. It is an image figure which shows the work model group piled up. It is an image figure which shows the state which removed the workpiece | work model hatched in FIG. FIG. 57 is an image diagram showing a result of re-execution of a physical simulation in a state where a hatched work model is removed in FIG. It is a flowchart which shows the detail of the procedure which reduces the number of work models by one in picking operation | movement simulation. 60A is a perspective view of bulk stacking data, FIG. 60B is a bulk stacking image obtained by converting FIG. 60A into a height image, FIG. 60C is a perspective view showing FIG. 60B as a point group, and FIG. 60E is a perspective view showing FIG. 60D as a point cloud. FIG. 61A is a schematic cross-sectional view showing an angle range in which a three-dimensional shape of a work having a weak specular reflection can be acquired, and FIG. 61B is a schematic cross-sectional view showing an angle range in which a three-dimensional shape of a work having a strong specular reflection can be acquired. FIG. 62A is an image diagram showing bulk loading data obtained by bulking resin workpieces, FIG. 62B is an enlarged view of the main part of FIG. 62A in which the resin workpiece is enlarged, and FIG. FIG. 62D is an image diagram showing the stacked bulk data, and FIG. 62D is an enlarged view of a main part in which a metal workpiece is enlarged in FIG. 62C. It is an image figure which shows a rose stacking image. It is a schematic diagram which shows the state which cannot see a workpiece | work from the camera side by a pattern projection system. FIG. 65A is a schematic diagram illustrating an example in which a workpiece is visible from the camera, FIG. 65B is an example in which the workpiece is not visible from the camera, and FIG. 65C is an example in which the workpiece is visible from any of a plurality of cameras. 22 is a flowchart illustrating a procedure of picking operation simulation according to the twelfth embodiment. 18 is a flowchart showing a procedure for generating a workpiece model unstacked state in a physical simulation according to a twelfth embodiment. It is a flowchart which shows the procedure which produces | generates a piled-up image. It is a schematic diagram which shows the attention position on a workpiece | work and the straight line which connects with each camera. It is a schematic diagram which shows a mode that it checks whether there is an obstruction on the straight line which connects a camera and an attention position. It is a block diagram which shows the robot simulation apparatus which concerns on Embodiment 13. FIG. It is an image figure which shows the height image which actually measured three-dimensionally the workpiece | work group piled up. It is a graph which shows the angle distribution histogram of resin workpieces. It is a graph which shows the angle distribution histogram of metal workpieces. It is a flowchart which shows the setting procedure of a simulation environment. It is an image figure which shows the height image which imaged the workpiece | work group of the piled-up state. FIG. 77A is a profile of actually captured 3D measurement data, and FIG. 77B is a diagram showing a state in which 3D search results are superimposed on FIG. 77A. 16 is a flowchart illustrating a simulation environment setting procedure according to the fourteenth embodiment. FIG. 79A is an image diagram showing a state in which a certain workpiece is displayed on the simulation, and FIG. 79B is an image diagram showing a state measured during actual operation of the workpiece in FIG. 79A.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiment described below is an example for embodying the technical idea of the present invention, and the present invention is not limited to the following. Moreover, this specification does not specify the member shown by the claim as the member of embodiment. In particular, the dimensions, materials, shapes, relative arrangements, and the like of the component parts described in the embodiments are not intended to limit the scope of the present invention only to specific examples unless otherwise specifically described. Only. Note that the size, positional relationship, and the like of the members shown in each drawing may be exaggerated for clarity of explanation. Furthermore, in the following description, the same name and reference sign indicate the same or the same members, and detailed description will be omitted as appropriate. Furthermore, each element constituting the present invention may be configured such that a plurality of elements are configured by the same member and the plurality of elements are shared by one member. It can also be realized by sharing.
(Robot system)

A workpiece stacking picking operation using the robot system 1000 is performed. A configuration example of such a robot system 1000 is shown in FIG. 1, and a block diagram of the robot system 1000 is shown in FIG. In such bulk picking, a plurality of workpieces WK stacked in the work space are sequentially taken out using a robot and arranged at predetermined positions. For example, a large number of parts randomly placed in the storage container BX are acquired by a sensor unit such as a camera or an illumination, and a three-dimensional shape is obtained. One piece of work WK is provided at the end of the arm ARM of the robot. It is gripped by the effector EET and arranged in a predetermined position, for example, a conveyor belt shape.
(Picking motion simulation)

When performing such a bulk picking operation with such a robot system 1000, it is performed to perform a simulation in advance and verify the bulk picking operation. For example, depending on the installation position of the robot, the work container BX, the relative position of the placement position, the height, and the like, the work may not be correctly gripped. Also, depending on the position and angle at which the sensor unit is installed, the workpiece cannot be imaged correctly, and it is difficult to accurately measure the three-dimensional shape. Therefore, a robot picker and a three-dimensional picking motion simulation of a place motion for performing picking in advance are performed using a robot simulation device. In the robot simulation apparatus, a CAD model having a three-dimensional shape of a work is used as a work model and is randomly arranged in a virtual work space, which is a virtual work space, to simulate a robot picking process, that is, bulk picking. When the user determines that the picking has not been sufficiently successful in such a simulation, the user corrects the parameter for operating the robot and restarts the simulation.
(Embodiment 1)
(Robot simulation device)

  Therefore, in the first embodiment, the third order excluding the parts that cannot be measured based on the posture of the workpiece model in the stacked state so that the simulation can be performed as close as possible to the state in which the actual workpiece is three-dimensionally measured. Original measurement data is generated, and a picking operation simulation is performed based on the generated measurement data. Specifically, based on the angle of the surface of the workpiece model, which is determined from the posture of the stacking state, the part with a steep angle above a certain level is excluded from the stacking data generated by the picking operation simulation as being impossible to measure three-dimensionally. . Accordingly, it is possible to realize a state close to a state in which an image of a piece-by-piece workpiece is actually picked up without performing a troublesome confirmation operation or setting adjustment such as installing a sensor unit or actually preparing a workpiece.

  FIG. 3 shows a block diagram of the robot simulation apparatus 100 according to the first embodiment of the present invention. The robot simulation apparatus 100 shown in FIG. 3 includes a work model setting unit 11, a stacking data generation unit 20, a stacking data storage unit 23, a picking motion simulation unit 30, and a simulation parameter adjustment unit 29.

  The work model setting unit 11 is a member for setting a work model obtained by modeling the three-dimensional shape of a work.

  The bulk stacking data generation unit 20 is a member for generating bulk stacking data in which a plurality of work models are stacked in the virtual work space in accordance with the posture condition set by the posture condition setting unit 16.

  The picking motion simulating unit 30 is a member for executing the bulk picking motion simulation for verifying the picking motion of the work model in the virtual work space with respect to the bulk stacking data generated by the bulk stacking data generation unit 20. is there.

  The bulk stacking data storage unit 23 is a member for storing at least one of bulk stacking data before execution of the picking motion simulation by the picking motion simulator 30 and bulk stacking data during execution of the picking motion simulation.

  The simulation parameter adjustment unit 29 is a member for adjusting simulation parameters related to the robot operation when there is a work model that cannot be taken out by the robot as a result of the picking operation simulation by the picking operation simulation unit 30. .

The picking motion simulating unit 30 uses the simulation parameters adjusted by the simulation parameter adjusting unit 29 to read the bulk stacking data stored in the bulk stacking data storage unit 23 before or during the simulation and picking motion The simulation is re-executed. Thereby, by storing the bulk data used in the picking operation simulation, it is easy to determine whether the result of adjusting the simulation parameters is effective.
(Display unit 3)

The robot simulation apparatus can also include a display unit 3 that can display the bulk data stored in the bulk data storage unit 23. As a result, the user can adjust the simulation parameters while visually confirming the stored stacking data, and the operability can be improved.
(Embodiment 1B)

Furthermore, the robot simulation apparatus, when there is a work model that cannot be picked up by the robot as a result of the picking motion simulation by the picking motion simulation unit 30, causes the work model pick-up failure to have a plurality of predefined causes. A cause analysis unit 24 for analyzing which of the candidates corresponds can be provided. This makes it easy to take measures to correct the problem by displaying the cause of the extraction failure that has become impossible to extract, and adjusting the appropriate settings while simulating the bulk picking operation Can be easily performed. Such an example is shown in FIG. 4 as Embodiment 1B. The robot simulation apparatus shown in this figure includes a work model setting unit 11, a bulk data generation unit 20, a bulk data storage unit 23, a picking motion simulation unit 30, a simulation parameter adjustment unit 29, and a display unit 3. And a cause analysis unit 24. Also in Embodiment 1B, the same members as those in Embodiment 1 described above are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate. Details of the cause analysis unit 24 will be described later.
(Embodiment 2)

  In the picking motion simulation, even if it is determined that the robot can be gripped by an end effector model simulating an end effector of a robot, it may not be possible to perform picking in bulk when attempting to actually operate. This is because, in the picking motion simulation, the determination is made with the work model WM having a virtual three-dimensional shape as shown in FIG. 79A. caused by. For example, when specular reflection is strong, such as when the workpiece is made of metal, a portion that cannot be measured is generated depending on the posture of the workpiece. For example, as shown in FIG. 79B, as a result of imaging with the sensor unit from above the work space, a three-dimensional shape cannot be measured for a portion where the inclination of the workpiece WK becomes steep. Furthermore, there is a problem that when a plurality of workpieces overlap each other, the shape of the workpiece on the lower side cannot be measured three-dimensionally. As a result, although a three-dimensional shape can be searched on the simulation, the search sometimes fails during actual operation.

Therefore, in the second embodiment, the third order excluding the part that cannot be measured based on the posture of the workpiece model in the stacked state so that the simulation can be performed as close as possible to the state in which the actual workpiece is three-dimensionally measured. Original measurement data is generated, and a picking operation simulation is performed based on the generated measurement data. Specifically, based on the angle of the surface of the workpiece model, which is determined from the posture of the stacking state, the part with a steep angle above a certain level is excluded from the stacking data generated by the picking operation simulation as being impossible to measure three-dimensionally. . Accordingly, it is possible to realize a state close to a state in which an image of a piece-by-piece workpiece is actually picked up without performing a troublesome confirmation operation or setting adjustment such as installing a sensor unit or actually preparing a workpiece.
(Physical simulation part)

  In addition, the robot simulation apparatus can also generate bulk stacking data by performing a physical simulation that virtually reproduces the bulk stacking state of stacked workpieces. Such an example is shown in FIG. The robot simulation apparatus shown in this figure includes a work model setting unit 11, a posture condition setting unit 16, a physical simulation unit 60, a bulk data generation unit 20, a bulk data storage unit 23, a region estimation unit 22, And a picking motion simulating unit 30. This robot simulation apparatus performs physical simulation that virtually reproduces a stacked state in which workpieces are stacked, and generates bulk data.

  As a result, it is possible to generate bulk loading data that matches the user's actual operating environment, such as excluding work models with a clearly unnatural posture, so that picking motion simulation can be performed, so actual operation without actually preparing work Appropriate setting adjustments and verifications can be made in advance in a manner close to the time.

  The unloading data generation unit 20 can generate unloading data excluding a work model that does not match the posture condition set by the posture condition setting unit 16. Alternatively, the bulk data generation unit 20 may be configured to generate bulk data excluding a work model having a physically low probability of occurrence. In this case, in addition to the user specifying a work model with a physically low probability of occurrence with the posture condition setting unit 16, a condition for the work model with a physically low probability of occurrence is prepared in advance on the robot simulation apparatus side. You can also. The setting of the posture condition based on the specified value is also included in the posture condition setting unit 16.

  Also, the bulk data generation unit 20 creates bulk data when a work model that does not match the posture condition set by the posture condition setting unit 16 exists in the bulk data, or when there is a predetermined ratio or more. You may comprise so that it may redo. As a result, if bulk data far from actual operation is obtained, the bulk data is recreated without performing the picking operation simulation, and only picking operation simulation is performed for the bulk data close to the actual operation. By performing the above, a more accurate simulation result can be obtained.

  The posture condition setting unit 16 uses, as posture conditions, conditions related to postures that can be taken by the work model when the work model set by the work model setting unit 11 is placed in a virtual work space that is a virtual work space. It is a member for setting. The posture condition may be set manually by the user, or may be presented in advance as a default value on the robot simulation apparatus side. The initial value of the posture condition presented by the robot simulation device may be adjustable by the user. The posture condition setting unit 16 can be configured to be able to set the allowable rotation angle and tilt angle from the reference posture of the work model as posture conditions. The posture condition setting unit 16 projects the contact position of the workpiece model with the floor surface or another workpiece model from directly above, the area of the circumscribed rectangle surrounding the projected contact position, and the maximum of the maximum circumscribed rectangle of the workpiece model. The area ratio may be configured to be set as the posture condition. Further, the posture condition setting unit 16 has a height position between the position of contact with the floor of the work model, the position of contact with another work model, or the contact position located at the bottom of the contact surface and the position of the center of gravity. It is also possible to configure so that the difference can be set as the posture condition. A workpiece model supported at a low position with respect to the center of gravity can be regarded as unstable. Furthermore, the posture condition setting unit 16 may be configured to be able to set the posture condition so that a specific surface of the work model is a bottom surface. In addition, the posture condition setting unit 16 can be configured to be able to set the arrangement interval in the X direction and / or the Y direction between the work models and the number of stages of the work models as posture conditions.

  The physical simulation unit 60 uses the work model set by the work model setting unit 11 to simulate the operation of inputting the work into the work space under the action of gravity by inputting the work model into the virtual work space. It is a member for. In addition, simulation in which such a work model is input to the virtual work space is referred to as physical simulation or input operation simulation. In addition, in the present specification, “throwing” and “throwing” are not limited to the operation of physically dropping from a high place, for example, in order to verify rolling, dropping while extruding in a horizontal direction from a predetermined height, It is used in the sense that it includes the action of throwing it in a parabolic shape and dropping it, or placing or placing the work model on the storage container or floor. Furthermore, in the physical simulation, in addition to sequentially inputting the work models one by one into the work space, it is also possible to simulate the operation of simultaneously inputting a plurality of work models, for example, dropping or standing.

  The physical simulation is repeated a plurality of times or collectively, and the result obtained, that is, based on the work model group in which a plurality of work models are stacked randomly or in a predetermined manner, The generation unit 20 generates virtual three-dimensional loose stacking data in which a plurality of work models are stacked in the virtual work space.

  The picking operation simulating unit 30 is a member for verifying the unloading picking operation of the work model in the virtual work space with respect to the generated unloading data. As a result, troublesome confirmation work and setting adjustment, such as whether or not to detect the three-dimensional position and orientation of the workpiece, can be performed in advance in a form close to actual operation without actually preparing the workpiece (details will be described later). ).

  In particular, the physics simulation unit 60 grips one workpiece model by the picking motion simulation unit 30 and starts at least the extraction operation. Then, the physics simulation unit 60 again performs physical simulation on the stacked data in a state after the workpiece model is extracted. Run. The timing for executing the physical simulation again may be after the work model is held and placed at the placement position, or may be in the middle of the work model take-out operation. That is, when the work model is gripped by the end effector model and this work model is lifted, the placement of another work model around this work model moves as one work model is removed Therefore, there is no need to wait until the gripped work model is placed. Therefore, the re-execution timing of the physics simulation is the timing when one work model is gripped and lifted, or after waiting until the other work model is moved and stabilized by the action of gravity. be able to. For example, it may be after a lapse of a predetermined time (for example, after 10 seconds) from the time when the work model is gripped and lifted.

  In response to this, the bulk data generation unit 20 updates the bulk data during the picking motion simulation by the picking motion simulator 30 according to the result of the physical simulation re-execution by the physical simulation unit 60. In this way, the picking motion simulating unit 30 also performs an operation by physical simulation, such as when one work model is extracted from the bulk data during the picking motion simulation, the work model group stacked in accordance with this is destroyed. Therefore, a more accurate simulation of bulk picking is realized.

Further, when the stacking data generation unit 20 excludes a work model that does not match the posture condition set by the posture condition setting unit 16, the physical simulation unit changes the state after the work model that does not match the posture condition is excluded. It is also possible to configure so that the physical simulation is applied again to the stacked data. As a result, work models with postures that cannot physically exist, such as floating in the air, can occur during physical simulation.By eliminating such work models, highly reliable physical simulation results can be obtained. Can be obtained.
(Initial posture)

  The physical simulation unit may be configured to determine an initial posture of the work model based on the posture condition set by the posture condition setting unit 16 and to execute a physical simulation for dropping the workpiece in the initial posture into the work space. Further, when the physical simulation is executed by the physical simulation unit, a work model that does not match the posture condition set by the posture condition setting unit 16 may be excluded.

  These members can be realized by, for example, a microprocessor (MPU), a CPU, an LSI, a gate array such as an FPGA or an ASIC, hardware or software such as a DSP, or a mixture thereof. In addition, each component does not necessarily have the same configuration as that shown in FIG. 3 and the like, and its functions are substantially the same, and one element has the functions of a plurality of elements in the configuration shown in FIG. Are included in the present invention.

  For example, in the example of FIG. 5, the physical simulation unit 60 is prepared separately from the bulk data generation unit 20, but the physical simulation unit may be integrated into the bulk data generation unit. Alternatively, the physical simulation unit and the bulk data generation unit may be integrated into the picking motion simulation unit.

  On the other hand, the robot system 1000 shown in the block diagram of FIG. 2 includes a sensor unit 2, an image processing unit 1, a display unit 3, an operation unit 4, a robot body 5, a robot controller 6, and a robot operation tool 7. Is provided. The sensor unit 2 is a member that images a work space and measures a three-dimensional shape. The image processing unit 1 is a member that performs a three-dimensional search, interference determination, and gripping solution calculation. The sensor unit 2 may be configured to measure a three-dimensional shape on the image processing unit 1 side only by imaging the work space. Methods for acquiring a three-dimensional shape include a pattern projection method, a stereo method, a lens focus method, a light cutting method, an optical radar method, an interference method, a TOF method, and the like. In this embodiment, the phase shift method is used in the pattern projection method.

  The configuration of the sensor unit 2 is determined according to such a three-dimensional shape measurement technique. The sensor unit 2 includes a camera, illumination, a projector, or the like. For example, when measuring the three-dimensional shape of the workpiece by the phase shift method, the sensor unit 2 includes a projector PRJ and a plurality of cameras CME1, CME2, CME3, and CME4 as shown in FIG. The sensor unit 2 may be configured integrally with a plurality of members such as a camera and a projector. For example, a 3D imaging head that is formed into a head shape by integrating a camera and a projector can be used as the sensor unit.

  The image processing unit 1 performs a three-dimensional search, interference determination, gripping solution calculation, and the like based on the workpiece stacking data obtained in this way. The image processing unit 1 can use a general-purpose computer installed with a dedicated image processing program, a dedicated controller, or the like. 2 shows an example in which the image processing unit 1 is configured by a separate member. However, the present invention is not limited to this configuration. For example, the sensor unit and the image processing unit are integrated, or the robot controller performs image processing. Parts can also be incorporated. Moreover, you may comprise so that the three-dimensional shape of a workpiece | work may be acquired on the image processing part side based on the data imaged on the sensor part side.

  The display unit 3 is a member for displaying the three-dimensional shape of the workpiece acquired by the image processing unit 1 and for confirming various settings and operation states. A liquid crystal monitor, an organic EL display, a CRT, or the like can be used. . The operation unit 4 is a member for performing various settings such as image processing, and an input device such as a keyboard or a mouse can be used. Further, by using the display unit 3 as a touch panel, the operation unit and the display unit can be integrated.

  For example, when the robot simulation apparatus or the image processing unit is a computer installed with a robot simulation program, a graphical user interface (GUI) screen of the robot simulation program is displayed on the display unit. Various settings can be made from the GUI displayed on the display unit, and processing results such as simulation results can be displayed. In this case, the display unit can be used as a setting unit for performing various settings.

  The robot controller 6 controls the operation of the robot based on information captured by the sensor unit 2. The robot operation tool 7 is a member for setting the operation of the robot body 5, and a pendant or the like can be used.

  The robot body 5 includes a movable arm ARM and an end effector EET fixed to the tip of the arm ARM. The robot body 5 is controlled by the robot controller 6 to operate the arm ARM, pick one workpiece WK, move it to a desired position, place it, and release it. Therefore, an end effector EET for gripping the workpiece WK is provided at the tip of the arm ARM. Examples of the placement position for placing the workpiece WK include a tray and a conveyor.

  As shown in FIG. 1, a plurality of workpieces WK are randomly stored in a storage container BX such as a return box. The sensor unit 2 is disposed above the work space. The sensor unit 2 includes a camera and illumination, and the sensor unit 2 can measure the three-dimensional shape of the workpiece WK. Based on the three-dimensional shape of the workpiece WK measured by the sensor unit 2, the robot controller 6 identifies the workpiece WK to be gripped from a plurality of workpieces, and controls the robot to grip the workpiece WK. To do. Then, while holding the work WK, the arm ARM is operated to move to a predetermined placement position, and the work WK is placed in a predetermined posture. In other words, the robot controller 6 is a position where the sensor unit 2 identifies the workpiece WK to be picked, grips the workpiece WK with the end effector EET, and places the gripped workpiece WK in a predetermined reference posture. The operation of the robot is controlled so that the end effector EET is opened by being placed at the placement position.

  Here, in the present specification, bulk picking means that a work WK randomly placed in a storage container BX as shown in FIG. 7A is gripped by a robot and placed at a predetermined position. An example of gripping and placing the workpiece WK stacked in a predetermined area without using the storage container as shown in FIG. 7B, or a workpiece stacked in a predetermined posture as shown in FIG. 7C It is also used to include an example of sequentially holding and placing WK.

  In the example of FIG. 1, the sensor unit 2 is fixed above the work space. However, any position where the work space can be imaged is sufficient. For example, the sensor unit 2 can be arranged at an arbitrary fixed position such as obliquely, laterally, or below. However, the aspect which arrange | positions a sensor part in the indefinite position which moves, such as on the arm ARM is excluded. Further, the number of cameras and illuminations included in the sensor unit is not limited to one and may be plural. Furthermore, the connection with the sensor unit 2, the robot, and the robot controller 6 is not limited to a wired connection, and may be a wireless connection.

  In addition to gripping the workpiece WK as shown in FIG. 8A, the workpiece gripping is performed by inserting the claw portion of the end effector EET2 into the hollow workpiece WK2 as shown in FIG. It is used to include an example of gripping by gripping and an end effector EET3 that sucks and grips a plate-like workpiece WK3 as shown in FIG. 8C. Below, the aspect which hold | grips the outer side surface of a workpiece | work from both sides is demonstrated as an example of the holding | grip of a workpiece | work. Also, as shown in FIG. 1, a plurality of workpieces are stored in a storage container BX and randomly stacked, and one by one is picked by the end effector EET for each of the plurality of workpieces WK. The simulation of the bulk picking operation that repeats the work of placing at the placement position will be described below.

Next, FIG. 9 shows a block diagram of a system configuration at the time of picking operation simulation. The picking operation simulation system shown in this figure includes an image processing unit 1, a display unit 3, and an operation unit 4. In this configuration, all simulations are performed on the image processing unit 1. That is, the member that becomes the image processing unit 1 during actual operation functions as a robot simulation device during a picking motion simulation. However, the configuration is not limited to the configuration in which the robot simulation device is also used as the image processing unit, and the robot simulation device can be prepared separately from the image processing unit.
(Embodiment 4)

In addition, the present invention is not limited to a configuration in which a picking motion simulation is directly executed on a work model group stacked by physical simulation, and three-dimensional measurement is possible for each part of the stacked work model group. It may be configured to perform a picking operation simulation based on the determination result. Such an example is shown in FIG. 10 as a robot simulation apparatus according to the fourth embodiment. The robot simulation apparatus shown in this figure includes a work model setting unit 11, a posture condition setting unit 16, a stacking data generation unit 20, a region estimation unit 22, a picking motion simulation unit 30, and a display unit 3. Prepare. In the fourth embodiment, the same members as those of the first embodiment described above are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate.
(Area Estimator 22)

  The region estimation unit 22 illustrated in FIG. 10 is a member for specifying an estimation region in which three-dimensional measurement is estimated to be difficult by the sensor unit based on the position and orientation of each work model in the stacking data.

  Based on the estimated area data obtained in this way, the picking motion simulation unit 30 executes a picking motion simulation for verifying the bulk picking motion of the work model in the virtual work space. That is, the picking motion simulation unit 30 removes the estimated area from the bulk data and executes the picking motion simulation.

In the above example, the estimation region estimated by the region estimation unit is the region where the sensor unit is estimated to be difficult to perform the three-dimensional measurement. It is good also as an estimation area.
(Embodiment 5)

In addition, a simulation data generation unit that generates simulation data obtained by removing the estimation region from the bulk data may be added. Such an example is shown in FIG. The robot simulation apparatus 200 shown in this figure includes a work model setting unit 11, an attitude condition setting unit 16, a physical simulation unit 60, a stacking data generation unit 20, a region estimation unit 22, and a picking motion simulation unit 30. A simulation data generation unit 40 and a display unit 3. In the fifth embodiment, the same members as those in the above-described first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate.
(Simulation data generator 40)

  The simulation data generation unit 40 is a member for generating, as simulation data, unpacked data that does not include the estimated region data specified by the region estimation unit 22. The picking operation simulation unit 30 executes a picking operation simulation using the simulation data generated by the simulation data generation unit 40.

  Further, the display unit 3 can display the bulk data generated by the bulk data generation unit 20. This bulk data can be updated in real time and displayed on the display unit 3 when it is updated by the physical simulation unit 60. As a result, after the work model is taken out as in the past, the physical simulation is not re-executed and the state where a floating work model exists is avoided, and the physical simulation is re-executed on the display unit 3 By displaying the result, it is possible to display the stacked workpieces without any discomfort to the user.

Furthermore, the display unit 3 can display the simulation data generated by the simulation data generation unit 40 and allow the user to confirm the data. At this time, as shown in FIG. 60D, FIG. 62A, and FIG. 62C described later, an area that cannot be measured in actual operation may be displayed separately from the measurable area. This makes it easier for the user to visually understand what the user sees during actual operation.
(Embodiment 6)

  Furthermore, a sensor model setting unit for setting a virtual sensor model can be added in the picking operation simulation. Such an example is shown in FIG. 12 as a robot simulation apparatus 300 according to the sixth embodiment.

A robot simulation apparatus 300 according to the sixth embodiment illustrated in FIG. 12 includes a simulation environment setting unit 10, a detection setting unit 50, a stacking data generation unit 20, a region estimation unit 22, and a simulation data generation unit 40. The picking operation simulation unit 30 and the display unit 3 are provided. The bulk data generation unit 20 includes a physical simulation unit 60. Also in the sixth embodiment, the same members as those in the first embodiment described above are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate.
(Detection setting unit 50)

The detection setting unit 50 is a member for performing various settings for detecting the position and orientation, determining the interference, and calculating the gripping solution. This setting includes settings related to position and orientation detection (three-dimensional search) (for example, data thinning amount, posture restriction, specification of important feature parts, etc.), end effector gripping posture setting for workpieces, etc. Is mentioned.
(Simulation environment setting unit 10)

The simulation environment setting unit 10 is a member for performing various settings for acquiring a stacked image. The setting of the simulation environment here includes the number of workpieces to be stacked, information on the storage container, floor information, design information of the cameras and projectors that constitute the sensor unit, and the like. Further, at the time of picking operation simulation, it is possible to generate a random misalignment that may occur in actual operation with respect to the storage container, and the range of misalignment at this time can be included in the setting of the simulation environment. The simulation environment setting unit 10 includes a work model setting unit 11, a sensor model setting unit 15, and an attitude condition setting unit 16.
(Work model setting unit 11)

The work model setting unit 11 sets conditions related to the work model. As the work model, CAD data created by CAD can be used. The CAD data format of the workpiece CAD data is not particularly limited, but in the present embodiment, the simplest STL format is used as the three-dimensional CAD data. STL is data composed only of a list of triangular polygon information (coordinates of three points and normal vectors of the surface). Alternatively, the work model may be constituted by point cloud data having three-dimensional information (a point cloud display image displayed on the display unit 3 is shown in FIG. 60E described later). Alternatively, the work model may be constituted by image data having height information, for example, a height image or a distance image.
(Surface condition parameter setting section)

The work model setting unit 11 may include a surface state parameter setting unit 12. The surface state parameter setting unit 12 is a member for inputting parameters relating to the surface state of the work model. The surface condition parameter defines, for example, the reflectance of the workpiece surface by a numerical value. Alternatively, the material of the workpiece is specified. In the case of the individual, “metal”, “glass”, “resin”, etc. may be cited as options and the user may select one of them. Alternatively, it may be selected from sensual options for distinguishing the surface state, for example, “sparkle”, “roughness”, “transparent”, “matte”, “white”, “black”. An angle threshold value (threshold value) to be described later can be changed by the surface condition parameter set by the surface condition parameter setting unit 12.
(Container model setting unit 14)

  Furthermore, the simulation environment setting unit 10 may include a container model setting unit 14 that sets a container model obtained by modeling a storage container for storing a work such as a returnable box. In this case, the estimation region can be estimated based on the relative positional relationship between the sensor model and the container model.

  The posture condition setting unit 16 can be configured to be able to set posture conditions so that a specific surface or side of the work model has a specific relationship with the container model.

The physical simulation unit may be configured to execute a physical simulation in which the number of stages of the container model set by the container model setting unit is set and a plurality of work models are simultaneously dropped into the container model.
(Sensor model setting unit 15)

  The sensor model setting unit 15 sets a camera model or a projector model that virtually indicates the position and orientation of any or all of the members constituting the sensor unit, for example, a camera for performing three-dimensional measurement of a workpiece. Whether the sensor model includes only the camera model or includes the projector model depends on a method for measuring the three-dimensional shape of the workpiece adopted by the sensor unit and a member for realizing the method. For example, in the case of the passive stereo system, only a camera model is sufficient because a projector is not necessary.

  In this case, the region estimation unit 22 can be configured to estimate a region that becomes a blind spot as viewed from the sensor model set by the sensor model setting unit 15 as a blind spot region that is difficult to measure. The definition of the blind spot area differs depending on the type of sensor model set by the sensor model setting unit 15. For example, when the three-dimensional measurement method employed in the sensor unit is a fringe projection method or a light cutting method, a blind spot region from either the camera model or the projector model is a blind spot region. In the case of the stereo method using a plurality of cameras, a blind spot region from any one of a plurality of camera models is a blind spot region.

Further, the blind spot area may be changed according to the position in the plane of the work model. Since the blind spot is more likely to occur in the peripheral part than in the central part of the measurement visual field, the accuracy of the simulation can be increased by changing the blind spot area corresponding to this.
(Update simulation data)

  Furthermore, when any one work model is taken out during execution of the picking motion simulation, it is possible to re-estimate a region that can be measured by the region estimation unit 22 with respect to the remaining work model and update the simulation data. . That is, when any one of the work models is extracted during the simulation of the bulk picking operation performed by the picking operation simulating unit 30, the simulation data generation unit 40 re-estimates a measurable region for the remaining work models. The simulation data can be updated. As a result, once a picking motion simulation has been executed in the past, the simulation data is maintained as it is even if the work models are sequentially taken out, leaving an unnatural work model that floats in the air. However, according to the fifth embodiment, it is possible to measure the work model positioned below, considering the change of the remaining work model that may occur when the upper work model is taken out. Accurate simulation of bulk picking is realized.

In this operation, in the first embodiment described above and other embodiments described later, the blind spot area re-estimated by the area estimation unit 22 is read again by the picking operation simulation unit 30 and the picking operation simulation result is updated. be able to.
(Embodiment 7)

  In the above example, based on the position and orientation of each work model in the stacking data, the region estimation unit estimates whether the sensor unit can perform three-dimensional measurement and identifies the estimation region. However, the present invention is not limited to this configuration, and can be configured to estimate the possibility of three-dimensional measurement according to the surface state of the workpiece model, for example. Such an example will be described as a robot simulation apparatus according to Embodiment 7 with reference to FIG. The robot simulation apparatus 600 shown in this figure includes a simulation environment setting unit 10, a bulk data generation unit 20, a region estimation unit 22, a simulation data generation unit 40, a picking motion simulation unit 30, and a display unit 3. Is provided. The bulk data generation unit 20 includes a physical simulation unit 60. Also in the seventh embodiment, the same members as those in the first embodiment described above are denoted by the same reference numerals and detailed description thereof is omitted as appropriate.

  The simulation environment setting unit 10 includes a work model setting unit 11 and an attitude condition setting unit 16. The work model setting unit 11 includes a surface state parameter setting unit 12 and an angle threshold determination unit 13. The surface state parameter setting unit 12 is a member for setting a surface state parameter related to the surface state of the work model. Based on the surface condition parameters set by the surface condition parameter setting unit 12, the angle threshold value determination unit 13 determines the angle threshold value of the workpiece model surface with respect to the measurement axis of the sensor unit that makes three-dimensional measurement difficult by the sensor unit. . Then, the region estimation unit identifies a region in the bulk data having an inclination angle larger than the angle threshold determined by the angle threshold determination unit 13 as an estimation region where it is estimated that three-dimensional measurement is difficult. As a result, the simulation data generation unit generates unstacked data that does not include the estimated region data estimated by the region estimation unit as simulation data, and the picking motion simulation unit 30 uses the simulation data. To execute a picking operation simulation. As a result, the result of three-dimensional measurement according to the surface state of the workpiece can be estimated, and a more accurate picking operation simulation can be performed. That is, since the angle threshold value for determining acceptance / rejection as the three-dimensional measurement data can be determined from the surface condition information of the workpiece specified by the user, there is an advantage that flexible adjustment according to the workpiece to be used can be easily performed.

  In particular, in conventional picking motion simulation, simply load a work model, perform physical simulation to generate bulk data, find the position and posture where each work model is stable, place the work model at that position and posture I was just there. For this reason, when actually measuring a workpiece in the workspace three-dimensionally, data such as an area that should not be measured, for example, occlusion occurs, or the surface of the workpiece is close to a mirror surface and data cannot be obtained. Will be. As a result, although the three-dimensional search can be performed on the picking operation simulation, the simulation result and the actual result may greatly deviate, such as a search failure in actual operation.

On the other hand, in the seventh embodiment, since the user can specify the surface condition parameter indicating the surface condition of the workpiece, it becomes easy to accurately predict whether the reflected light of the projector can be properly received by the camera by the sensor unit. The accuracy of determining whether or not three-dimensional measurement is possible at each position on the workpiece can be improved.
(Simulation Environment Setting Procedure According to Embodiment 7)

  Here, the procedure for setting the simulation environment according to the seventh embodiment will be described based on the flowchart of FIG.

  First, in step S1401, the surface state of the work model is designated. Here, the surface state parameter setting unit 12 specifies a surface state parameter corresponding to the target workpiece as the surface state of the workpiece model.

  Next, in step S1402, the CAD data of the work model with the specified surface state is stacked in a predetermined number to obtain three-dimensional packing data. In this example, the process is internally performed for each work while the height image is generated using the Z buffer method. After writing all the data to one buffer, the height image is completed.

  In step S1403, a height image is generated from the bulk data. Here, a method for generating a height image will be described. First, a two-dimensional height image is drawn by rendering that removes shadows based on three-dimensional bulk data composed of a work model which is CAD data of a work. As a rendering method, a Z buffer method, a Z sort method, a scan line method, a ray tracing method, or the like can be used. Here, a plurality of work models are drawn on a height image using the Z buffer method. In the Z buffer method, it is realized as Z buffer = height image. In addition, the Z buffer is not initialized at infinity, but is initialized to the minimum with information on the floor and the box. In the process of the Z-buffer method, a normal image that holds normal information of the surface of the work model is also created for each pixel at the same time. Here, the normal line data is also updated for the pixel in which the Z buffer is overwritten. In the present specification, the Z buffer method is exemplified as a conversion method to a height image, but the present invention is not limited to the Z buffer method, and other methods can be appropriately used as described above.

  In this manner, the height image is redrawn from the three-dimensional bulk data. Next, in step S1404, it is determined whether all work models have been drawn. If not, the process returns to step S1403 to repeat the process. As a result, if all the work models have been drawn, the process advances to step S1405.

In step S1405, an angle threshold for removing data is determined for each pixel based on the designated surface state. The angle threshold value is determined by the angle threshold value determination unit 13 and will be described in detail later. Then, in step S1406, those whose normal angle in the pixel exceeds the determined threshold are removed. This process is also performed by, for example, the angle threshold value determination unit 13 of the work model setting unit 11. Further, in step S1407, it is determined whether or not the process has been performed for all pixels. If not, the process returns to step S1406 to repeat the process. Then, when it is determined that the process is performed for all the pixels, the process is terminated. In this way, the environment setting for the picking operation simulation is performed.
(Surface condition parameter)

  Here, the procedure for designating the surface state of the work model in step S1401 will be described in detail. Surface condition parameters for designating the surface condition include workpiece material, surface roughness, processing method, color, reflectance, and the like. These may be used alone or in combination.

  Examples of parameter values representing the workpiece material include metals, resins, and ceramics. Examples of parameter values representing surface roughness include numerical values such as Ra, triangle symbols, names of finished surfaces (rough finish, mirror finish, etc.), and the like. Furthermore, examples of parameters representing the processing method include grinding, milling, and lapping. Furthermore, the parameters representing the color include black, white, silver, glossy, glossless, and the like.

  Further, as a method for specifying the workpiece, the same surface state may be specified for the entire workpiece, or a partially different surface state may be specified. In order to designate the surface of such a workpiece, a workpiece model virtually representing the workpiece is displayed on the display unit. Here, as an example of a partially different surface state designation method, there is a method of setting a work model displayed three-dimensionally on the display unit as shown in FIGS. 15A to 15B. For example, on the perspective view of the work model WM shown in FIG. 15A, the user directly designates the surface for which the surface state is to be set with a pointing device such as a mouse. The designated area DSA may be colored or highlighted as shown in FIG. 15B to change the display mode so that the designated area can be easily visually distinguished from other areas. preferable.

  Alternatively, as shown in FIG. 16A, in a state where the work model is displayed on the wire frame WMF, the vertex DSP that constitutes the surface for which the surface state is to be designated is individually designated or selected as shown in FIG. 16B. May be. In this case, the area defined by the selected vertex is specified as the area for designating the surface state.

Moreover, the method is not limited to the method of setting in the 3D display state, and may be a method of setting in the 2D display state. For example, a plan view of the workpiece shown in FIG. 17 viewed from the directions A and B is displayed as shown in FIG. In this manner, a development view WMD of the work model may be created and displayed on the display unit, and designation may be performed on the development view WMD.
(How to determine the angle threshold)

  Next, a method for determining the angle threshold in step S1405 of FIG. 14 from the state in which the surface state of the workpiece model is designated as described above will be described. Here, based on the position of the light source of the projector model and the expression of the reflection model, the ratio of the light at the observation point to the incident light is calculated for each work angle, and a threshold is set for this ratio. It is determined that the workpiece angle cannot be measured at a workpiece angle outside the set threshold value, and is removed from the three-dimensional measurement data. Hereinafter, a specific method will be described.

  Many models of reflection in simulation have been studied and proposed mainly in the field of computer graphics. Here, for the sake of simplicity, a specific procedure is described using a Phong reflection model. In the present invention, the reflection model is not limited to the Phong reflection model, and other reflection models may be used.

According to the Phong reflection model, the reflected light from the surface is expressed as follows.

(1) Formula In the above formula, each term represents the following.
I: reflected light; Id: diffuse reflection component; Is: specular reflection component; Ia: ambient light component

Here, since the purpose is to calculate how much light rays emitted from the light source reach the observation point, Ia = 0 is set here for simplicity. Note that when the ambient light is strong, there may be an effect on whether or not the measurement is possible. Therefore, it can be calculated as Ia ≠ 0. As a result of assuming Ia = 0, the equation (1) can be expressed as the following equation.

... (2) formula

Next, each component will be described. Here, the mathematical formula describing each reflection component assumes a case where the incident light is a parallel light beam. If the light source is in another form, a model formula suitable for the light source may be used.
[Diffusion reflection component]

First, a Lambert reflection model can be used as a diffuse reflection model. The Lambert reflection model is a property that an ideal diffuse reflection surface should have, and is a reflection model in which luminance is constant when viewed from any direction. This is shown in FIG. 19A. Here, assuming that the luminance of the incident light is I, the incident angle is α, the ratio of the diffuse reflection component is kd, and the reflectance of the diffuse reflection light is rd, the luminance Id of the diffuse reflection light is expressed by the following equation.

In the above equation, α represents the angle between the workpiece normal and the incident light. In addition, although it demonstrates using a Lambert reflection model here, in this invention, it is not limited to a Lambert reflection model, You may use another diffuse reflection model.
[Specular reflection component]

  Next, examples of the specular reflection model include a Phong specular reflection model, a Blinn-Phong specular reflection model, a Torrance-Sparrow specular reflection model, and a Blinn specular reflection model. In this specification, the Phong specular reflection model will be described. However, the present invention is not limited to this, and other specular reflection models including those described above can also be used.

FIG. 19B shows a schematic diagram of the Phong specular reflection model. In this figure, α indicates the normal to the workpiece surface and the angle of the incident light, β indicates the specular reflection direction and the angle of the observation point, and n indicates a parameter indicating the degree of light spread. As shown in FIG. 19B, in the case of specular reflection, reflection is performed with a slight spread around the regular reflection direction. In addition, when the observation point deviates from the regular reflection direction, the luminance decreases. Here, assuming that the luminance of the incident light is I, the angle between the specular reflection direction and the observation point is β, the ratio of the diffuse reflection component is ks, and the reflectance of the specular reflection light is rs, the luminance Is of the diffuse reflection light is expressed by the following equation. Is done.

N in the above equation is a coefficient representing the degree of specular reflection. The larger n, the smaller the reflection. Here, calculation is performed assuming that not only the ambient light but also the self-light emission of the object and the light absorption by the object are absent. The brightness Ir of the reflected light is expressed by the following equation.

  Regarding how to set each coefficient in the above equation, a value measured by a measuring instrument using an integrating sphere or the like may be set in advance. Alternatively, it may be simply obtained by applying illumination from a plurality of directions, or may be input by the user. Here, each coefficient is expressed as kd = 0.5 ks; ks = 0.5; rd = 0.7; rs = 0.98; incident angle (α) = 25 °; degree of specular reflection spread (n) = 200 The observation point angle = 15 °. The distribution of the reflected light with this coefficient is as shown in FIG.

Thus, the intensity of light changes depending on the observed direction. The fact that the intensity of the observed light is small means that the three-dimensional measurement cannot be performed because the difference from the portion not exposed to light is small. Here, considering that the workpiece is irradiated with structured light (structured light) from the projector, the light hits the observation point depending on how much the reflected light from the part hit by the light reaches the observation point. Since it is determined whether the contrast with the non-existing part can be obtained, whether or not the measurement can be performed is determined by the degree. For example, consider the following embodiment.
Example 1

  Diffuse reflection: Specular reflection = 9: 1; Work tilt = 40 °; Incident light intensity = I; kd = 0.9I; ks = 0.1I; rd = 0.70; rs = 0.98; α = 40 °; β = 50 °; n = 100

In this case, the luminance when light hitting a certain point is observed from the observation point is expressed by the following equation.

(Example 2)

  Diffuse reflection: specular reflection = 1: 9; workpiece tilt = 40 °; incident light intensity = I; kd = 0.1I; ks = 0.9I; rd = 0.70; rs = 0.98; = 40 °; β = 50 °; n = 100

In this case, the luminance when light hitting a certain point is observed from the observation point is expressed by the following equation.

  Thus, in the incident light, the observation point direction, and the regular reflection direction as shown in FIG. 21A, even if the incident direction, the observation direction, and the workpiece angle are the same, if the ratio of diffuse reflection and specular reflection is different, The observed brightness will vary greatly. Here, a situation as shown in FIG. 21B is considered. Here, assuming that the intensity of incident light = I; kd = 0.1I; ks = 0.9I; rd = 0.70; rs = 0.98; n = 100, the workpiece shown in FIG. The ratio of incident light to reflected light is as shown in the graph of FIG.

  The low ratio of incident light and reflected light means that even if light strikes a certain part of the work, it does not return to the observation point, so it cannot be distinguished from the part that is not illuminated. means. That is, this ratio is considered to be close to the contrast value obtained when the work surface is observed. Therefore, a threshold value is provided for this ratio, and the angle threshold value is used when calculating the measurement result. For example, when the threshold value of the ratio is 0.4, a workpiece having a work angle of 20 ° can be measured if the diffuse reflection ratio is larger than 0.6.

  Further, the threshold value may be a single fixed value as described above, may be a function, or may be specified by the user.

  When the specular reflection component is large, if the observation point is just in the regular reflection direction, a considerably strong light comes in the direction of the observation point (for example, when the work angle is 10 ° in FIG. 21B). The pixel may become saturated and cannot be measured. In order to determine such a location, a threshold value may be provided on the upper side.

  Schematic diagrams when measurement is performed using the threshold value thus determined are shown in FIGS. 23A to 23D. In these drawings, FIG. 23A is an image diagram showing an example of a two-dimensional display of a height image having a high diffuse reflection ratio, FIG. 23B is a perspective view of FIG. 23A three-dimensionally displayed, and FIG. 23C is a high mirror reflection ratio high. FIG. 23D is a perspective view of FIG. 23C, respectively.

If this method is used, a more realistic simulation can be performed. In addition, since the setting method for this can be set with keywords such as “material” and “surface roughness” that are easy for the user to understand, the setting can be performed more easily.
(Embodiment 8)

  In the above example, the example in which the bulk data generation unit 20 and the region estimation unit 22 are provided separately from the picking motion simulation unit 30 has been described. However, the present invention is not limited to this configuration, and the picking motion simulation unit may be provided with a function for generating bulk data and bulk images. Such an example is shown in the functional block diagram of FIG. 24 as a simulation apparatus 400 according to the eighth embodiment. The robot simulation apparatus 400 shown in this figure includes a picking motion simulation unit 30B, a simulation environment setting unit 10, a detection setting unit 50, and a bulk data storage unit 23. Also in the eighth embodiment, the same members as those in the first embodiment described above are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate.

The picking motion simulation unit 30B includes a stacked image generation unit 31, a position / orientation detection unit 32, and a gripping solution calculation unit 33. The stacked image generation unit 31 generates a stacked image as a two-dimensional height image indicating a stacked state from three-dimensional stacked data in which a plurality of work models are stacked. This bulk image can be a plan view viewed from the sensor unit. The position and orientation detection unit 32 is a member for detecting the position and orientation of each work model included in the stacked image. The gripping solution calculation unit 33 determines the interference between the work models based on the position and orientation of each work model, and when the gripper can be gripped by an end effector model simulating an end effector, an end for gripping the work model The posture of the effector model is calculated.
(Picking motion simulation procedure)

  Next, the procedure of the picking operation simulation will be described based on the flowchart of FIG. First, in step S2501, a simulation environment is set. For example, the user makes settings for generating a stacked image from the simulation environment setting unit 10 in FIG.

  Next, in step S2502, a stacked image of work models is generated. Here, a physical simulation is executed in order to virtually reproduce the state in which the workpieces are stacked in the work space. The physical simulation repeats the operation of dropping the work models one by one in the virtual work space, and constitutes loose stacking data in which a plurality of work models are randomly stacked (details will be described later).

  In this way, when the stacked images are obtained in step S2502, it is next determined in step S2503 whether to output position / orientation information indicating the position and orientation of the work model. When outputting, it progresses to step S2504, and after outputting the position and orientation information of the work model, it proceeds to step S2505. On the other hand, if position / orientation information is not output, the process advances to step S2505 without passing through step S2504.

  In step S2505, an attempt is made to detect the position and orientation of each work model. Further, in step S2506, it is determined whether or not the position and orientation of each work model can be detected. If so, the process proceeds to step S2507.

  On the other hand, if it cannot be detected, the process jumps to step S2511 to extract information on the remaining work model. The information on the remaining work model includes the number of work models that cannot be picked as a result of incorrect recognition, and the position and orientation information for the work model for which position and orientation information can be detected.

  Subsequently, in step S2512, it is determined whether or not to output the position and orientation information of the extracted work model. In the case of outputting, the process proceeds to step S2513, where the position / orientation information of the work model is output and the process is terminated.

  If the position and orientation of the workpiece model can be detected in step S2506, in step S2507, an attempt is made to determine the interference between the end effector model and the surrounding data and to calculate the gripping solution. In step S2508, it is determined whether or not the gripping solution can be calculated. If it can be calculated, the process proceeds to step S2509. If it cannot be calculated, the process jumps to step S2511 to obtain the position and orientation information regarding the remaining work model as described above. After extracting and determining the output in step S2512, the process ends.

  If the gripping solution can be calculated in step S2508, the process proceeds to step S2509, and the number of work models constituting the stacking data is reduced by one. In step S2510, it is determined whether or not an unprocessed work model remains. If it remains, the process returns to step S2502 to repeat the above processing from the generation of the stacked images, and if not, the processing ends. To do.

In this manner, the picking operation simulation is executed by the picking operation simulation unit. The executed result can be displayed on the display unit as necessary. Based on this, the user can determine whether the current setting is appropriate. Also, various settings are changed as necessary. For example, the installation position of the robot body, the position where the storage container is placed, the attitude of the sensor unit, and the like are adjusted. By inputting the changed settings to the simulation environment setting unit, executing the picking operation simulation again, and confirming the suitability, it is possible to determine appropriate setting conditions according to the user's environment.
(Procedure for creating bulk images)

  Next, the details of the procedure for generating a stacked image of workpiece models in step S2502 of the flowchart of FIG. 25 will be described with reference to the flowchart of FIG.

  First, in step S2601, it is determined whether to read initial state data. This determination prompts the user to determine whether to read the initial state data. It is assumed that the initial state data is stored in advance in the bulk data storage unit 23 or the like.

  When the initial state data is read, the process proceeds to step S2602, and the work model is set to the position and posture recorded in the stored initial state data. Next, in step S2603, a physical simulation is executed. For example, the work model is dropped from a predetermined height (details will be described later). In step S2604, the simulation parameter determines whether or not the number of work models specified by the simulation environment setting unit has been arranged. If not yet, the process returns to step S2602 to repeat the process. If it is determined that the designated number of work models have been dropped, the process advances to step S2608 to generate a stacked image based on the position and orientation of the work model, and the process ends.

  On the other hand, if it is determined in step S2601 that the initial state data is not read, the process advances to step S2605 to set the initial state of the work model. The initial state includes the start position and orientation at which the physical simulation is performed on the work model. For example, when the user wants to stack the work model at a desired position, such as when the work model is to be placed at a position near the wall of the storage container, such position and posture are determined. Alternatively, the position and orientation of the work model are determined based on random numbers.

In step S2606, a physical simulation is executed on the work model. Further, in step S2607, it is determined whether or not the designated number of work models have been arranged. If not yet, the process returns to step S2605 to repeat the processing. If it is determined that the designated number of work models have been dropped, the process proceeds to step S2608 in the same manner as described above, and a stacked image based on the position and orientation of the work model is generated, and the process ends. In this way, a stacked image is generated.
(Physical simulation)

Note that it is not always necessary to drop the work model when performing the physical simulation. For example, the physical simulation may be started from a state in which a work model is arranged on the floor surface, or another work model is arranged in contact with an existing work model. That is, the operation of dropping the work model is not always necessary in the physical simulation. However, when the work is actually placed on the floor or other work stacked in bulk, the stress or resistance of the actual work, such as gravity or friction, such as falling, tilting or sliding Consider the possibility of moving in response. In addition, not only immediately after placing the workpiece, but also by picking another workpiece after that, the balance of the stacked workpiece will be lost and the movement of the workpiece model will be taken into consideration, and the virtual movement of the workpiece model will be considered. Simulate. In other words, the work or work model referred to in the present specification is not limited to dropping the work etc., but the placed work etc. is tilted, collapsed, slipped, or caused by friction. It is used in the sense that it moves from a stable state to an unstable state due to stress or resistance acting on the natural world, such as a non-slip state or a change in the amount of movement depending on the coefficient of restitution of the workpiece.
(Reproduction of initial state of simulation)

  Here, an example in which a cylindrical work model WMC as shown in FIG. 27 is stacked and a picking operation simulation is performed will be described. Here, as a result of performing a picking operation simulation on the initial state of the unloading data obtained by unloading the work model WMC illustrated in FIG. 28A and performing a process of taking out the work models one by one, in the state illustrated in FIG. Consider the case where the picking motion simulation is completed.

  Here, when the initial state of the bulk data is random, the same state as the state obtained in the previous physical simulation cannot be created. That is, since random bulk data is generated every physical simulation, the initial state shown in FIG. 28A is transient and has no reproducibility.

  On the other hand, when the user reviews the picking operation parameter settings and wants to confirm the effect, it reproduces the state that did not work in the previous picking operation simulation and confirms this bulk data. Is the most efficient. For example, when the gripping solution of the end effector model for a certain work model is not found and all the work models cannot be extracted, a case where the user increases the grip teaching position as a countermeasure is considered. In this case, in order to determine whether or not the settings changed by the user were effective for the problem that occurred last time, it is best to reproduce and check the state that did not go well last time or a state close to this. It is valid.

  In addition, it may be possible to start not only from the state in which the problem has occurred but also from the state of the process in which the problem has occurred. For example, if bulk data with a problem is configured by adding work models sequentially, whether the work model can be handled several times before the stage where the problem occurs and the number of work models is few It is possible to confirm.

  In order to realize these, in the present embodiment, data in a state that the user wants to reuse, such as an initial state, an end state, or an intermediate state of bulk data, is stored, and after adjusting the picking operation parameter The stored bulk data can be called up and verified.

  For example, by saving the end state of FIG. 28D in the bulk data storage unit 23, it is estimated that the user has failed in the picking operation, and the simulation parameter setting unit resets the expected simulation parameter. Then, you can run the picking motion simulation again and check if picking is possible. Alternatively, in order to verify the picking operation from the previous stage as well as the state where the picking operation simulation has failed, it is also possible to store the unpacking data in the middle of picking in the unloading data storage unit 23. it can. For example, FIG. 28C shows the stacking data in a state that has been traced ten times from the end state of FIG. 28D, and from this stage, the picking operation in which the simulation parameters are reset can be verified. Alternatively, as shown in FIG. 28B, tracing back to older data, it is also possible to save the bulk data that has been traced 20 times from the end state of FIG. 28D, read it, and execute the picking operation simulation again.

  Further, the bulk data may be stored at a predetermined timing as described above, or may be automatically stored at a predetermined timing. As the predetermined timing, for example, the initial state before execution of the picking operation simulation, the final state after completion of the picking operation simulation, the state where there is no work model that can be picked any more, or every time the work model is taken out in the middle You may make it save to.

  Further, as another predetermined timing, for example, the number of workpieces to be piled up is every fixed number such as every 5 pieces or every 10 pieces, or every 5 minutes every 10 minutes in the elapsed time for creating loosely piled data. And so on. Further, there may be a limit on the capacity and number of bulk data to be automatically saved. For example, if the automatically stored bulk data exceeds 500, the old data is overwritten, or if the sum of the automatically stored data exceeds 500 MB, the old data is overwritten and deleted in order. Rules such as can be set.

  Furthermore, as a user interface screen for reading these automatically or manually stored data retroactively, for example, general buttons are arranged in a VTR such as fast forward and rewind, a jog shuttle, a jog dial, etc. An operation unit that can smoothly switch between the feeding direction and the returning direction with a rotary member can be prepared. Or you may comprise so that the past position may be designated with a linear member like a scroll bar or a slider.

Also, instead of reading such stored bulk data, new bulk data can be randomly generated. For example, during the picking operation simulation, the user may be configured to select whether to use past unloading data or new unloading data.
(Data storage in the process of physical simulation)

  In general, physical simulation is used to perform simulation with bulk data. In the physical simulation, the motion of an object such as a workpiece having a mass is dynamically simulated in a virtual work space in which gravity is set. When each object collides, it is shown how the object moves after the collision in consideration of a repulsion coefficient and a friction coefficient set in advance. Such a physical simulation procedure will be described below with reference to the flowcharts of FIGS. 29A and 29B.

  First, in step S2901, a virtual space is created. In step S2902, the created virtual space is set. Here, parameters related to the entire physical simulation, such as gravity in the virtual space, are set. Note that creation and setting of the virtual space may be performed simultaneously.

  In step S2903, a storage container for stacking work models in bulk is set. For example, the length, width, height length, etc. of the storage container are set. Next, in step S2904, the arrangement of storage containers for stacking work models in bulk is set. In a physical simulation that generates bulk data, the storage container is often arranged as a static object.

  In step S2905, the size of the work model is set. For example, settings specific to the work model such as the size, shape, density, and weight of the work model are performed. In step S2906, the work model position information is read.

  In step S2907, it is determined whether or not the initial position of the work model is designated. If not specified, in step S2908, the initial position and orientation of the work model are set randomly, and the flow advances to step S2910. On the other hand, if the initial position of the work model is set, in step S2909, the position and posture of the work model are set to the designated position and posture, and the process proceeds to step S2910. The initial position of the work model may be set by reading a setting file such as text data, or may be specified by the user using a GUI or the like.

  In step S2910, a work model is generated. In step S2911, it is determined whether or not a specified number of work models have been generated. If not, the process returns to step S2906 to repeat the above steps. On the other hand, if the designated number of work models has been generated, the process advances to step S2912 to simulate the position / posture of the work model after unit time. Here, not only the work model but also the position and orientation of the work model after a unit time are calculated for all objects in consideration of collisions with other work models. In step S2913, it is determined whether all work models are stable in their positions and postures. If not stable, the process returns to step S2912, and the above process is repeated. On the other hand, if it is determined that the position is stable, the process advances to step S2914 to determine whether or not to save the position / posture information of the work model. The user determines whether or not to save. The user saves the data when he / she feels the necessity to save the data, for example, when there is a possibility that the setting will be reviewed or the simulation may be performed again later. In the case of saving, the process proceeds to step S2915, and after saving and outputting the position / posture information of the work model, the process ends. Here, the user may specify whether or not to output each time. Alternatively, logging may always be performed inside the physical simulation unit. Furthermore, the data to be stored may be designated by the number of times. For example, an arbitrary timing or number can be designated such as every N times or 10 times. On the other hand, if it is determined that the storage is unnecessary, the processing is terminated as it is.

In this way, a physical simulation is performed in consideration of the position and orientation of the work model. According to this method, it is possible to hold a state where the user feels the necessity of reviewing the setting in step S2914 and call it later. As a result, it becomes possible to clearly determine whether the review of the settings made by the user has been effective for the problem that the user is facing. For this reason, when the user starts up the robot, correct settings can be made in a shorter time, thereby shortening the startup time and stabilizing the actual operation.
(Restriction of work model posture)

  Further, in the generation of three-dimensional bulk stacking data and bulk stacking images, a restriction may be imposed on the posture of the work model placed on the virtual work space. That is, in the conventional bulk picking simulation, the work model is randomly placed on the virtual work space, resulting in a simulation in a state different from the actual operating environment. For example, as shown in FIG. 30, the T-shaped work model WMT is present in an unstable posture such that it stands upright. In actual operation, it is unlikely that a workpiece exists in such an unstable posture.

  In addition, the manner in which the workpieces to be picked are actually provided is not limited to the randomly stacked state, and there are many cases in which the workpieces are stacked in a stacked state or in a state in which the orientation of the workpieces is fixed, Appropriate setting adjustments and verification according to the work status during actual operation were not possible.

  Therefore, in this embodiment, simulation results that match the environment and conditions during actual operation can be obtained by preparing unloading data with certain restrictions added, instead of making the workpiece posture completely random. I am trying to do it. In other words, individual settings for generating bulk data can be made by customizing the generation of bulk data according to the use of the user, the shape of the workpiece, and the like.

  Such restriction of the posture of the workpiece can be performed when the physical simulation is executed. As an example, (I) when performing a physical simulation, a work model is not dropped in a random posture on a virtual work space, but dropped in a state in which posture restriction is added. In addition, it is possible to perform posture restriction by removing a posture that exceeds the posture restriction in the physical simulation process.

  Alternatively, (II) when performing a physical simulation, instead of dropping work models one by one in a random posture on the virtual work space, in a state where a plurality of them are arranged at regular intervals in a prescribed posture, It may be dropped at the same time.

In this way, since bulk data matching the state at the time of actual operation can be generated, the result of the picking operation simulation can also be executed in a state close to that at the time of actual operation, and a more accurate simulation result can be obtained. As a result, it is possible to perform appropriate setting adjustment and verification in advance in a form close to actual operation without actually preparing a work. Hereinafter, in step S2502 of FIG. 25, a procedure for generating the stacking state of the workpiece model by the stacking data generation unit 20 after limiting the posture by the posture condition setting unit 16 will be described.
(I: Posture restriction in physics simulation that drops work models one by one)

First, a description will be given of posture restriction when dropping work models one by one in a physical simulation. Here, a method for restricting only the posture of the work model will be described. Here, the position of the work model is random with no restrictions. As a method of adding posture restriction,
1. Posture restriction is added when dropping a work model in physical simulation.
2. After the physics simulation is executed, work models outside the posture limit range are excluded.
3. Above 1. 2. Do both.
There are three patterns.
(1. Posture restriction when dropping the work model)

  First, 1. A procedure for adding posture restriction when dropping a work model in the physical simulation will be described with reference to the flowchart of FIG. Here, it is assumed that the posture allowed for the work model is set in advance by the posture condition setting unit 16 as the posture condition. For example, the angle range of the posture allowed for the work model is defined as the posture limit range. Alternatively, the allowable rotation angle and tilt angle from the reference posture of the work model can be set as the posture condition.

  With the posture conditions set in this way, first, in step S3101, the posture and drop position of the work model are generated based on random numbers. Next, in step S3102, it is determined whether the posture of the generated work model is a posture within a posture restriction range set by the posture condition setting unit. This determination is performed by the bulk data generation unit. If it is determined that the position is not within the limit range, the process returns to step S3101 to regenerate a new work model position and drop position. Note that the workpiece model to be dropped by generating the posture and the dropping position can be one by one, or the posture and the dropping position can be generated collectively for a plurality of workpieces and dropped simultaneously. Good.

On the other hand, if it is determined that the posture limit range, the process proceeds to step S3103, and a physical simulation for dropping the work model is performed. In step S3104, it is determined whether or not the specified number of work models have been dropped. If not yet, the process returns to step S3101 to repeat the process. In this way, when it is determined that the designated number of work models have been dropped, the process is terminated. As described above, it is possible to exclude the presence of an unnatural work model by determining whether the posture before dropping the work model is within the posture limit range and excluding the work models outside the limit range. With this method, since the physical simulation to drop itself is not performed for the work model outside the limit range, there is an advantage that the processing related to the simulation can be simplified.
(2. Posture restriction after dropping the work model)

  On the other hand, an unnatural work model can also be eliminated by determining whether or not the posture after dropping the work model is within the limit range. Hereinafter, 2. After executing the physical simulation, the procedure for generating the stacking data by excluding the workpiece model outside the posture limit range will be described based on the flowchart of FIG.

  First, in step S3201, a work model posture and a drop position are generated based on random numbers. Next, in step S3202, a physical simulation for dropping the work model is performed. Again, the workpiece model to be dropped by generating the posture and drop position can be one by one, or the posture and drop position can be generated collectively for a plurality of workpieces and dropped simultaneously. Also good.

  In step S3203, it is determined whether there is a work model outside the posture restriction range set by the posture condition setting unit among the dropped work models. If it is determined that there is a workpiece outside the posture limit range, the process proceeds to step S3204, and a physical simulation of dropping the workpiece model outside the posture limit range and dropping again is performed, and then the process returns to step S3203 to make a determination. repeat.

On the other hand, if it is determined that there is no workpiece model outside the posture limit range, the process proceeds to step S3205 to determine whether or not the specified number of workpiece models have been dropped. If not yet, the process returns to step S3201 to perform processing. repeat. In this way, when it is determined that the designated number of work models have been dropped, the process is terminated. As described above, it is possible to eliminate the presence of an unnatural work model by determining whether the posture after dropping the work model is within the posture restriction range and excluding the work model outside the restriction range. . With this method, since the posture is determined with respect to the dropped work model, there is an advantage that an unnatural work model can be more reliably excluded.
(3. Posture restriction before and after dropping the work model)

  Furthermore, an unnatural work model can be eliminated by determining whether or not the posture before dropping the workpiece model and the posture after transmission are within the limit range. Here, the posture after dropping the dropping posture allowed for the work model is set in advance by the posture condition setting unit. These can be set to the same posture, but preferably the posture of the workpiece model at the time of dropping and the posture of the workpiece model after dropping are preferably set individually. 3. A procedure for generating the stacking data by excluding the work model outside the posture limit range before and after the execution of the physical simulation will be described based on the flowchart of FIG.

  First, in step S3301, a work model posture and a drop position are generated based on random numbers. Next, in step S3302, it is determined whether or not the posture of the generated work model is a posture within the posture restriction range set by the posture condition setting unit. If it is determined that the position is not within the posture limit range, the process returns to step S3301 to regenerate a new posture and drop position of the work model. In this case as well, the work model to be dropped by generating the posture and the dropping position can be one by one, or a plurality of workpieces can be generated at once to generate the posture and dropping position, and these can be dropped simultaneously. May be.

  On the other hand, if it is determined that the posture limit range, the process proceeds to step S3303, and a physical simulation for dropping the work model is performed. In step S3304, it is determined whether there is a workpiece model outside the posture restriction range set by the posture condition setting unit among the dropped workpiece models. If it is determined that there is a workpiece outside the posture limit range, the process proceeds to step S3305, a physical model that excludes the workpiece model outside the posture limit range, drops again, and returns to step S3304 for determination. repeat.

  On the other hand, if it is determined that there is no work model outside the posture limit range, the process proceeds to step S3306, where it is determined whether or not the specified number of work models have been dropped. If not yet, the process returns to step S3301 to perform processing. repeat. In this way, when it is determined that the designated number of work models have been dropped, the process is terminated. As described above, it is possible to eliminate the presence of an unnatural work model by determining whether the posture after dropping the work model is within the posture restriction range and excluding the work model outside the restriction range. . With this method, since the posture of the work model is confirmed before and after the fall, the unnatural work model can be most reliably excluded.

In this way, there are workpieces that stand upright in an unnatural manner by adding restrictions to the posture of the workpiece when the workpiece model is dropped, after it is dropped, or at both timings. It is possible to avoid a situation where an operation simulation is executed. In addition, it is possible to realize a stacked state in which only one side is visible depending on the shape of the workpiece. For example, as shown in FIGS. 34A and 34B, for a work WK6 having different front and back areas, only the front side of each work WK6 can be seen in a stacked state as shown in FIG. However, it is possible to generate such a stacked state according to actual operation and virtually reproduce it. For example, the allowable posture condition is set by the posture condition setting unit so that a specific surface of the work model is a bottom surface. As a result, time-consuming work such as setting adjustment and verification in the picking operation simulation can be performed in advance in a form close to actual operation without actually preparing a work. In addition, where the optimal setting changes depending on how the target workpieces are stacked, it is possible to match the setting before executing the picking motion simulation by limiting the posture condition .
(Embodiment 9)

  In the above method, the example in which only the posture of the work model is limited when generating the bulk data has been described. However, the present invention is not limited to this example, and both the posture and position of the work model can be limited. For example, as shown in FIG. 36A, a state in which plate-like workpieces WK7 are aligned and packed in the storage container, or a state in which cylindrical workpieces WK8 are stacked in the storage container as shown in FIG. 36B. Think. The workpieces in such a stacked state are not randomly present in the storage container but are subjected to a restriction that they are in a substantially constant posture or position. For this reason, such a stacked state cannot be reproduced by a method in which workpieces are generated completely at random, such as a conventional physical simulation. Therefore, in order to execute an accurate picking operation simulation for a workpiece in such a stacked state, restrictions are provided not only on the posture of the workpiece but also on the position as a condition for generating the stacked state. Hereinafter, such a method will be described as Embodiment 9 based on the flowchart of FIG.

  First, in step S3701, drop positions and postures of a plurality of work models are set at a prescribed posture and a prescribed interval. Here, arrangement conditions such as the posture, position, and interval allowed for the work model are set by the posture condition setting unit. For example, the posture of the work model, the number and arrangement intervals in the X direction, the number and arrangement intervals in the Y direction, the number of work model stages, and the like can be given. For example, in order to physically simulate the stacked state of the workpieces WK7 in FIG. 36A, as the arrangement conditions, the number of arrangements in the X direction is set to 8, the number of arrangements in the Y direction is set to 1, and the number of stages is set to 1. Further, for example, in order to physically simulate the stacked state of the workpieces WK8 in FIG. 36B, as the arrangement conditions, the number of arrangement in the X direction is set to 5, the number of arrangement in the Y direction is set to 3, and the number of stages is set to 3.

  Next, in step S3702, disturbance is applied to the positions and postures of the plurality of work models based on the random numbers. In step S3703, a physical simulation is performed by dropping the work model at the same time. In this way, a physical simulation can be performed by dropping the work model at the same time in a state where the work model is prepared at a specified arrangement position in a specified posture. In particular, before dropping the work model, by applying a disturbance to the position and posture based on random numbers, data in a different state can be created each time.

In step S3704, it is determined whether or not the work model has been dropped for the designated number of steps. If not yet, the process returns to step S3701 to repeat the process. If it is determined that all work models have been dropped, the process is terminated. In this way, it is possible to create simulation stacking data that reproduces a state in which the workpieces are aligned and packed in the storage container or a stacking state in a stacked state. In particular, it is possible to limit both the position and the posture of the work model by dropping the work model at the same time in a state where a plurality of the postures are arranged at regular intervals in a prescribed posture. As a result, time-consuming operations such as setting adjustment and verification can be performed in advance in a form close to actual operation without actually preparing a work. In particular, the optimum setting changes depending on how the target workpieces are stacked, and the setting can be customized and adjusted in advance.
(Attitude condition setting screen)

Next, an example of a procedure for setting such a posture condition will be described with reference to FIGS. In these drawings, FIG. 38 is an image diagram showing a state in which the basic posture is defined for the work model WM6 shown on the display unit on the user interface screen of the robot simulation program which is one mode of the posture condition setting unit. 39 shows a posture condition setting screen 110 for setting posture conditions in the same manner.
(Basic posture setting)

  First, the basic attitude when dropping the work model is defined. For example, the posture of the work model WM6 with respect to the dropping direction is adjusted on the basic posture setting screen shown in FIG. In this example, the Z-axis direction is the dropping direction. For example, the drop posture can be freely changed by dragging the work model WM6 on the screen.

  Next, an allowable angle for the basic posture state is designated. The angle designation method designates two angles, a tilt angle with respect to the axis in the drop direction and a rotation angle with respect to the axis in the drop direction. For example, the tilt angle and the rotation angle are input numerically on the posture condition setting screen 110 shown in FIG. In this example, the range of the rotation angle is indicated by ±, but the lower and upper limit values may be individually specified. For example, when the inclination angle is specified as 0 ° and the rotation angle ° is specified as ± 180 °, as shown in FIG. 40, the basic posture is dropped in a posture with no inclination in the dropping direction. Note that the rotation angle with respect to the axis in the dropping direction is randomly changed. If the tilt angle is specified as 15 ° and the rotation angle as ± 0 °, as shown in FIG. 41, there is no change in the rotation direction with respect to the axis in the dropping direction with respect to the basic posture, but a disturbance is added only to the tilt angle. The work model will be dropped with the correct posture. If neither the tilt angle range nor the rotation angle range is zero, the workpiece model is dropped in a posture in which the tilt angle and the rotation angle are randomly changed within the respective ranges.

In this way, the basic posture and the range of the tilt angle and the rotation angle with respect to the basic posture are set as one set, and a plurality of sets are set as necessary. For example, only one set can be set for a loosely stacked state where only the workpiece surface is visible. On the other hand, for the bulk stacking where the side of the workpiece does not face up, but both the front and back sides are facing up, the “set in the basic posture with the front side facing up” 2 sets of "set in the basic posture with the face up" is defined, and each set is randomly chosen to select which set is selected, realizing the desired stacking state it can.
(Work model placement condition setting)

Next, a method for setting the work model arrangement conditions described as the ninth embodiment will be described below. As a method for setting not only the posture of the work model but also the position and the like, for example, 1. a method of inputting the number of arrangements and arrangement intervals in the X direction, the number of arrangements and arrangement intervals in the Y direction, and the number of stages of the work model; Examples include a method of first selecting an arrangement pattern and inputting a necessary arrangement condition accordingly. First, 1. The method will be described. As another mode of the posture condition setting unit, for example, from the arrangement condition setting screen 120 as shown in FIG. 42, the number of arrangements and arrangement intervals of the work model, the number of arrangements and arrangement intervals of the Y direction, and the number of stages of the work model are set. , Input each.
(Placement pattern selection)

Next, 2. The method will be described. In this method, the user first selects a work model arrangement pattern, and then inputs necessary parameters according to the selection. As the work model arrangement pattern selection screen, for example, as shown in FIGS. 43A, 43B, and 43C, when the work model is completely random in the storage container, the work model is arranged in one direction. The three patterns when arranged in a plurality of stages are each displayed on the display unit. These may be displayed as a list, or a plurality of screens may be switched and displayed. When the user selects any arrangement pattern, an arrangement condition setting screen for setting detailed arrangement condition parameters is displayed. For example, when the completely random arrangement pattern in FIG. 43A is selected, only the number of loose stacks is input as the arrangement condition. When the arrangement pattern arranged in one direction in FIG. 43B is selected, the number of arrangement, the arrangement direction (X or Y), and the arrangement interval are input. Or when the arrangement pattern arranged in multiple directions and multiple stages in FIG. 43C is selected, the number of arrangements in the X direction, the arrangement interval, the number of arrangements in the Y direction, the arrangement interval, and the number of stages in the Z direction are input. . In this way, the user can make necessary settings sequentially without omission according to the guidance displayed on the display unit.
(Attitude stability parameter determination unit 17)

  As described above, by setting the posture conditions such as the allowable posture of the work model in advance by the user and eliminating the work model that does not match the conditions, the posture of the unnatural or unstable posture is not included in the stacking data. The work model can be excluded. However, the present invention may be configured to automatically detect that the workpiece model is in an unstable posture so that it can be determined whether or not the workpiece model is in a stable posture. For example, when a work model with an unstable posture is detected, such a work model is automatically excluded from the bulk data, so that the bulk data is similarly close to the actual operation environment. It can be expected to improve the accuracy of picking motion simulation. Also, in this case, a work model with poor posture stability is displayed on the display unit as a work model candidate to be excluded, and the user is prompted to eliminate it, or such a work model with such posture stability is selected. Guidelines can also be given to the user. Furthermore, even if the user does not manually set posture conditions, etc., the robot simulation device automatically eliminates unnatural and unstable posture work models and sets the appropriate conditions. It is also possible to execute a picking operation simulation, which is advantageous in terms of user convenience.

  Such calculation of posture stability can be performed by the posture stability parameter determination unit. The posture stability parameter determination unit 17 can be included in the posture condition setting unit 16, for example, as shown in FIG. For example, when the posture condition is set by the posture condition setting unit 16, a preferable numerical range can be presented to the user based on the posture stability calculated by the posture stability parameter determination unit 17. The user can set the angle of the posture range or the like as the posture condition, or fine-tune it with reference to the presented numerical value. Further, the posture stability parameter determination unit is not necessarily included in the posture condition setting unit, and may be a separate member.

Furthermore, a posture stability parameter may be defined in advance on the robot simulation apparatus side, and a work model with low posture stability may be automatically excluded according to a predetermined posture stability parameter. In this case, it is not necessary for the user to set posture conditions including posture stability parameters. Thus, the aspect in which the user does not set the posture condition is also included in the present invention. In other words, the posture condition setting unit 16 is not necessarily configured to accept the setting of the posture condition by the user, and the setting of the posture condition by the posture condition setting unit 16 includes not only user designation but also setting by default value. Used in.
(Attitude stability calculation method)

  Here, a method of calculating the posture stability will be described based on the flowchart of FIG. Here, a T-shaped work model WMT shown in FIG. 30 will be described as an example. First, in step S4401, a cube circumscribing the shape of the work model WMT is obtained. For example, a circumscribed cube for the work model WMT shown in FIG. 45A is obtained as shown in FIG. 45B. Next, a surface having the maximum area among the six surfaces constituting the circumscribed cube is obtained. For example, in the example of FIG. 45B, the side surface has the maximum area as shown in FIG. 45C.

  Next, in step S4402, a physical simulation is performed on the work model WMT, and a region obtained by projecting a portion where the work model WMT is in contact with another object from directly above is obtained. For example, as a result of the physical simulation, when the work model WMT of FIG. 45A is placed in the virtual work space in a tilted position as shown in FIG. 46A, the part where the work model WMT is in contact with the floor surface of the virtual work space 46B is a projection area viewed from right above.

  As a result of the physical simulation, when the work model WMT in FIG. 45A is placed in the virtual work space in an upright posture as shown in FIG. 47A, the work model WMT is in contact with the floor surface of the virtual work space. The projected region of the part is as shown in FIG. 47B.

  Alternatively, as a result of the physical simulation, when the work model WMT of FIG. 45A is placed in a posture leaning against the wall surface of the storage container as shown in FIG. 48A, the work model WMT comes into contact with the floor surface of the virtual work space. The projected area of the existing part is as shown in FIG. 48B.

  In step S4403, a rectangle circumscribing with a minimum area is obtained for the projection region obtained in this way. For example, the minimum area circumscribed rectangle for the projection region shown in FIG. 49A is as shown in FIG. 49B. The minimum area circumscribed rectangle for the projection area shown in FIG. 50A is as shown in FIG. 50B. Further, the minimum area circumscribed rectangle for the projection region shown in FIG. 51A is as shown in FIG. 51B.

Finally, in step S4404, the ratio of the area of the circumscribed rectangle obtained in step S4403 to the area of the maximum surface obtained in step S4401 is obtained, and this is used as posture stability, and those having a ratio below a certain level are determined as unstable postures. . For example, in the case of FIG. 46B and FIG. 49B, (the area of the minimum area circumscribed rectangle) / the maximum area) = 28 / 49≈about 57%. In the case of FIGS. 47B and 50B, (the area of the minimum area circumscribed rectangle) / the maximum area) = 4 / 49≈about 8%. Further, in the case of FIGS. 48B and 51B, (the area of the minimum area circumscribed rectangle) / the maximum area) = 20 / 49≈about 41%. From this result, the upright posture work model WMT of FIG. 47A having low posture stability can be excluded as unstable. Here, the threshold value for determining the unstable posture may be a fixed value, or may be adjustable by the user as necessary.
(Embodiment 10)

  In the above example, the configuration in which the user presets the allowable posture of the work model and excludes the work model that does not match the conditions, in other words, the exclusion condition for excluding the work model from the stacking data is preset. Thus, an example has been described in which the accuracy and reproducibility of picking motion simulation is improved so that an unnatural work model is not included in the bulk loading data. Here, the user creates customized unloading data by using the posture condition setting unit to limit the posture and position of the work model. However, according to the present invention, it is not always necessary for the user to manually set the exclusion condition for excluding the work model from the bulk data, and the condition for excluding the work model may be set automatically. Such an example is shown in the block diagram of FIG. 52 as a robot simulation apparatus according to the tenth embodiment. The robot simulation apparatus shown in this figure includes a work model setting unit 11, a stacking data generation unit 20, a display unit 3, an excluded work selection unit 18, an exclusion condition calculation unit 19, and a picking motion simulation unit 30. Is provided. In the tenth embodiment, the same members as those in the first embodiment described above are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate.

  The excluded workpiece selection unit 18 is a member that allows the user to select a workpiece model that is to be excluded from the picking operation simulation among the workpiece models constituting the stacking data displayed on the display unit 3.

  The exclusion condition calculation unit is a member for automatically calculating an exclusion condition for excluding the work model for the work model to be excluded selected by the exclusion work selection unit.

This allows the user to automatically exclude the work model from the position and orientation of the work model specified to be excluded without manually setting conditions for excluding the unnatural posture work model. And a work model that meets such an exclusion condition can be excluded from the bulk data. As a result, it is possible to eliminate the trouble of setting the work removal conditions in detail on the user side. In addition, the user can select a part of the work model to be excluded, and the exclusion condition is automatically calculated, and other work models to be excluded are also automatically selected. Is planned.
(Embodiment 11)

  Further, it is possible to use both the manual setting of the exclusion condition for the user to exclude the work model and the automatic calculation of the exclusion condition on the robot simulation apparatus side. Such an example is shown in the block diagram of FIG. 53 as a robot simulation apparatus according to the eleventh embodiment. The robot simulation apparatus shown in this figure includes a work model setting unit 11, a detection setting unit 50, a bulk data generation unit 20, a display unit 3, an excluded workpiece selection unit 18, an exclusion condition calculation unit 19, A picking motion simulation unit 30. The detection setting unit 50 includes an attitude condition setting unit 16. Also in the eleventh embodiment, the same members as those in the first embodiment described above are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate. In this way, while the user manually sets the work model exclusion conditions, the robot simulation device also automatically calculates the exclusion conditions, and if necessary, corrects the exclusion conditions and proposes a preferred angle range. Thus, it is possible to support the setting operation of the exclusion condition such as the posture condition by the user.

  Note that the method of limiting the posture of the work model is not limited to the method of FIG. 38 and the like described above, and other methods can be appropriately employed. For example, when the posture of the workpiece model is expressed by a ZYX Euler angle, the angle Rx with respect to the X axis, the angle Ry with respect to the Y axis, and the angle Rz with respect to the Z axis may be limited so as to be within a prescribed range. it can. Further, the first rotation angle R1 with respect to the Z axis, the second rotation angle R2 with respect to the Y axis, and the third rotation angle R3 with respect to the Z axis are limited so as to be within a prescribed range. May be.

Further, in the aspect of stacking the workpieces and storing them in the storage container, there may be an insole as shown in FIG. 54 or a partition as shown in FIG. In such a case, it is possible to generate a stacked state in consideration of such insole and partition by executing physical simulation with data modeling the insole and partition between each stage. . These insole and partition can also be modeled by CAD data or the like. In addition, a picking operation simulation when picking these insoles and partitions with a robot can also be performed. In this case, the insole and partition are excluded by a suction-type end effector, and an opening or a slit is provided in advance on a part of the insole or partition so that it can be easily gripped by a gripping type or insertion type end effector. It can also be left.
(Changes in remaining work model group due to extraction of one work model)

  Here, the change of the work model group after taking out one work model from the work model group will be examined. Conventionally, when it is determined that one of the workpiece models stacked in bulk can be gripped by the end effector model and is picked up by the picking motion simulation, the state of the workpiece models stacked in bulk after or after picking up Was not considered. For example, in the work model group stacked in bulk shown in FIG. 56, a work model surrounded by a broken-line circle is supported by the existence of a work model indicated by hatching located below the work model. For this reason, if the hatched work model is removed, it should fall down.

  However, in the conventional physics simulation, even if the hatched work model disappears, the position and posture of other work models did not change as shown in FIG. This is probably because the physical simulation operation itself is a heavy process, and it is considered that it is sufficient to execute it once. However, like the work model surrounded by a broken circle shown in FIG. 57, it is impossible to actually stop in a posture that floats in the air without any support, and picking in such a state is impossible. Even if the motion simulation is performed, it does not match the position and orientation of the actual workpiece, so even if the picking motion simulation succeeds, there may be a problem in the reproducibility and accuracy of the picking motion simulation, such as failure during actual operation. Conceivable. Also, not only the workpieces located under the workpiece, but also workpieces that are in contact with the surroundings, if these are removed, the workpiece model in the vicinity of them will be affected and the balance will be lost. The position of the work model may change. Conventionally, the present inventors have found that the change of the remaining work group due to such work removal is not considered, and this affects the accuracy of the picking operation simulation.

Therefore, based on this knowledge, it is possible to estimate that the positions and postures of the remaining workpieces will change as a result of the workpieces being taken out. In order to realize a high picking motion simulation, the present invention has been achieved. Specifically, the physical simulation is allowed to be re-executed. As a result, like the work model surrounded by the broken-line circle shown in FIG. 58, after the work model at the position is removed, it is possible to reproduce the state in which the work model group is changed by the action of gravity. Based on this, the picking operation simulation can be executed in a state closer to actual operation.
(Scope of physical simulation re-execution)

The scope of re-execution of the physical simulation is configured so that, for example, after one work model is taken out by the picking motion simulator, it is re-executed for all of the remaining work models included in the bulk data. can do. Alternatively, after the extraction of one work model, the physical simulation is re-executed for the work models existing around the extracted work model among the remaining work models included in the bulk data. It may be configured. Alternatively, after the extraction of one work model, the work model that has been in contact with the extracted work model among the remaining work models included in the bulk data, or the robot or the like in the process of extraction. The physical simulation may be re-executed for the work model that interferes with the work model held by the robot.
(When to re-execute physics simulation)

  The timing for re-executing the physical simulation is typically the timing for taking out the work model during the picking motion simulation. That is, it is a timing to reduce the number of work models by one as in step S2509 in FIG. For example, the physical simulation can be re-executed every time one work model is extracted in the picking motion simulation. However, the present invention does not limit the timing at which the physical simulation is re-executed to the timing at which the work model is taken out. For example, the physical simulation may be re-executed at a predetermined frequency while the picking operation simulation is being executed. Alternatively, it is possible to determine whether or not to re-execute the physical simulation every time one work model is taken out. In this case, if it is determined that it is not necessary, the physical simulation is not re-executed.

Further, when one work model is taken out and there is no work model in contact with the taken out work model, it may be determined that re-execution of the physical simulation is unnecessary.
(Display user interface)

Furthermore, it is also possible to update the updated bulk data in real time on the display unit and visually indicate the state of change of the bulk data to the user. In particular, by increasing the frequency of updating the display content on the display unit, it is possible to display a lot of data, that is, how the work model group that has been piled up moves or collapses as the work model is extracted, like a movie, and more to the user It is possible to provide a display mode that is easy to understand visually.
(Procedure to reduce the number of work models by one)

  Details of the procedure for reducing the number of work models by one in step S2509 of FIG. 25 will be described with reference to the flowchart of FIG.

  First, in step S5901, the extraction parameters such as the extraction path of the work model to be extracted by the picking operation are set. The take-out parameters include, for example, which work is gripped at which position, in which direction, at what speed, and the work characteristics (for example, coefficient of restitution).

  Next, in step S5902, it is determined whether the physics simulation unit is activated. If activated, the process proceeds to step S5904 to activate the physics simulation unit. On the other hand, if it is not activated, the process proceeds to step S5903, where the physics simulation unit is activated and then proceeds to step S5904.

  In step S5904, a work model take-out operation is executed. At this stage, whether or not to re-execute the physical simulation may be determined according to the state of the work model. For example, as described above, the user is prompted from the user interface screen of the display unit to select whether or not to re-execute the physical simulation. Alternatively, when the work model to be taken out is not in contact with another work model, it is determined that the physical simulation is not re-executed.

  When the physical simulation is re-executed, for example, when the physical simulation is re-executed by the user, the state of each work model after a certain time is confirmed in step S5905. In other words, when a workpiece model is taken out and some other movement such as collapse or slipping or tilting of another workpiece model occurs, it waits until the movement ends and becomes stationary, and then stops. Display status. Such movement of the work model group may be displayed on the display unit in real time, or may be internally performed by the robot simulation apparatus as non-display. In this case, since the state after the physical simulation is generated by calculation, it is not always necessary to physically wait for a certain period of time.

  Next, in step S5906, it is determined whether or not the workpiece model to be picked up does not interfere with other workpiece models. If there is an interference, the process returns to step S5904 to repeat the process. If there is interference, the process proceeds to step S5907. Assuming that the work model to be taken out is taken out, the work model is deleted from the virtual work space.

  Further, in step S5908, the state of each work model after a certain time is confirmed, and the process proceeds to step S5909, where it is determined whether the positions and orientations of all work models have not changed since the previous confirmation. If a change has occurred, the process returns to step S5908 to repeat the process. If no change has occurred, the process ends. In this way, in the picking motion simulation, by re-executing the physical simulation at the required timing and range, it is possible to reproduce the changes that may occur each time the workpiece is picked sequentially and obtain a more accurate simulation result. Can contribute to the setting of appropriate conditions with little deviation from actual operation.

  In the above example, in the picking operation simulation, the example of performing the simulation for extracting the workpiece model has been described. However, the present invention is not limited to this example, and for example, the workpiece model to be extracted is not extracted without performing the simulation of extracting the workpiece model. One may be deleted, and the physical simulation may be started with the state after the deletion as an initial state. This method can simplify the simulation process. The physical simulation may be re-executed every predetermined number of times, not every time. Furthermore, the range of the physical simulation may be limited so that the physical simulation is performed only on the work model near the work model to be taken out. This setting can be performed by, for example, a simulation environment setting unit described later. Alternatively, the situation around the work model to be taken out may be confirmed, and the physical simulation may be executed only when there is another work model on the route from which the work model is taken out.

  In this way, it is possible to provide the user with a picking operation simulation that is closer to the actual operation. It is also possible to confirm an exhaustive function using a picking operation simulation. For example, in the picking operation, a technique is provided in which a work model around the work model taken out last time is not set as a next pick-up target. In this method, after a work model is taken out, a work model that is assumed to have no change in position or posture is moved to a picking operation without performing re-imaging. However, if the state after taking out one of the work models is not re-physically simulated, the place where it should have collapsed is not collapsed, so that the function cannot be correctly verified. Therefore, by executing the physical simulation again after taking out the work model as described above, it is possible to verify even the function whose operation changes depending on the movement of the work model.

As described above, in the picking motion simulation in which workpiece models are sequentially extracted from the bulk data, after any one of the workpiece models is taken out by the robot, the workpieces in the state after the workpiece model is taken out are stacked. The bulk data can be updated by applying a physical simulation that applies the action of gravity to the model group. As a result, when any one of the work models is taken out, the other work model that is in contact with the taken out work model becomes physically unnatural, for example, a state where it floats inside. Can be prevented.
(Generation of bulk image of work model)

Now, the generation of the stacked image of the work model obtained by the physical simulation in step S2502 in FIG. 25 will be described in detail. Based on the position / orientation data obtained by stacking a plurality of work models obtained by the physical simulation shown in FIG. 26, three-dimensional loose stacking data (3D measurement data) is generated. In step S2502, a height image is generated as a stacked image from the stacked data. A height image is an image in which a height value is stored in each pixel value constituting the image. Typically, it is an image in which the height is expressed by a luminance value. As the height image generation method, the Z buffer method can be suitably used as described above.
(Invalid pixel)

  After generating the height image in this manner, the normal image created at the same time is referred to, and pixels whose surface is inclined more than a certain level are made invalid pixels. In the height image and the normal image, each pixel has a one-to-one correspondence. Note that the invalid pixel is a pixel indicating a location where three-dimensional measurement could not be performed. For example, if the pixel value is 0, the pixel can be determined as an invalid pixel.

  For example, a portion that is not visible by both the projector and the camera constituting the sensor unit, that is, a pixel is set as an invalid pixel. In addition, pixel invalidation may be weighted according to an angle range in addition to being uniformly invalidated with a certain threshold as a reference.

  With the above configuration, it is possible to construct accurate bulk data considering the possibility of three-dimensional measurement during actual operation. If the surface of a workpiece having high specular reflection, such as a metal workpiece, is tilted more than a certain level, the projection pattern from the projector constituting the sensor unit will not be reflected, and three-dimensional measurement will not be possible. This is not taken into consideration in the conventional picking motion simulation, and the simulation is performed on the premise that the three-dimensional shape of all workpieces is measured, resulting in discrepancies and deviations from actual operation. It was. For example, as a result of the physical simulation, in the state where the bulk stacking data as shown in FIG. 60A is obtained, in the bulk stacking image that is simply a height image without considering the inclination of each surface of the work model WM, FIG. As shown in the plan view and the point cloud display image of FIG. 60C. In this stacked image, a steep inclined surface that cannot be measured three-dimensionally during actual operation can be measured.

  On the other hand, according to the present embodiment, it is possible to generate simulation data that reproduces the availability of measurement of the three-dimensional shape of the workpiece on the picking motion simulation, instead of using the stacking data as it is. As a result, from the stacking data of FIG. 60A, a steep slope as shown in the plan view of FIG. 60D and the perspective view of FIG. 60E is displayed as invalid pixels that cannot be three-dimensionally measured. Can be generated. As a result, the picking operation simulation can be brought close to that during actual operation, and a preliminary setting operation can be performed based on a more accurate simulation.

As described above, if there is CAD data, it is possible to efficiently perform verification in a state close to actual operation without preparing or stacking actual workpieces. However, it is also possible to perform a picking operation simulation with actual workpieces prepared and stacked.
(Angle threshold)

  Next, the procedure for setting the threshold value of the inclination angle of the surfaces constituting the work model will be described. Here, FIG. 61A shows an angle range in which the sensor unit 2 in FIG. 6 can be used to obtain the three-dimensional shape of the work WK4 having a weak specular reflection, and FIG. 61A shows an angle range in which the three-dimensional shape of the work WK5 having a strong specular reflection can be obtained. Are shown in FIG. 61B. As shown in these figures, the material that composes the workpiece is weak in specular reflection, such as resin, in other words, if diffuse reflection is strong, and in the case of metal, etc., the specular reflection is strong, in other words, diffuse reflection. A difference occurs in the measurable angle range from the case of weakness. That is, if the specular reflection is strong, the projection pattern of the projector is difficult to be reflected on the surface of the workpiece, and measurement is impossible if the surface is slightly inclined. In the resin work WK4 shown in FIG. 61A, measurement is possible as long as it is an inclined surface within a range of 75 ° from the vertical surface lowered from the sensor unit 2 to the work. On the other hand, the metal workpiece WK5 shown in FIG. 61B can be measured if it is an inclined surface within a range of 45 ° from the vertical surface. The angle threshold value may be expressed as an inclination angle from the vertical direction as described above, or may be expressed as an inclination angle between the camera and the projector.

Therefore, in the present embodiment, it is possible to set an angle threshold indicating how many inclined surfaces can be measured with respect to the horizontal plane. By making it possible to adjust the angle threshold according to the material of the target workpiece, an appropriate picking operation simulation according to the workpiece is realized.
(Detection setting unit 50)

  Such an angle threshold can be configured to be performed from the detection setting unit 50 of FIG. 24 described above. Based on the angle threshold setting performed by the detection setting unit 50, the region estimation unit 22 applies the measurement axis extended from the virtual three-dimensional measurement light source of the sensor model set by the sensor model setting unit 15. Thus, an area having an inclination angle equal to or greater than a predetermined angle threshold is estimated as an area where three-dimensional measurement is difficult.

  The angle threshold may be a predetermined fixed value. Alternatively, the angle threshold value may be changed according to the surface state parameter set by the surface state parameter setting unit 12. Further, the angle threshold value may be changed based on the three-dimensional measurement data actually measured by the sensor unit.

  As described above, the angle threshold value can be set according to the target workpiece. For example, assuming a resin workpiece, an example of a stacked image obtained when an angle threshold is set so that measurement is possible up to an inclined surface of 75 ° is shown in FIG. 62A, and a resin workpiece model WM4 is shown as an enlarged view of the main part. The enlarged views shown are shown in FIG. 62B, respectively. Also, assuming a metal workpiece, an example of a stacked image obtained when an angle threshold is set so that measurement can be performed up to a 45 ° inclined surface is shown in FIG. 62C, and a main part in which the metal workpiece model WM5 is expanded. Enlarged views are shown in FIG. 62D, respectively. As shown in these figures, since the metal workpiece can be measured only with respect to the resin workpiece, if the target of the picking motion simulation is a metal workpiece, this limited shape feature It is possible to verify in advance whether or not the position and posture of the workpiece can be correctly detected.

  Further, it is also possible to determine probabilistically using random numbers instead of simply binarizing whether or not measurement can be performed based on the angle threshold. Here, in actual measurement, since the surface state is not uniform, the boundary where measurement is possible / impossible is not a beautiful boundary. As a result, it becomes a form in which the ease and difficulty of becoming an invalid pixel change stochastically. For this reason, random numbers are used to reduce the probability of exclusion when the tilt angle is small, and increase the probability of exclusion when the tilt angle is large. In this way, a state closer to actual measurement can be realized by stochastically determining with a random number.

As described above, according to the robot simulation apparatus, in the picking motion simulation, it is possible to adjust and optimize the setting parameters using an image in a state close to the measurement result in actual operation. For example, as a result of the picking operation simulation, when it is found that the workpiece detection itself is difficult, it is possible to take measures such as the need to improve the workpiece transfer method and the stacking method. In this way, an appropriate countermeasure can be verified in advance without actually preparing a workpiece or installing a sensor unit.
(Elimination of difficult areas based on camera position)
Embodiment 12

  The method of estimating the difficult region by the region estimation unit is not limited to the above-described inclination angle of the surface, and can be determined based on the position of the camera of the sensor unit. Such a robot simulation method will be described below as a twelfth embodiment.

  As described above, when generating simulation data in a bulk picking simulation, there must be a three-dimensional CAD model of the workpiece, and position and orientation data of each workpiece model when the CAD model is stacked as a workpiece model. For example, by using the above-described Z buffer method or the like, it is possible to generate a virtual image as viewed from above, that is, a stacked image.

  However, in the Z buffer method, all data that can be seen from above is drawn. As a result, even if a work model is stacked around it, it is drawn as long as it can be seen from above. For example, consider the example shown in the stacked image of FIG. Here, when the sensor unit that performs the three-dimensional measurement is, for example, a TOF sensor using the TOF method, such an image is actually obtained. However, when the projection position and the imaging position are different as in the pattern projection method as shown in FIG. Therefore, as shown in FIG. 64, although it can be seen from the projector model PM, it is impossible to measure a deep part at a position that cannot be seen from the camera model CMM. In order to obtain a picking operation simulation result close to actual operation, it is desirable to remove such points.

  Therefore, in the twelfth embodiment, after creating a height image, it is determined whether or not each pixel constituting the height image is visible from the camera model, and only what is visible is left. For example, the region estimation unit 22 in FIG. 3 or the like uses, as virtual optical axes, line segments respectively connecting virtual three-dimensional measurement light sources when performing three-dimensional measurement with a sensor model and points constituting the stacked data. Define and determine whether there is another work model point on the virtual optical axis. If there is a point of another work model, it is estimated that the three-dimensional measurement is difficult, and the estimation region is defined by a set of points where such three-dimensional measurement is difficult. Accordingly, it is possible to estimate the difficulty of measurement based on the presence or absence of interference between the virtual optical axis and the work model, thereby improving the accuracy of the simulation.

  As a specific method for determining a blind spot area by such another work model, a line segment connecting this point and the camera model is defined at each point (voxel) constituting the three-dimensional bulk data. . If there is another voxel point on this line segment, it is determined that the point cannot be seen from the camera model.

As shown in FIG. 6, when there are two or more cameras in the sensor unit 2, a point visible from at least one camera model is left. For example, in FIG. 65A, since the work model WM is visible from the camera model CMM, it is treated as an effective voxel. On the other hand, as shown in FIG. 65B, when a part of the work model WM6 is hidden behind another work model WM7 and cannot be seen from the camera model CMM, it is treated as an invalid voxel. Also, as shown in FIG. 65C, when the work model WM is visible in any of the plurality of camera models CMM2 and CMM3, it is handled as an effective voxel. By doing in this way, it is possible to perform an accurate stacking simulation by excluding a part that is deep and invisible but removing it from the stacked image and obtaining an image close to the image captured during actual operation. .
(Picking Operation Simulation Procedure According to Embodiment 12)

Here, the procedure of the picking operation simulation according to the twelfth embodiment will be described based on the flowchart of FIG. Detailed description of the same operations as those in the flowchart of FIG. 25 according to the first embodiment will be omitted as appropriate. First, in step S6601, a simulation environment is set. Next, in step S6602, a stacked image of the work model is generated. The details of the procedure for generating the stacked image of the work model will be described with reference to the flowchart of FIG.
(Procedure for generating stacked images according to Embodiment 12)

First, in step S6701, the posture and drop position of the work model are set based on random numbers. Next, in step S6702, the physical simulation is performed by dropping the work model. Further, in step S6703, it is determined whether or not the designated number of work models has been dropped. If not, the process returns to step S6701 to repeat the process. If it is determined that the designated number of work models have been dropped, the process advances to step S6704 to generate a stacked image of work models based on the position and orientation of each work model. Here, a height image is generated from three-dimensional bulk data.
(Procedure for creating bulk images)

  Here, details of the procedure for generating the stacked images in step S6704 will be described based on the flowchart of FIG. In step S6801, the stacking data is converted into the position and orientation in which the CAD data is registered. In step S6802, this is redrawn on the height image by the Z buffer method. In step S6803, it is determined whether all work models have been drawn. If not, the process returns to step S6801 and the process is repeated. As a result, if all the work models have been drawn, the process advances to step S6804.

  In step S6804, a straight line connecting one point of the height image and the camera model is calculated. Here, it is assumed that the position of the camera model in the three-dimensional virtual work space is defined in advance. After that, as shown in FIG. 69, a linear expression connecting each camera model CMM at each pixel of the height image of the work model WM is calculated. For example, it is approximated by the following equation.

  ax + by + cz = d

  In step S6805, as shown in FIG. 70, it is checked whether an obstacle, here, another work model WM exists on a straight line connecting to the camera model CMM. As a specific method, the image is converted into a two-dimensional line on the height image. Each point of the height image on the line and the Z value on the line at that position are calculated, and if the point of the height image is at a higher position, it is determined that it is not visible from the camera model.

  In step S6806, points that are not visible from all camera models are removed. Further, in step S6807, it is determined whether or not the process has been performed for all pixels. If not, the process returns to step S6804 to repeat the process. Then, when it is determined that the process is performed for all the pixels, the process is terminated.

  As described above, when the stacked images are obtained in step S6602 of FIG. 66 through the procedure of FIGS. 68 and 67, the position and orientation of each work model are detected in step S6603. Further, in step S6604, it is determined whether or not the position and orientation of each work model has been detected. If it can be detected, the process proceeds to step S6605. If not, the process jumps to step S6609 to extract information on the remaining work model. After that, the process ends.

  If the position and orientation of the work model can be detected, in step S6605, the interference determination between the end effector model and the surrounding data is performed, and the gripping solution is calculated. In step S6606, it is determined whether or not the gripping solution can be calculated. If it can be calculated, the process proceeds to step S6607. If the calculation cannot be calculated, the process jumps to step S6609 to extract information on the remaining work model. Exit.

  If the gripping solution can be calculated, the number of work models is reduced by one in step S6607. In step S6608, it is determined whether or not an unprocessed work model remains. If it remains, the process returns to step S6602, and the process is repeated. If not, the process proceeds to step S6609. In step S6609, information regarding the remaining work model is extracted.

  In this manner, the picking operation simulation is executed by the picking operation simulation unit. In this way, it is possible to more accurately estimate the measurement difficulty region where the three-dimensional shape of the workpiece cannot be measured by the sensor unit using the camera.

  In the above example, the invalid pixel is determined based on whether it can be seen from the camera side. However, not only the camera but also the position of the projector is defined, and the condition is set so that it can be seen from either the projector or the camera. You can also. Furthermore, without clearly defining the position of the camera, it is possible to perform simulation by obtaining the same effect by deleting points lower than a certain point from surrounding points.

As described above, according to the robot simulation device, a simulation is performed by virtually constructing a state close to the state of picking up the stacked workpieces without actually arranging the sensor units or stacking the workpieces. This can be done in advance. In particular, by performing a simulation in consideration of a region where three-dimensional measurement cannot be performed during actual operation, the difference between the simulation and actual operation can be reduced, and a highly reliable simulation is possible.
(Angle threshold value determination part 54)
(Embodiment 13)

  In the above example, the example in which the angle threshold is set based on virtual stacking data has been described. However, the present invention is not limited to this configuration, and the angle threshold value can also be set based on the result of measuring the state in which the workpieces are actually stacked in the sensor unit. In this case, three-dimensional measurement is actually performed by the sensor unit with respect to a workpiece group in which workpieces are randomly stacked. For this reason, a sensor unit is connected to a robot simulation apparatus that is an image processing unit.

  Details of such a robot simulation apparatus will be described as Embodiment 13 based on the block diagram of FIG. The robot simulation apparatus 500 shown in this figure includes a sensor unit 2, a measurement data storage unit 52, an angle threshold value determination unit 54, a work model setting unit 11, a physical simulation unit 60, a bulk data generation unit 20, A simulation data generation unit 40 and a picking operation simulation unit 30 are provided. Also in the thirteenth embodiment, the same members as those in the first embodiment described above are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate.

  The measurement data storage unit 52 stores three-dimensional measurement data having information on an inclination angle with respect to the measurement axis of the sensor unit 2 at each position, which is obtained by three-dimensionally measuring a plurality of stacked workpieces with different postures. It is a member for doing. The measurement data storage unit 52 can use a fixed storage medium such as a hard disk, a semiconductor memory, or other portable media.

  The angle threshold value determination unit 54 determines whether or not three-dimensional measurement can be performed based on the tilt angle information of each point constituting the three-dimensional measurement data of the plurality of workpieces stored in the measurement data storage unit 52 with respect to the measurement axis of the sensor unit 2. It is a member for determining the angle threshold value used as the reference | standard which determines this. The measurement axis of the sensor unit 2 is an optical axis of measurement light emitted from a three-dimensional measurement light source for the sensor unit 2 to perform three-dimensional measurement. The measurement axis of the sensor unit 2 may be a predetermined direction such as a vertical direction.

  The area estimation unit shown in FIG. 71 includes a virtual inclination angle of each point constituting each work model in the bulk stacking data generated by the bulk stacking data generation unit 20, an angle threshold determined by the angle threshold determination unit 54, and , And a point having a virtual inclination angle larger than the angle threshold is specified as an estimation region estimated to be difficult to measure. As a result, the simulation data can be acquired after excluding data presumed to be difficult to measure on the basis of the angle threshold value determined based on the tilt angle of the workpiece actually measured by the sensor unit 2, and thus more accurate. High picking motion simulation is possible.

  Further, the robot simulation apparatus 500 according to the thirteenth embodiment may include the display unit 3. The display unit 3 displays the inclination angle of each point based on the inclination angle information of each point constituting the three-dimensional measurement data of the plurality of workpieces stored in the measurement data storage unit 52 with respect to the measurement axis of the sensor unit 2. An angle distribution histogram showing the accumulated distribution can be displayed.

  Here, a method for determining the posture of a work model capable of three-dimensional measurement based on actually measured data will be described. When determining whether or not the three-dimensional measurement of the workpieces stacked in bulk by the sensor unit is possible, the preset angle threshold is referred to as described above. However, the angle threshold varies depending on the workpiece. For example, a resin workpiece that is relatively easy to measure can be measured even if the surface is inclined. However, it cannot be measured with a glossy metal workpiece. Therefore, it is desirable to adjust the angle threshold according to the target workpiece. However, the setting is not easy, and a desired result cannot be obtained depending on the set value.

Therefore, in the thirteenth embodiment, an optimum angle threshold is determined based on three-dimensional measurement data obtained by actually measuring a workpiece three-dimensionally. Specifically, the angle threshold determination unit 54 calculates the surface inclination for each pixel of an image obtained by actually capturing a group of workpieces, and measures the entire angle distribution. Using this angular distribution, it is determined how much the workpiece can be measured even if the surface is tilted, and an optimum threshold value is determined. With this method, the angle threshold value can be determined based on actually measured three-dimensional measurement data, regardless of variations in workpiece materials and surface conditions, compatibility with the sensor used, fluctuation due to humidity, temperature, etc. Therefore, a highly reliable angle threshold can be selected.
(Angle distribution histogram)

  An angle distribution histogram indicating a distribution in which the inclination angles of the respective points constituting the workpiece three-dimensional measurement data are accumulated is performed by, for example, the angle threshold value determination unit 54. Or you may integrate an angle threshold value determination part and a measurement data storage part in a sensor part.

  As an example, FIGS. 73 and 74 show angle distribution histograms from height images obtained by actually three-dimensionally measuring a group of workpieces stacked as shown in FIG. FIG. 73 shows an angle distribution histogram of a resin work, and FIG. 74 shows an angle distribution histogram of a metal work. Since the surface of the resin workpiece is irregularly reflected, the three-dimensional shape can be measured even if the inclination between the surface and the sensor measurement axis is large to some extent. Therefore, as shown in FIG. 27, the points measured over a wide angle range are distributed. On the other hand, since a metal workpiece has a large specular reflection component, measurement becomes impossible when the inclination between the surface and the sensor measurement axis increases. Therefore, as shown in FIG. 28, the distribution is solid in a region where the inclination angle is small. Such an angle distribution histogram is displayed in, for example, an angle distribution histogram display area of the display unit 3. The angle distribution histogram display area is realized by a GUI of a robot simulation program installed in a computer which is a robot simulation device, for example.

  From the angle distribution histogram displayed in this way, it is possible to determine in which angle range the three-dimensional measurement result is obtained. The user can manually set the angle threshold by numerically specifying the angle range in which three-dimensional measurement is performed at many points, that is, pixels, from the angle distribution histogram displayed on the display unit or directly from the screen. . For example, in the examples of FIGS. 73 and 74, 60 ° for a resin work and 35 ° for a metal work are specified as an angle range including 80% or more of pixels.

  The angle threshold may be set manually by the user or automatically by the robot simulation apparatus. For example, the angle threshold determination unit automatically calculates an angle that can be three-dimensionally measured at 80% or more of the pixels as the angle threshold. In this case, the display of the angle distribution histogram is arbitrary, and may be calculated internally without being displayed. In other words, it is not essential to display in a histogram form if information on the angular distribution is obtained. Further, the automatically calculated angle threshold value may be displayed on the display unit. At this time, the range satisfying the angle threshold may be highlighted and displayed on the angle distribution histogram.

The picking motion simulation using the robot simulation apparatus 500 according to the thirteenth embodiment is performed by the same method as that shown in FIGS. 67 and 68 described above. The angle threshold is set, for example, when setting the simulation environment in step S6701 of FIG. Here, the procedure for setting the simulation environment will be described based on the flowchart of FIG.
(Simulation Environment Setting Procedure According to Embodiment 13)

  First, in step S7501, the sensor unit measures the three-dimensional shape of a group of workpieces in a stacked state. Thereby, a height image having height information as shown in FIG. 76 is obtained. In this example, three-dimensional measurement is performed by the pattern projection method, but other methods such as stereo measurement and TOF can also be used.

  Next, in step S7502, tilt information is generated for each pixel constituting the captured three-dimensional height image. Here, a normal map can be created by performing a differentiation process on the height image. The slope of each point is calculated from the Z component of this normal. The slope of each point can be expressed by the following equation.

  θ = acosNz (Nz: Z component of normal)

  In step S7503, an angular distribution indicating the distribution of inclination information is calculated. Here, an inclination distribution within the selected range is created. Here, when the floor surface included in the height image is included in the calculation, the ratio of the inclination 0 increases. Therefore, it is preferable to perform mask processing so as to exclude such regions. For example, the height below a certain level is ignored. Also, normal information tends to be unstable in the vicinity of invalid pixels, and this may be ignored.

  Finally, in step S7504, an angle threshold value used for the picking motion simulation is determined from the calculated angle distribution. For example, an angle at which the cumulative distribution is 80% or more is set as a threshold of an angle that can be measured. In this way, the angle threshold can be determined as one of the environment settings for the picking motion simulation.

  In the above example, the angle distribution is created from the result of imaging the stacked workpiece group once, but the present invention is not limited to this configuration. For example, the distribution may be created from the result of imaging the work group multiple times. For example, by creating an angle distribution from an average of height images obtained by a plurality of times of imaging, it is possible to reduce the influence of sporadic noise and improve stability.

  As described above, according to the thirteenth embodiment, when performing the simulation of bulk picking, the position and posture of the workpiece are not assumed to be known. Instead, an image imitating the imaging at the sensor unit is created, and a three-dimensional search process is performed on the simulated image to obtain the position and orientation of the workpiece. A simulation is performed using the position and orientation obtained in this way. Thereby, it is possible to obtain a simulation result closer to the actual according to the actual position and orientation of the workpiece, not the virtually obtained position and orientation of the workpiece.

  In addition, in order to obtain a simulated image that is closer to the actual situation, it is not possible to measure all of the workpiece surface in three dimensions, but whether or not measurement is possible depending on the angle of the workpiece surface. It may be deleted. For example, as in Embodiment 1 described above, invalid pixels are removed from the stacking data by the region estimation unit. Specifically, the three-dimensional measurement is performed based on the inclination angle information of each point constituting the three-dimensional measurement data of the plurality of workpieces stored in the measurement data storage unit 52 illustrated in FIG. 71 with respect to the measurement axis of the sensor unit 2. Are removed as invalid pixels. Then, the angle distribution is calculated by the angle threshold value determination unit 54 based on the data from which the invalid pixels are removed.

As described above, according to the robot simulation apparatus 500 according to the thirteenth embodiment, the optimum angle threshold for performing the imaging simulation can be determined based on the actual measurement data, and a more realistic stacking simulation can be performed.
(Embodiment 14)

  In the above example, the example in which the angle threshold value is determined based on a histogram or the like of the height image obtained by actually capturing the workpieces in the stacked state with the sensor unit has been described. Furthermore, according to the present invention, a three-dimensional search can be performed on an actually captured image, and an angular range that can be measured can be determined using the search result. Such an example will be described as Embodiment 14 with reference to FIGS. 77A and 77B. FIG. 77A shows a profile of three-dimensional measurement data which is point cloud data obtained by three-dimensional measurement of a workpiece group actually in a stacked state, and FIG. 77B shows a state where search results are superimposed on this profile. ing. Here, based on the three-dimensional search result, each point on the surface of the work model used as the search model is arranged on the measurement data as shown in FIG. 77B. Then, at each point on the surface, it is checked whether there is corresponding measurement data, and an angular range that can be measured is determined. In FIG. 77A, the convex portion of the profile indicates a measurable region, and the flat portions on both sides indicate a non-measurable region. In contrast, a three-dimensional search is performed with a work model corresponding to the same kind of work. For example, if a three-dimensional search is performed with a cylindrical workpiece model, as shown in FIG. 77B, it is determined that the matched part, the three points on the left side in the figure, can be measured, and the non-matched part, two on the right side in FIG. A point is determined to be unmeasurable.

  Since the procedure for picking motion simulation using the robot simulation apparatus according to the fourteenth embodiment and the procedure for generating the stacked images can use the same procedures as those in FIGS. 66 and 67 described above, detailed description thereof will be omitted. The angle threshold value setting using the search result of the three-dimensional search is performed, for example, when setting the simulation environment in step S6701 of FIG. Here, the procedure for setting the simulation environment will be described based on the flowchart of FIG.

  First, in step S7801, a search model for a three-dimensional search is registered. Here, the search model can be registered using the work model, here, three-dimensional CAD data as described above. Or you may register using the height image actually imaged with the sensor part. At the time of registration, normal line information indicating the inclination of the surface at each point on the search model is stored.

  Next, in step S7802, an actual workpiece stacking state is created and three-dimensional measurement is performed. Here, three-dimensional measurement is performed by the projection method and the point cloud data is acquired (FIG. 77A). However, the present invention is not limited to this, and other methods such as stereo measurement and TOF may be used.

  Next, in step S7803, a three-dimensional search is performed on the obtained point cloud data using a search model to determine the position and orientation of each workpiece (FIG. 77B).

  In step S7804, it is confirmed whether or not a corresponding measurement point exists at each point of the search model. Here, as a result of the three-dimensional search, points on the search model are arranged in the three-dimensional space using the position and posture of the work determined. When the three-dimensional measurement data exists in the vicinity of the arranged point, it is determined that the point has been measured (FIG. 77B).

  Finally, in step S7805, an angle threshold is determined based on the inclination range of the point where the measurement point exists. Here, using the position and orientation of the search model obtained by the three-dimensional search, the normals of the points on the workpiece that can be measured are arranged in the three-dimensional space. Then, the slope of this normal is calculated. By summing up the normal angle ranges of the points that can be measured in this way, the threshold value of the surface inclination that can be measured by the workpiece is determined by the angle threshold value determination unit. In this way, the angle threshold value can be determined based on the result of the three-dimensional search.

  In addition, by giving normal information to each point that constitutes the search model used in the 3D search, point cloud data that is the actual measurement value of the workpiece superimposed with the search model that obtained the position and orientation as a result of the 3D search Even if the tilt information is not obtained, the tilt information on the search model side can be used. For example, in FIG. 31B, by using the information on the search model side as the tilt information of the position indicated by the point, it is possible to supplement the information of the point where the actual measurement value is not obtained (for example, the point where measurement is impossible). . At this time, the inclination information on the search model side may be used for the point where the actual measurement value is obtained from the point cloud data.

  The robot simulation apparatus, the robot simulation method, the robot simulation program, the computer-readable recording medium, and the recorded device of the present invention can be suitably used for the purpose of verifying the robot picking operation.

DESCRIPTION OF SYMBOLS 1000 ... Robot system 100, 200, 300, 400, 500, 600 ... Robot simulation apparatus 1 ... Image processing part 2 ... Sensor part 3 ... Display part 4 ... Operation part 5 ... Robot main body 6 ... Robot controller 7 ... Robot operation tool 10 ... simulation environment setting unit 11 ... work model setting unit 12 ... surface condition parameter setting unit 13 ... angle threshold determination unit 14 ... container model setting unit 15 ... sensor model setting unit 16 ... attitude condition setting unit 17 ... attitude stability parameter determination unit DESCRIPTION OF SYMBOLS 18 ... Exclusion workpiece selection part 19 ... Exclusion condition calculation part 20 ... Bulk accumulation data generation part 22 ... Area estimation part 23 ... Bulk accumulation data storage part 24 ... Cause analysis part 29 ... Simulation parameter adjustment part 30, 30B ... Picking operation simulation Unit 31 ... Stacked image generation unit 32 ... Position and orientation detection unit 33 ... Achieving calculation unit 40 ... simulation data generation unit 50 ... detection setting unit 52 ... measurement data storage unit 54 ... angle threshold value determination unit 60 ... physical simulation unit 110 ... posture condition setting screen 120 ... arrangement condition setting screen EET, EET2, EET3 ... End effector ARM ... Arm WK ... Work WK2 ... Hollow work WK3 ... Plate-like work WK4 ... Resin work WK5 ... Metal work WK6 ... Work WK7 ... Plate-like work with different surface and back surface areas WK8: Cylindrical work WM, WM6, WM7 ... Work model WM4 ... Resin work model WM5 ... Metal work model WMT ... T-shaped work model WMC ... Cylindrical work model PRJ ... Projectors CME1, CME2, CME3, CME4 ... Camera CMM, CMM2, CMM3 ... Camera Del BX ... container SL ... insole DIV ... partition DSA ... designated area WMF ... wireframe shaped workpiece model DSP ... development view PM ... projector model specified vertices WMD ... workpiece model

Claims (11)

A robot simulation device for simulating a bulk picking operation in which a three-dimensional shape of a plurality of workpieces stacked in a work space is measured by a sensor unit and sequentially taken out by a robot,
A measurement data storage unit for storing three-dimensional measurement data for each position, which is obtained by three-dimensionally measuring a plurality of stacked workpieces with a sensor unit, and having information on an inclination angle with respect to a measurement axis of the sensor unit for each position;
Angle threshold value that serves as a reference for determining whether or not three-dimensional measurement is possible based on the inclination angle information of each point constituting the three-dimensional measurement data of a plurality of workpieces stored in the measurement data storage unit with respect to the measurement axis of the sensor unit An angle threshold value determination unit for determining
A work model setting unit for setting a work model obtained by modeling the three-dimensional shape of the work;
A bulk data generation unit for generating virtual bulk data in which a plurality of the work models are stacked in a virtual work space that is a virtual work space;
The virtual inclination angle of each point constituting each work model in the bulk data generated by the bulk data generation unit is compared with the angle threshold determined by the angle threshold determination unit. An area estimation unit for identifying a point having a large virtual inclination angle as an estimation area estimated to be difficult to measure;
A data generation unit for simulation for generating, as data for simulation, unpacking data that does not include the data of the estimation region specified by the region estimation unit,
A robot simulation apparatus comprising: a picking motion simulating unit for executing a simulation for verifying a bulk picking motion of the work model in the work space using the simulation data. The robot simulation apparatus according to claim 1, further comprising:
An angle distribution indicating a distribution in which the inclination angles of the respective points constituting the three-dimensional measurement data of the plurality of workpieces stored in the measurement data storage unit are accumulated based on the inclination angle information with respect to the measurement axis of the sensor unit. A robot simulation apparatus comprising a display unit capable of displaying a histogram. The robot simulation device according to claim 2,
A robot simulation apparatus configured such that the angle threshold value determination unit accepts a manual setting of an angle threshold value by referring to an angle distribution histogram displayed on the display unit. The robot simulation device according to claim 2 or 3,
A robot simulation apparatus, wherein the display unit is configured to be able to display simulation data generated by the simulation data generation unit. The robot simulation device according to any one of claims 1 to 4,
Based on the inclination angle information of each point constituting the three-dimensional measurement data of the plurality of workpieces stored in the measurement data storage unit with respect to the measurement axis of the sensor unit, a point with low reliability of the three-dimensional measurement is set as an invalid pixel. A robot simulation device configured to be removed. The robot simulation device according to any one of claims 1 to 5,
The angle threshold value determination unit determines the inclination angle of each point based on the inclination angle information with respect to the measurement axis of the sensor unit of each point constituting the three-dimensional measurement data of the plurality of workpieces stored in the measurement data storage unit. A robot simulation device configured to automatically calculate an angle threshold based on an angle distribution histogram indicating an accumulated distribution. A robot simulation device for simulating a bulk picking operation in which a three-dimensional shape of a plurality of workpieces stacked in a work space is measured by a sensor unit and sequentially taken out by a robot,
A work model setting unit for setting a work model obtained by modeling the three-dimensional shape of the work;
A bulk data generation unit for generating virtual bulk data in which a plurality of the work models are stacked in a virtual work space that is a virtual work space;
A measurement data storage unit for storing three-dimensional measurement data obtained by three-dimensionally measuring a plurality of stacked workpieces with different postures;
A three-dimensional search is performed on the three-dimensional measurement data of a plurality of workpieces stored in the measurement data storage unit using the workpiece model set in the workpiece model setting unit as a search model, and the search model and the three-dimensional measurement data An angle threshold value determination unit for determining an angle threshold value serving as a reference for determining whether or not three-dimensional measurement is possible based on the position and orientation of the point where
The virtual inclination angle of each point constituting each search model in the bulk data generated by the bulk data generation unit is compared with the angle threshold determined by the angle threshold determination unit. An area estimation unit for identifying a point having a large virtual inclination angle as an estimation area estimated to be difficult to measure;
A data generation unit for simulation for generating, as data for simulation, unpacking data that does not include the data of the estimation region specified by the region estimation unit,
A robot simulation apparatus comprising: a picking motion simulating unit for executing a simulation for verifying a bulk picking motion of the work model in the work space using the simulation data. The robot simulation device according to claim 7,
As a result of the three-dimensional search, the angle threshold determination unit determines that the point where the work model and the three-dimensional measurement data match can be measured, and arranges the normal of the measurable point in the three-dimensional space A robot simulation apparatus configured to calculate a normal inclination angle of a measurable point by calculating a normal inclination and totaling a normal angle range of the measurable points. A robot simulation method for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially picked up by a robot,
A step of three-dimensionally measuring a plurality of stacked workpieces with different postures, storing the information of the tilt angle with respect to the measurement axis of the sensor unit as three-dimensional measurement data for each position;
Determining an angle threshold value serving as a reference for determining whether or not three-dimensional measurement is possible, based on inclination angle information of each point constituting the stored three-dimensional measurement data of the plurality of workpieces with respect to the measurement axis of the sensor unit; ,
A process of setting a work model obtained by modeling the three-dimensional shape of the work;
A step of generating virtual loose stacking data in which the plurality of set work models are stacked in a virtual work space which is a virtual work space;
It is difficult to measure a point having a virtual inclination angle larger than the angle threshold value by comparing the virtual inclination angle of each point constituting each work model in the generated stacking data with the determined angle threshold value. Identifying the estimated area as
A step of generating loose stacking data that does not include the data of the identified estimation region, as simulation data;
Using the generated simulation data to execute a simulation for verifying the bulk picking operation of the work model in the work space;
A robot simulation method including: A robot simulation program for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially taken out by a robot,
A function of storing, as a three-dimensional measurement data having a tilt angle with respect to a measurement axis of the sensor unit for each position, three-dimensionally measuring a plurality of stacked workpieces with different positions,
A function of determining an angle threshold value serving as a reference for determining whether or not three-dimensional measurement is possible based on inclination angle information with respect to a measurement axis of a sensor unit of each point constituting the stored three-dimensional measurement data of a plurality of workpieces; ,
A function to set a work model that models the 3D shape of the work,
A function for generating virtual loose stacking data in which the plurality of set work models are stacked in a virtual work space that is a virtual work space;
It is difficult to measure a point having a virtual inclination angle larger than the angle threshold value by comparing the virtual inclination angle of each point constituting each work model in the generated stacking data with the determined angle threshold value. A function that is identified as an estimated area, and
A function of generating loose stacking data that does not include the data of the specified estimation area, as simulation data;
A function for executing a simulation for verifying a bulk picking operation of the work model in the work space, using the generated simulation data;
A robot simulation program for realizing a computer.   The computer-readable recording medium which recorded the program of Claim 10, or the apparatus which memorize | stored.