JP2019028773A - Robot simulation device and robot simulation method - Google Patents

Abstract

To enable a gripping position to be adopted or rejected based on an objective data by making a gripping position low in contribution rate toward the extraction successful rate of a work comparable with the other gripping position when a plurality of gripping positions is registered in the work.SOLUTION: A robot simulation device sets a plurality of gripping candidate positions gripped by a robot in association with each search model, and determines whether or not gripping by the robot is possible with respective gripping candidate positions associated with a work model that succeeds in the search. The picking action simulation in which the gripping candidate position determined to be grippable is gripped by the robot is repeatedly executed. Information showing use frequency of each search model and/or each gripping candidate position is comparably displayed.SELECTED DRAWING: Figure 36

Description

  The present invention relates to a robot simulation apparatus and a robot simulation method.

  For example, at the assembly site of various products, a plurality of parts (workpieces) housed in a return box or the like are gripped by an end effector such as a hand unit attached to an arm of an industrial robot, sequentially taken out, and transported to a predetermined place. Things have been done.

  Industrial robots are increasingly used in combination with a robot vision as a visual sensor. A robot system equipped with a robot vision first picks up an appropriate position of a pre-registered workpiece after picking up the workpiece height information by imaging the workpiece to be grasped by the robot vision, The operation of transferring the workpiece to a predetermined place and placing it (place) is repeatedly performed.

  Moreover, although the workpiece | work before conveyance may be stacked in the aligned state, it may not be aligned, but may be in the non-aligned state that was randomly stacked, that is, it may be stacked in bulk. When the workpieces are stacked in bulk, the position and posture of the workpiece are different for each return box, and there is usually no workpiece having the same position and posture. Such an operation of sequentially taking out the stacked workpieces, transporting the workpieces to a predetermined place, and placing the workpieces is called “bulk picking”. Bulk picking is much more difficult than picking aligned workpieces, so it is possible to simulate robot picks and place motions three-dimensionally using a robot simulation device before operating the robot. Yes (see, for example, Patent Documents 1 to 4).

  In such a robot simulation apparatus, a CAD model having a three-dimensional shape of a work is used as a work model, and is randomly arranged in a virtual work space to simulate a take-out operation by the robot.

Japanese Patent No. 4153528 JP 2015-215259 A JP 2002-331480 A Japanese Patent Laying-Open No. 2015-171745

  By the way, when performing a bulk picking operation by a robot, there are many cases in which a plurality of gripping positions and gripping postures at each gripping position are registered for one kind of work. This is to increase the success rate of picking up the workpiece, but even if a plurality of gripping positions are registered, not all of them are effective. That is, even if it is possible to set multiple gripping positions based on the shape of the workpiece, some of the gripping positions can be actually gripped by the hand part in bulk picking work, or bulk picking When the work is performed, it is assumed that the gripping frequency of some gripping positions is significantly lower than that of other gripping positions, and the setting of the gripping positions makes little sense.

  However, it is not easy for the user to determine a gripping position that is suitable or inappropriate for gripping the workpiece by comparing it with other gripping positions. Judgment may also be difficult.

  In general, there are a plurality of types of hand parts used in bulk picking operations, and it is well known that the success rate of picking up a workpiece varies greatly depending on the shape of the hand part. However, in selecting the hand part, not only the shape of the work but also the number of work, the work space (container box shape), etc. have a great influence. It is difficult to determine whether it is suitable by comparison with other hand units.

  The present invention has been made in view of the above points, and the purpose of the present invention is to register a gripping position that has a low contribution rate for improving the workpiece removal success rate when a plurality of gripping positions are registered in the workpiece. The gripping position can be selected based on objective data so that it can be easily compared with the gripping position of the workpiece, and the success rate of picking up the workpiece can be improved when registering the shape of multiple hands. Therefore, it is possible to easily select a hand part based on objective data by making it possible to easily compare the shape of a hand part having a low contribution rate with the shape of another hand part.

  In order to achieve the above object, a first invention is a robot simulation apparatus for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially taken out by a robot, and a three-dimensional shape of the workpiece is modeled A work model setting unit for setting a converted work model, a bulk data generation unit for generating data in a bulk state in which a plurality of work models set in the work model setting unit are stacked in a virtual work space, and A search model registration unit for registering a plurality of search models used for search processing for detecting the position and orientation of the work model with respect to the bulk data generated by the bulk data generation unit, and gripped by the robot A plurality of gripping candidate positions are obtained by the search model registration unit. Using the search model registered by the search model registration unit, the gripping position setting unit that can be set in association with each registered search model, and the bulk data generated by the bulk data generation unit, A search processing unit that executes a search process for detecting a position and orientation, and a gripping candidate position associated with a work model that has been successfully searched by the search processing unit, determine whether or not gripping by a robot is possible. A picking motion simulation unit that repeatedly executes a picking motion simulation for gripping a gripping candidate position determined to be gripped by the robot, and each search model and / or each gripping candidate position when gripping is successful in the picking motion simulation. Compare information indicating the frequency of use Characterized in that it comprises a Shimesuru display control unit.

  According to this configuration, it is possible to register a plurality of search models and set a plurality of gripping candidate positions for each search model in association with the search model. A search process that detects the position and orientation of the work model is executed using the registered search model, and the robot can grip each of a plurality of gripping candidate positions associated with the work model that has been successfully searched. In some cases, a picking operation for gripping the gripping candidate position by the robot is simulated. By repeatedly executing this simulation, it is possible to display information indicating the frequency of use of each search model and / or each gripping candidate position when the gripping is successful.

  A second invention is a robot simulation apparatus for simulating a bulk picking operation in which a plurality of works stacked in a work space are sequentially picked up by a robot, and for setting a work model that models the three-dimensional shape of the work Generated by the above-described work model setting unit, a bulk data generation unit that generates bulk data by stacking a plurality of work models set in the work model setting unit in a virtual work space, and the bulk data generation unit A search model registration unit for registering a plurality of search models used for search processing for detecting the position and orientation of the work model for the bulk data, and a hand unit capable of inputting a plurality of shapes of the robot hand unit Generated by the shape input unit and the bulk data generation unit A search processing unit that executes a search process for detecting the position and orientation of the work model from the bulk data using the search model registered by the search model registration unit, and a work model that has been successfully searched by the search processing unit A picking motion simulation unit that repeatedly executes the picking motion simulation using the hand unit input by the hand unit shape input unit using each hand unit, and a work model by each hand unit obtained by the picking motion simulation. And a display control unit that displays information indicating the success degree of gripping in a comparable manner.

  According to this configuration, it becomes possible to input a plurality of shapes of the hand portion of the robot by the hand portion shape input unit. The picking operation simulation is repeatedly executed using each hand unit input by the hand unit shape input unit, and information indicating the success degree of gripping the work model by each hand unit can be displayed in a comparable manner.

  According to a third aspect of the present invention, the bulk data generation unit is configured to execute a physical simulation that simulates an operation in which a work is placed in the work space under the action of gravity using a plurality of work models, Each time the picking motion simulation by the picking motion simulation unit ends, the physical simulation is newly executed to update the data. According to this configuration, the picking operation simulation closer to the actual operation can be performed, so that the reliability of the information displayed in a comparable manner is increased.

  According to a fourth aspect of the present invention, the display control unit is configured to be able to display each search model and / or each gripping candidate position that has not been used for gripping or has been used with a very low frequency. According to this configuration, as a result of the picking motion simulation, the user can know search models and gripping candidate positions that are not used for gripping or are used very frequently, so that they can be used in practice without them. Can do.

  According to a fifth aspect of the invention, the picking motion simulation unit is configured to continue the picking motion simulation until gripping by each hand unit cannot be performed, and the display control unit is configured to end the picking motion simulation by the picking motion simulation unit. The number of remaining work models is configured to be displayed for each hand unit. According to this configuration, the success degree of gripping by the hand unit is displayed in a specific form of the remaining number of work models, so that the user can easily select the hand unit.

  A sixth invention is a robot simulation method for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially taken out by a robot, and a workpiece model setting for setting a workpiece model modeling a three-dimensional shape of a workpiece A bulk data generation step for generating data in a bulk state in which a plurality of work models set in the work model setting step are stacked in a virtual work space; and for the bulk data generated in the bulk data generation step, A search model registration step for registering a plurality of search models used in search processing for detecting the position and orientation of a work model, and a plurality of gripping candidate positions gripped by the robot are registered in the search model registration step. Searchmo A search for detecting the position and orientation of the work model from the bulk data generated in the bulk data generated in the bulk data generated in the grip position setting step and the bulk data generation step. For each of the search processing step for executing the process and the gripping candidate positions associated with the work model that has been successfully searched in the search processing step, it is determined whether or not gripping by the robot is possible. A picking motion simulation step for repeatedly executing a picking motion simulation for gripping a gripping candidate position by a robot, and information indicating the use frequency of each search model and / or each gripping candidate position when gripping is successful in the picking motion simulation. Comparable table Characterized in that it comprises a display step of.

  A seventh invention is a robot simulation method for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially taken out by a robot, and a workpiece model setting for setting a workpiece model modeling a three-dimensional shape of a workpiece A bulk data generation step for generating data in a bulk state in which a plurality of work models set in the work model setting step are stacked in a virtual work space; and for the bulk data generated in the bulk data generation step, A search model registration step for registering a plurality of search models used in search processing for detecting the position and orientation of a work model, a hand portion shape input step for inputting a plurality of hand portion shapes of the robot, and the bulk data generation Step Using the search model registered in the search model registration step, the search processing step for executing the search process for detecting the position and orientation of the work model from the bulk data generated in step 1, and the search was successful in the search processing step. A picking motion simulation step for repeatedly executing a picking motion simulation for gripping a work model by the hand portion input in the hand portion shape input step, and a degree of success of gripping the work model by each hand portion obtained by the picking motion simulation And a display step for displaying information indicating that the information can be compared.

  According to the first invention, a plurality of gripping candidate positions can be set in association with each other for a plurality of search models, and a search process for detecting the position and orientation of the work model is executed using the registered search models. If each of the gripping candidate positions associated with the work model that has been successfully searched can be gripped by the robot, a picking operation of gripping the gripping candidate position by the robot can be simulated. Since this simulation is repeatedly executed and information indicating the frequency of use of each search model and / or each gripping candidate position when gripping is successful can be displayed in a comparable manner, the contribution rate for improving the success rate of picking up the workpiece Can be easily compared with other gripping positions, and the gripping positions can be selected based on objective data.

  According to the second invention, a plurality of shapes of the robot hand portion are input by the hand portion shape input portion, a picking operation simulation is executed using each input hand portion, and the gripping of the work model by each hand portion is performed. Since the information indicating the degree of success can be displayed in a comparable manner, the shape of the hand part, which has a low contribution rate to improving the workpiece removal success rate, can be easily compared with the shape of the other hand parts. Based on this, the hand part can be selected.

It is a schematic diagram which shows a mode that the bulk picking operation | movement is performed using a robot system. It is a block diagram which shows schematic structure of a robot system. It is a block diagram which shows the detailed structure of a robot system. It is a perspective view which shows an example of a sensor part. 5A is a schematic diagram showing an example of gripping a workpiece with an end effector, FIG. 5B is a schematic diagram showing an example of gripping a workpiece having a cavity from the inner surface, and FIG. 5C shows an example of sucking and gripping a plate-like workpiece. It is a schematic diagram. 6A is a schematic cross-sectional view showing an example in which workpieces are randomly placed in a storage container and stacked, FIG. 6B is a schematic cross-sectional view showing an example in which workpieces are stacked on a floor surface, and FIG. It is a perspective view which shows the state arranged on the top. It is a flowchart which shows the procedure of robot simulation. It is a flowchart which shows the setting procedure of a simulation environment. It is a flowchart which shows the registration procedure of the search model when CAD data of a workpiece | work exists. It is a perspective view which shows the work model based on three-dimensional CAD data. It is a figure which shows the height image of the workpiece | work model shown in FIG. It is a perspective view which shows another work model based on three-dimensional CAD data. It is a figure which shows the height image of the workpiece | work model shown in FIG. It is a flowchart which shows another example of the registration procedure of a search model when CAD data of a workpiece | work exists. It is a flowchart which shows the example of the registration procedure of the search model when there is no CAD data of a workpiece | work. It is a figure which shows the height image of a work model in case there is no CAD data. It is a figure which shows the height image which has arrange | positioned the feature point. It is a flowchart which shows the example of the registration procedure of a hand model. It is a side view which shows the hand model based on three-dimensional CAD data. It is a figure which shows the cross section of a hand model. FIG. 21A is a diagram illustrating a gripping position setting screen on which a gripping posture list is displayed, and FIG. 21B is a diagram illustrating a gripping position setting screen on which a gripping posture setting area is displayed. It is a figure which shows the state by which the hand model is arrange | positioned in the position which can hold | grip the workpiece holding candidate position. It is a figure which shows the state by which the hand model was arrange | positioned in the position which can hold | grip a candidate grip position about each of six search models. FIG. 24A is a diagram showing a work model before the number of polygons is reduced, and FIG. 24B is a diagram showing a work model after the number of polygons is reduced. It is a figure which shows the workpiece | work model after reducing the number of polygons until the original three-dimensional shape of a workpiece | work collapses greatly. It is a figure which shows the screen which displayed the physical simulation image and the height image simultaneously. It is a figure which shows the screen which displayed the physical simulation image and height image which looked at changing a viewpoint simultaneously. It is a figure which shows the screen which displays the state which detected the work model in the height image. FIG. 29A is a diagram showing a case where the work model constituting the height image used for the three-dimensional search is before the compression process, and FIG. 29B is a work model before the compression process and a search result (search for the work before the compression process). It is the figure which superimposed and displayed the position of each feature point of a model. FIG. 30A is a diagram showing a case where the work model constituting the height image used for the three-dimensional search is after the compression process, and FIG. 30B is a work model after the compression process and a search result (search for the work before the compression process). It is the figure which superimposed and displayed the position of each feature point of a model. It is a figure in the case where the superimposed display of the result of a three-dimensional search (position of each feature point of a search model) is displayed in two dimensions and in three dimensions. FIG. 32A is a diagram illustrating a case where the hand model and the three-dimensional measurement point do not interfere with each other, and FIG. 32B is a diagram illustrating a case where the hand model and the three-dimensional measurement point are interfering with each other. It is a figure which shows the image in picking operation | movement simulation. It is a statistics accumulation | storage flowchart which shows the procedure which performs picking operation | movement simulation in multiple times and accumulate | stores a result. It is the table | surface which displayed the result of the picking operation | movement simulation as a list | wrist in order of a grip registration number. It is the table | surface which displayed the list of the result of picking operation | movement simulation in descending order. It is the table | surface which displayed the result of the picking operation | movement simulation as a list according to the shape of a hand part. It is a flowchart which shows the procedure at the time of operation. It is a figure which shows the case where an attention area | region is set within the imaging range. It is a graph which shows the relationship between exposure time and the number of effective pixels. It is a figure which shows the setting point of exposure time. It is a screen for exposure time setting, and is a figure which shows the case where exposure time is in a recommended value. It is an exposure time setting screen, and shows a case where the exposure time is shorter than the recommended value. It is an exposure time setting screen and is a diagram showing a case where the exposure time is longer than the recommended value. FIG. 45A is a diagram in which a height image generated based on measurement data obtained by a three-dimensional measurement unit is displayed in a three-dimensional manner from a slightly oblique direction, and a feature point of a search model is superimposed on the position of a search result. 45B, in addition to FIG. 45A, estimates the work surface that cannot be measured by the three-dimensional measuring unit using the relevance to other surfaces, and includes the feature points of the search model of another surface. FIG. It is a flowchart which shows another procedure at the time of operation. FIG. 47A is an image showing a case where an erroneous determination result is obtained by performing interference determination without using the surface relationship, and FIG. 47B is an image in which interference determination is performed using the surface relationship and the correct determination result is obtained. It is an image which shows the case where it has come out. It is a figure which shows the image displayed on a display part when performing interference determination. It is a figure explaining the case where the cube-shaped workpiece | work is piled up in bulk.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It should be noted that the following description of the preferred embodiment is merely illustrative in nature, and is not intended to limit the present invention, its application, or its use.

  FIG. 1 is a schematic diagram illustrating a configuration example of a robot system 1000 according to an embodiment of the present invention. In FIG. 1, a plurality of workpieces WK stacked in a work space in a manufacturing factory for various products are sequentially picked up using a robot RBT, conveyed to a stage STG installed at a predetermined place, and the stage STG. An example of performing bulk picking placed on top is shown.

  The robot system 1000 includes a robot RBT and a robot controller 6 that controls the robot RBT. The robot RBT is an industrial general-purpose robot, and a base portion is fixed to a floor surface of a factory or the like. The robot RBT is also called, for example, a manipulator or the like, and is configured to be capable of 6-axis control. The robot RBT includes an arm part ARM extending from the base part, and an end effector EET provided at the tip of the arm part ARM. The arm part ARM can be configured as a multi-joint type including a plurality of joint parts as movable parts. The end effector EET can be moved to a desired position within the movable range by the operation of each joint portion of the arm portion ARM and the rotation operation of the arm portion ARM itself.

  The end effector EET can be configured by a hand unit or the like that grips the workpiece WK. 5A, the end effector EET sandwiches and grips the outside of the workpiece WK, and the claw portion is inserted into the workpiece WK2 having a hollow portion as shown in FIG. 5B to expand the end effector EET. As shown in FIG. 5C, there are an end effector EET2 for sucking and holding a plate-like work WK3 as shown in FIG. 5C, and any end effector can be used. Further, in this specification, “gripping” is meant to include all examples such as a method of sandwiching the outside of the work WK shown in FIG. use.

  The robot RBT is controlled by the robot controller 6. The robot controller 6 controls the operation of the arm unit ARM and the opening / closing operation of the end effector EET. Further, the robot controller 6 acquires information necessary for controlling the robot RBT from the robot setting apparatus 100 shown in FIG. For example, the robot setting device 100 acquires the three-dimensional shape of the workpiece WK, which is a large number of parts randomly placed in the storage container BX shown in FIG. 1, using the sensor unit 2 such as a three-dimensional camera or illumination. Thus, the position and posture of the workpiece WK are detected, and the robot controller 6 acquires the information.

  A functional block diagram of the robot system 1000 is shown in FIG. A robot system 1000 shown in this figure includes a robot setting device 100, a sensor unit 2, a display unit 3, an operation unit 4, and a robot operation tool 7 in addition to the robot RBT and the robot controller 6.

  The operation unit 4 performs various settings relating to physical simulation, picking operation simulation, and image processing, which will be described later. Further, the sensor unit 2 captures an image of the workpiece WK and acquires a three-dimensional shape of the workpiece WK. Further, on the display unit 3, confirmation of setting and operation state, confirmation of simulation, and the like are performed. Further, the robot simulation device 200 and the image processing device 300 provided in the robot setting device 100 perform various simulations, three-dimensional search, interference determination, grip solution calculation, and the like.

  On the other hand, the robot controller 6 is a known member configured to control the robot RBT in accordance with a signal output from the robot setting device 100. The robot operation tool 7 performs operation settings for the robot RBT. In the example of FIG. 2, the operation unit 4 and the robot operation tool 7 are separate members, but they may be common members.

  The sensor unit 2 is a member called a robot vision or the like that images a work space or a work WK. In this embodiment, the work WK is at least an imaging object. It becomes possible to acquire three-dimensional shape data indicating the three-dimensional shape of the workpieces WK stacked in bulk from the image captured by the sensor unit 2. Note that methods for obtaining a three-dimensional shape include, for example, a pattern projection method, a stereo method, a lens focus method, a light cutting method, an optical radar method, an interference method (white interference method), a TOF method, and the like. Any method may be used. Since each method is conventionally well-known, detailed description is abbreviate | omitted. In the present embodiment, among the pattern projection methods, a phase shift method is used in which pattern light having a periodic illuminance distribution is applied to the imaging target and light reflected by the surface of the imaging target is received.

  The sensor unit 2 is also a component of the shape measuring apparatus 400 that measures the shape of the workpiece WK that is a measurement object. The shape measuring device 400 can be a part of the robot setting device 100, but can also be configured as a separate member. A specific configuration of the sensor unit 2 is determined according to a three-dimensional shape measurement technique. The sensor unit 2 includes a camera, an illumination, a projector, or the like. For example, when the three-dimensional shape of the workpiece WK is measured by the phase shift method, as shown in FIG. 4 as the sensor unit 2, a projector (light projecting unit) PRJ and a plurality of cameras (light receiving units) CME1, CME2, CME3, CME4. The projector PRJ is a member that irradiates light onto the work WK. The cameras CME1, CME2, CME3, and CME4 are members having an image sensor that receives light projected by the projector PRJ and reflected from the surface of the work WK. The light source of the light projecting unit can be composed of, for example, a plurality of light emitting diodes, a liquid crystal panel, an organic EL panel, a digital micromirror device (DMD), and the like.

  In addition, the sensor unit 2 may be configured by a plurality of members such as the cameras CME1, CME2, CME3, CME4, and the projector PRJ, or may be configured integrally. For example, the sensor unit 2 may be a 3D imaging head that is formed into a head shape by integrating the cameras CME1, CME2, CME3, CME4, and the projector PRJ.

  Further, the generation of the three-dimensional shape data itself can be performed by the sensor unit 2. In this case, the sensor unit 2 is provided with an image processing IC or the like that realizes a function of generating three-dimensional shape data. Alternatively, the 3D shape data is not generated on the image processing apparatus 300 side, and the raw image captured by the sensor unit 2 is image processed on the image processing apparatus 300 side to generate 3D shape data such as a 3D image. It is good also as composition to do.

  Furthermore, by executing calibration based on the image captured by the sensor unit 2, the actual position coordinates of the workpiece WK (the coordinates of the movement position of the end effector EET) and the image displayed on the display unit 3 are displayed. The upper position coordinates can be linked.

  The robot setting device 100 performs three-dimensional search, interference determination, grip solution calculation, and the like based on the three-dimensional shape data of the workpiece WK obtained by the sensor unit 2. The robot setting apparatus 100 can be configured by a general-purpose computer in which a dedicated image processing program is installed, a dedicated image processing controller, and dedicated hardware. In addition, the image processing program may be installed in a dedicated computer that specializes hardware such as a graphic board for image inspection processing.

  2 illustrates an example in which the sensor unit 2, the robot controller 6, and the like are configured by members separate from the robot setting device 100, the present invention is not limited to this configuration. For example, the sensor unit 2 and the image The processing apparatus 100 can be integrated, or the robot controller 6 can be incorporated into the robot setting apparatus 100. Thus, the division of the members shown in FIG. 2 is an example, and a plurality of members can be integrated. For example, the operation unit 4 that operates the robot setting device 100 and the robot operation tool 7 that operates the robot controller 6 may be a common member.

  However, the sensor unit 2 is separated from the robot RBT. That is, the present invention is directed to a form called an off-hand type in which the sensor unit 2 is not provided in the arm unit ARM of the robot body 5. In other words, an aspect called an on-hand type in which the sensor unit 2 is provided in the end effector EET is not included in the present invention.

  The display unit 3 is a member for displaying the three-dimensional shape of the workpiece WK acquired by the robot setting device 100, displaying various simulation images, and confirming various settings and operation states. A monitor, an organic EL display, a CRT, or the like can be used. The operation unit 4 is a member for performing various settings such as various simulations and image processing, and an input device such as a keyboard and a mouse can be used. Moreover, the operation part 4 and the display part 3 can also be integrated by making the display part 3 into a touch panel.

  For example, when the robot setting apparatus 100 is configured by a computer in which an image processing program is installed, a graphical user interface (GUI) screen of the image processing program is displayed on the display unit 3. Various settings can be performed from the GUI displayed on the display unit 3, and processing results such as simulation results can be displayed. In this case, the display unit 3 can be used as a setting unit for performing various settings.

  The robot controller 6 controls the operation of the robot based on information captured by the sensor unit 2. The robot operation tool 7 is a member for setting the operation of the robot RBT, and a pendant or the like can be used.

  As shown in FIG. 1, a plurality of workpieces WK are randomly stored in the storage container BX. The sensor unit 2 is disposed above the work space. Based on the three-dimensional shape of the workpiece WK obtained by the sensor unit 2, the robot controller 6 identifies the workpiece WK to be gripped from among the plurality of workpieces WK and holds the robot RBT so as to grip this workpiece WK. To control. Then, while holding the workpiece WK, the arm unit ARM is operated to move to a predetermined placement position, for example, on the stage STG, and the workpiece WK is placed in a predetermined posture. In other words, the robot controller 6 grips the workpiece WK to be picked specified by the sensor unit 2 and the robot setting device 100 with the end effector EET, and places the gripped workpiece WK in a predetermined reference posture at the placement location ( The operation of the robot RBT is controlled so as to be placed on the stage STG) and to release the end effector EET. Examples of the stage STG include a conveyor belt and a pallet.

  Here, in the present specification, the bulk picking means that the work WK randomly placed in the storage container BX as shown in FIG. 6A is gripped by the robot RBT and placed at a predetermined position. As shown in FIG. 6B, an example of gripping and placing the workpiece WK stacked in a predetermined area without using a storage container, or stacked in a predetermined posture as shown in FIG. 6C It is used to include an example in which the workpiece WK is sequentially held and placed. Further, the workpieces WK are not necessarily in a stacked state, and the workpieces WK randomly placed on a plane without overlapping the workpieces WK are also referred to as bulk stacking in this specification. (Even if the picking is done sequentially and there is no overlap between the workpieces WK at the end of picking, it is still the same reason as that called bulk picking). Note that the present invention is not necessarily limited to bulk picking, and can be applied to picking up workpieces WK that are not stacked.

  In the example of FIG. 1, the sensor unit 2 is fixed above the work space. However, the sensor unit 2 may be fixed at any position where the work space can be imaged. , Downward, diagonally downward, etc. However, the aspect which arrange | positions the sensor part 2 in the movable indefinite position like on the arm part ARM is excluded. Further, the number of cameras and illuminations included in the sensor unit 2 is not limited to one and may be plural. Furthermore, the connection with the sensor unit 2, the robot RBT, and the robot controller 6 is not limited to a wired connection, and may be a known wireless connection.

  In performing the bulk picking operation in the robot system 1000, teaching including setting for performing the bulk picking operation can be performed in advance. Specifically, registration is made of which part of the workpiece WK is gripped by the end effector EET in what kind of posture, the gripping position and the posture, and the like. Such setting can be performed by the robot operation tool 7 such as a pendant. Further, as will be described later, the setting can be performed on the vision space without operating the actual robot.

  The display unit 3 virtually displays a work model that virtually represents the three-dimensional shape of the work WK and an end effector model that is constituted by three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector EET. Each three-dimensionally in a three-dimensional space. Further, the display unit 3 can also display the basic direction image of the work model as a six-sided view. As a result, each posture of the work model can be displayed as a six-sided view, so that the gripping position setting work can be performed, and the gripping position setting work, which has been conventionally troublesome, can be easily performed.

(Robot simulation)
The robot system 1000 includes a robot simulation device 200 for simulating a bulk picking operation in which a plurality of workpieces WK stacked in a work space are sequentially picked up by a robot RBT. The robot simulation apparatus 200 can be integrated with the image processing apparatus 300 and can be provided inside the robot setting apparatus 100 together with the image processing apparatus 300. The image processing apparatus 300 is an apparatus used for controlling a robot RBT that performs a picking operation for gripping and sequentially taking out a plurality of works WK stacked in a work space.

  As shown in FIG. 3, the robot setting device 100 includes a work model setting unit 20, a data capacity compression unit 21, a physical simulation unit 22, a bulk data generation unit 23, a search processing unit 24, a picking operation simulation unit 25, and a display control unit. 26, search model registration unit 27, gripping position setting unit 28, three-dimensional measuring unit 30, interference determination unit 31, gripping position determination unit 32, height image generation unit 33, pixel determination unit 34, hand unit shape input unit 35, An exposure control unit 40 and an exposure time setting unit 41 are provided. Each of these units constitutes the robot simulation device 200 or the image processing device 300, and can be configured by hardware or software. Further, all of these parts can be integrated into one apparatus, or the robot setting apparatus 100 can be configured with a plurality of apparatuses by separating any part from other parts. These units can be realized by, for example, a microprocessor (MPU), a CPU, an LSI, a gate array such as an FPGA or an ASIC, hardware or software such as a DSP, or a mixture thereof. In addition, each component does not necessarily have the same configuration as that shown in FIG. 3 and the like, and its functions are substantially the same, and one element has the functions of a plurality of elements in the configuration shown in FIG. Are included in the present invention.

  For example, the exposure control unit 40 and the exposure time setting unit 41 can be provided in the sensor unit 2. The display control unit 26 can be provided in the display unit 3. The robot setting device 100 is also provided with a storage unit 42 for storing various information. The storage unit 42 can be configured by a semiconductor memory, a hard disk, or the like. The storage unit 42 may be provided with a reading device that can read information stored in various storage media such as a CD-ROM and a DVD-ROM.

  Before the operation of the robot system 1000, that is, before the work WK is grasped and placed at an actual work site, the robot simulation apparatus 200 performs a three-dimensional simulation of the grasping and placing actions of the robot RBT. It is configured to be able to. In the robot simulation, the position and orientation of the work model are detected by using a physical simulation that generates a bulk stack state in which a plurality of work models are randomly stacked, and a bulk stack state data generated by the physical simulation. A picking operation simulation for executing a picking operation simulation is performed. Hereinafter, the robot simulation apparatus 200 and the robot simulation method realized by the robot simulation apparatus 200 will be specifically described based on a robot simulation flowchart shown in FIG.

  When starting the robot simulation, after starting the robot simulation flowchart shown in FIG. 7, an environment (simulation environment) for executing the simulation is set in step SA1. The setting of the simulation environment includes the number of workpieces to be stacked, information on the storage container BX, floor information, design information on the camera CME and the projector PRJ constituting the sensor unit 2, and the like. Further, at the time of picking operation simulation, it is possible to cause random misalignment that may occur in actual operation to the storage container BX, and it is possible to include the range of misalignment at this time in the setting of the simulation environment.

  The main setting procedure of the simulation environment is shown in the simulation environment setting procedure flowchart of FIG. In step SB1 after the start of the simulation environment setting procedure flowchart, a search model for the work WK is registered. In subsequent step SB2, a hand model of the robot RBT is registered. The search model of the work WK in step SB1 is a model that represents the shape of the work WK used when executing a search process described later. The hand model of the robot RBT in step SB2 is a model representing the shape of the hand part (end effector EET). Step SB1 and step SB2 may be switched in order. In addition, a container model that models the storage container BX that stores workpieces, such as a returnable box, may be set.

  The procedure for registering the search model for the work WK in step SB1 is as shown in the search model registration flowchart of FIG. In the search model registration flowchart of FIG. 9, after the start, in step SC1, the three-dimensional CAD data (CAD model) of the work WK is read and temporarily stored in the storage unit 42 of the robot simulation apparatus 200. An example of a work model based on three-dimensional CAD data is shown in FIG. The CAD data is data of a work model, and data having a format generally used conventionally can be used.

  In this embodiment, the simplest STL format is used as the format of CAD data. The STL format is data composed only of a list of triangular polygon information (the coordinates of three points and the normal vector of the surface). Alternatively, the work model may be constituted by point cloud data having three-dimensional information. Alternatively, the work model may be constituted by image data having height information, for example, a height image or a distance image.

  Step SC1 is a work model setting step for setting a work model obtained by modeling the three-dimensional shape of the work, and is performed by the work model setting unit 20 included in the robot setting device 100 shown in FIG. There are cases where the number of polygons of the CAD data read in step SC1 is several thousand to several tens of thousands.

  In step SC2 following step SC1, a circumscribed cuboid of the CAD model is defined, and origin correction processing is performed to correct the center of the circumscribed cuboid to the CAD origin. The robot setting device 100 automatically determines the origin of the work model from the coordinate information of the three-dimensional CAD data. For example, a virtual cuboid circumscribing the work model can be defined for the three-dimensional CAD data of the work model, and the center of gravity of the virtual cuboid can be set as the origin of the work model.

  After that, the process proceeds to step SC3, where the six sides of the CAD model read in step SC1 and the CAD model are viewed from the “up”, “down”, “left”, “right”, “front”, and “back” directions. Generate height image data. That is, six height image data are generated so as to be a plan view, bottom view, left side view, right side view, front view, and rear view of the CAD model. A height image is obtained from the height image data. “Up” is the height image seen from the positive direction (plus side) of the Z axis, “Low” is the height image seen from the negative direction (minus side) of the Z axis, and “Left” is from the negative direction of the X axis. Viewed height image, “right” is a height image viewed from the positive direction of the X axis, “front” is a height image viewed from the negative direction of the Y axis, and “rear” is viewed from the positive direction of the Y axis. Each height image is shown. However, these are only examples, and different coordinate systems may be used, and the positive and negative directions of each axis are based on a coordinate system orthogonal to this axis with the X = Y line in the XY plane as the axis. You may use the height image seen from. In addition, when generating a height image from three-dimensional CAD data, directions orthogonal to the axis of the CAD data ("up", "down", "left", "right", "front", "back") It is not necessary that the height image is viewed from the above, and for example, the posture (viewpoint) of the work model may be arbitrarily changed, and the height image may be generated from the changed viewpoint.

  In this step SC3, since the CAD model is composed of three-dimensional CAD data, the three-dimensional CAD data is obtained when the X, Y, and Z coordinates of the CAD model are viewed from the plus and minus directions. By converting to a height image, height images can be obtained for the six faces of the CAD model. FIG. 11 shows a height image of the work model shown in FIG. In FIG. 10, it is displayed so that it becomes so high that it becomes white (it becomes brighter).

  In step SC4 subsequent to step SC3, the height images that look the same among the six height images generated in step SC3 are deleted. Appearance coincidence / disagreement is the height of a total of six height images viewed from the top and bottom of the workpiece (Z-axis positive / negative direction), front-back (Y-axis positive / negative direction), and left and right (X-axis positive / negative direction). It is generated based on the image data, and it is confirmed whether or not the height images match. Here, it is checked whether or not they are coincident by rotating at a step angle of 90 °, and the surface having the same appearance as the other surface is excluded from the registration target of the search model. Such exclusion can be performed manually by the user, can be automatically performed by the robot simulation apparatus 200 or the image processing apparatus 300, or can be performed in combination.

  A specific example will be described. For example, when the work WK is a rectangular parallelepiped, the height image viewed from the CAD model is the same as the height image viewed from the bottom. Delete one. Since the height image of the CAD model viewed from the left and the height image viewed from the right are the same, one of them is deleted. Also, since the height image of the CAD model viewed from the front and the height image viewed from the rear are the same, one of them is deleted. Thereby, since the number of search models can be reduced, the speed of each process mentioned later can be improved. Even if the workpiece WK has a complicated shape, the height image viewed from one direction may be the same as the height image viewed from the other direction. In this case, one of the height images can be deleted. . When the work WK is a cube, five of the six surfaces can be deleted.

  For example, when a work model having a three-dimensional shape as shown in FIG. 12 is read in step SC1, a height image obtained by viewing the work model from each direction can be obtained as shown in FIG. In the case of the work model shown in FIG. 12, since the height image viewed from the positive direction of the X axis is the same as the height image viewed from the negative direction of the X axis, one of them is omitted in step SC4. . Furthermore, since the height image viewed from the positive direction of the Y axis is the same as the height image viewed from the negative direction of the Y axis, one of them is omitted in step SC4. Therefore, through step SC4, there are four height images as shown in FIG.

  Thereafter, the process proceeds to step SC5, and the height image data remaining in step SC4 is stored. Information indicating whether the height image data to be stored is an image viewed from the top, bottom, left, or right of the CAD model, that is, information about the direction is added to the height image data, and the information about the direction and the height image are added. The data is stored in the storage unit 42 of the robot simulation apparatus 200 in association with the data. As a result, each height image data can be stored in a state in which the relationship information of each surface of the upper, lower, left and right sides is held, so that a plurality of height image data obtained by viewing one work model from different directions can be obtained. It is possible to register them in the robot simulation apparatus 200 or the image processing apparatus 300 in association with each other.

  Therefore, when the height image data is read from the storage unit 42, for example, height images viewed from six directions of the CAD model can be obtained in association with each other. As described above, a plurality of height image data obtained by viewing the work model from different directions can be registered as a search model. The search model is a model used for search processing for detecting the position and orientation of the workpiece WK stacked in the work space, and a plurality of types can be registered.

  In the search model registration flowchart shown in FIG. 9, the surface relationship information is stored in step SC5. However, the present invention is not limited to this. For example, as shown in the search model registration flowchart shown in FIG. Although steps SD1 to SD4 are the same as steps SC1 to SC4 in the flowchart shown in FIG. 9, in step SD5, only the height image data may be stored without storing the surface relationship information.

  The search model registration flowchart shown in FIG. 9 is the procedure when CAD data of the work WK exists, but if there is no CAD data of the work WK, there is no CAD data of the work WK shown in FIG. A plurality of height image data obtained by viewing the work model from different directions can be registered as a search model in accordance with a flowchart showing an example of a search model registration procedure. After the search model registration flowchart shown in FIG. 15 is started, in step SE1, the work WK with the surface to be registered facing upward is placed on the plane and three-dimensional measurement is performed. This three-dimensional measurement can be performed using the sensor unit 2 and the three-dimensional measurement unit 30 of the robot system 1000. The measurement data obtained by the three-dimensional measurement is output from the three-dimensional measurement unit 30, and based on the measurement data, the height image data of the surface that the workpiece WK wants to register can be obtained. An example of the height image thus obtained is shown in FIG.

  In step SE2 following step SE1 in the flowchart shown in FIG. 15, the height image obtained from the height image data obtained in step SE1 is registered as a search model. After registration, the process proceeds to step SE3, where it is determined whether or not registration is completed for the amount required for the search. This determination can be performed by the user, but may be performed by the robot simulation apparatus 200 or the image processing apparatus 300. That is, if the shapes of the workpiece WK viewed from the top, bottom, left, and right, front and rear directions are all different, it is preferable to perform step SE1 and step SE2 on all six surfaces. However, in the case of the above-described rectangle or the shape shown in FIG. If there is a surface having the same shape, all of the six surfaces need not be performed. When registration is completed for the amount required for the search in step SE3, the process ends.

  As described above, a search model registration step of registering a plurality of height image data obtained by viewing the work model from different directions as a search model can be performed. The search model registration step is performed by the search model registration unit 27 included in the robot setting device 100 shown in FIG.

  As shown in FIG. 17, feature points B and C can be arranged in the height image. The feature points B and C are points representing the shape of the search model, and a plurality of feature points B and C can be arranged in one height image. The feature points include an outer shape designation point B arranged along the peripheral edge of the search model and a surface shape designation point C arranged in the plane of the search model. The outer shape indicating point B is a point indicating the outer shape of each surface constituting the search model, and is arranged on the outer shape line of the surface with a predetermined interval. The surface shape designation points C are arranged at a predetermined interval on the inner side excluding the peripheral portion of each surface constituting the search model. By arranging the feature points, the approximate shape of the search model can be grasped using the feature points, and the amount of data to be handled can be reduced. It is preferable to change the colors of the outer shape indicating point B and the surface shape indicating point C. The outer shape indicating point B and the surface shape indicating point C are displayed on the display unit 3.

  Next, the registration procedure of the hand model of the robot RBT performed in step SB2 of the simulation environment setting procedure flowchart shown in FIG. 8 will be described. The procedure for registering the hand model of the robot RBT is shown in the hand model registration procedure flowchart of FIG. 18. In step SF1 after the start of this flowchart, polygon data (CAD data) of the hand unit (equal to the end effector EET) is stored. Read. In this step SF1, since the polygon data of the hand part is read, the shape of the hand part can be input. Accordingly, step SF1 is a hand portion shape input step for inputting the shape of the hand portion of the robot RBT. This hand portion shape input step is performed by the hand portion shape input portion 35 of the robot setting device 100 shown in FIG. Is called.

  In step SF2 following step SF1, the direction in which the cross section of the hand part is created is determined. If it demonstrates based on the hand model shown in FIG. 19, the axis | shaft A will be set to a hand model. The axis A passes through the center of the hand model and extends in the longitudinal direction of the hand model. A cross section in a direction perpendicular to the axis A is created. The number of cross sections to be created is not one, but a plurality of cross sections are created from the front end to the base end of the hand model. In this example, the case where five cross sections of cross section 1 to cross section 5 are created is shown, but the number of created cross sections can be set arbitrarily. The position for creating the cross section can be set to the vertex position of the polygon data. The created cross section is shown in FIG.

  In step SF3 of the flowchart shown in FIG. 18, a cross-section model is created using the cross-section created in step SF2. That is, the cross-sectional model of the hand unit can be created based on the cross-sectional shape and information on the cross-sectional position with respect to the axis A set in the hand model in step SF1. In addition, another method may be used for creating the cross-sectional model, for example, a method in which cross-sections are created at constant distances in the direction of the axis A and cross-sections having the same shape are combined.

  The shape of the hand portion of the robot RBT is various. In addition to the shape shown in FIG. 19, for example, there are a long claw portion shape, a short claw shape, a narrow claw width shape, a wide claw shape, a claw portion with a rod shape, etc. . These hand units can also be registered in accordance with the hand model registration procedure flowchart of FIG. That is, the hand part shape input unit 35 shown in FIG. 3 can input and register a plurality of hand part shapes of the robot RBT.

  After registering the hand model of the robot RBT through step SB2 of the simulation environment setting flowchart of FIG. 8 as described above, the process proceeds to step SB3. In step SB3, the surface of the work gripped by the hand portion of the robot RBT is selected. The surface of the workpiece can be represented by the height image registered in the search model registration flowchart shown in FIG. 9, and can be gripped by the hand unit of the robot RBT from a plurality of registered height images. The user selects a height image corresponding to the surface. When selecting a height image, the display control unit 26 can create a screen for selecting a height image and display it on the display unit 3. On the height image selection screen, a plurality of height images registered in the search model registration flowchart shown in FIG. 9 are displayed at a time, and the user selects from these height images by operating the operation unit 4. be able to.

  After selecting the height image in step SB3, the process proceeds to step SB4, and the position and orientation of the hand unit when the surface selected in step SB3 is gripped are registered. In step SB4, a gripping position setting screen (GUI) 50 as shown in FIG. The gripping position setting screen 50 can be generated by the gripping position setting unit 28 included in the robot setting device 100 shown in FIG. The generated gripping position setting screen 50 can be displayed as shown in FIG. 21 when the display control unit 26 controls the display unit 3.

  As shown in FIG. 21A, the gripping position setting screen 50 includes a model display area 51, a search model selection area 52, and a gripping position display area 53. In the model display area 51, the hand model of the robot RBT and the search model selected in the search model selection area 52 are displayed. A to D are displayed in the search model selection area 52, and a search model is assigned to each, for example, A is a height image of the CAD model viewed from above, and B is a CAD model left. The height image viewed from the right, the height image viewed from the right of the CAD model can be assigned to C, and the height image viewed from the front of the CAD model can be assigned to D. When the user selects an arbitrary alphabet in the search model selection area 52, the search model assigned to the alphabet is displayed in the model display area 51.

  In the grip position display area 53, a grip position addition button 53a is provided. When the grip position addition button 53a is operated, a grip posture setting area 54 is displayed together with the model display area 51 as shown in FIG. 21B. The grip position addition button 53a is a virtual button and can be operated with the mouse of the operation unit 4 or the like.

  In the gripping posture setting area 54, there is an input area 55 in which X-axis coordinates, Y-axis coordinates, Z-axis coordinates, rotation angles around the X axis, rotation angles around the Y axis, and rotation angles around the Z axis can be individually input. Is provided. When a numerical value is input to each input area 55, the hand model displayed in the model display area 51 moves so as to correspond to the input value. As a result, the hand model can be placed at a desired position, and while adjusting the position of the hand model, which part of the search model is gripped in what posture by the hand model, that is, the gripping position and posture are set. It becomes possible to do. The setting of the grip position and orientation can be performed by directly operating the hand model with the mouse of the operation unit 4 in addition to inputting numerical values in the input areas 55, for example.

  The gripping position set here is a gripping candidate position gripped by the robot RBT. As shown in FIG. 21A, a plurality of gripping candidate positions gripped by the robot RBT can be set in association with each search model registered in the search model registration unit 27. For example, two gripping candidate positions can be set in association with one search model, and four gripping candidate positions can be set in association with another search model. The set gripping candidate position can be registered in the storage unit 42 of the gripping position setting unit 28 of the robot setting device 100 in a state associated with the search model.

  In general, a plurality of gripping candidate positions are often registered for one workpiece. This is because, if a plurality of gripping candidate positions are registered, an optimal solution can be selected from a plurality of gripping solutions, so that the temporarily obtained gripping solution candidate may interfere with other objects of the hand unit, for example. This is because if there is another gripping solution candidate, the possibility that it is determined that gripping is possible increases. When a plurality of gripping candidate positions are registered, if registration is performed from the beginning each time, the number of steps is increased when registering similar gripping candidate positions, and the work becomes troublesome. Therefore, by copying the already registered gripping candidate position information, changing a part of the position parameters set at this gripping candidate position, and saving it as a new gripping candidate position, it is possible to save time and effort. A plurality of gripping candidate positions can be easily registered. Similarly, an existing gripping candidate position can be read out, and the position parameter can be appropriately corrected and overwritten and saved.

  FIG. 22 shows a state where the hand model is arranged at the gripping candidate position. FIG. 23 shows a state in which the hand models are arranged at the gripping candidate positions for each of the six search models, but the gripping candidate positions need not be set for all the search models.

  When registering the gripping candidate position, the relative position and orientation of the end effector model when gripping the work model are registered with respect to the origin of the search model. On the other hand, when picking the workpiece WK with the actual end effector EET, the robot controller 6 actually picks up the robot RBT from the vision coordinates which are the coordinates of the three-dimensional space (vision space) obtained by imaging the workpiece WK with the sensor unit 2. It is necessary to convert to robot coordinates used when operating.

Specifically, the position and posture of the work model are obtained by the position (X, Y, Z) and posture (R _X , R _Y , R _Z ) in the vision space (note that the posture (R _X , R _Y , R _Z ) shows a posture expressed by a ZY-X system Euler angle). Similarly, the posture of the end effector model that holds this is also obtained as the position (X, Y, Z) and posture (R _X , R _Y , R _Z ) of the robot setting device 100 in the virtual three-dimensional space. In order for the robot controller 6 to operate the robot RBT based on the position and orientation in such a vision space, these are determined based on the position (X ′, Y ′, Z ′) and orientation (R _X ′, R _Y ) in the robot space. ', R _Z '). The process of calculating a conversion formula for converting the position and orientation calculated in the displayed coordinate system into the coordinate system position and orientation in which the robot controller 6 operates the end effector EET is called calibration. This calibration can be performed by a conventionally known method.

  In step SB5 of the flowchart shown in FIG. 8, it is determined whether or not the necessary number of gripping candidate positions has been registered. If there are many parts that can be gripped by the hand unit, the number of gripping candidate positions to be registered increases. However, since this is a matter to be determined by the user, the determination in step SB5 is determined by the user. Become. If it is determined NO in step SB5 and the necessary number of gripping candidate positions cannot be registered and there are other gripping candidate positions to be registered, the process proceeds to step SB4 via step SB3. Set and register candidate positions. On the other hand, if it is determined as YES in step SB5 and a necessary number of gripping candidate positions have been registered, this flow ends.

  In the simulation environment setting step of step SA1 shown in FIG. 7, the search model can be set and registered, and the hand model can be set and registered as described above. Can be entered and set.

  In step SA2 following step SA1, bulk information is input. As the bulk information, for example, the number of workpieces to be stacked at one time, the shape of a location to be stacked (whether it is a plane or a box), the size of the location to be stacked, and the like are input.

  When bulk information is input in step SA2, the process proceeds to step SA3, and the three-dimensional CAD data (CAD model) of the workpiece WK is read. The data read in step SC1 of the search model registration flowchart of FIG. 9 can be read and used in step SA3.

  After reading the CAD data, the process proceeds to step SA4, where the number of polygons of the CAD data read in step SA3 is reduced. Reducing the number of polygons of CAD data means that the data capacity of the work model is compressed and reduced. It is also possible to compress the data capacity by thinning out the work model data.

  Three-dimensional CAD data is generally represented by polygon data expressed as a collection of triangular surface information. As a method for reducing the number of polygons while maintaining the shape of the work model as much as possible, for example, the progressive method proposed by Hoppe et al. There is a method called a mesh method or a QEM (Quadratic Error Metirc) method proposed by Garland et al. Any of these conventionally known methods may be used to reduce the number of polygons.

  For example, in the QEM method, the edges of adjacent polygons are deleted, two vertices are joined to generate new vertices, and this operation is repeated to greatly reduce the number of polygons constituting the entire CAD data. Data capacity can be compressed. Step SA4 only needs to be able to compress the data capacity of the CAD data read in step SA3. Therefore, the data capacity may be compressed by various methods other than the method of reducing the number of polygons.

  This step SA4 is to create a work model with a reduced number of polygons while keeping the shape of the work model represented by CAD data as much as possible, and to reduce the computation load during physical simulation by the physical simulation unit 22 described later. Step to perform. Step SA4 corresponds to a data capacity compression step for performing processing for compressing the data capacity of the work model set by the work model setting unit 20, and is performed by the data capacity compression unit 21 of the robot setting device 100 shown in FIG. .

  As a specific example of data capacity compression, FIG. 24A shows a work model before the number of polygons is reduced, and the number of polygons is 26,214. FIG. 24B shows the work model after the reduction of the number of polygons, and there are 414 polygons. FIG. 25 shows a work model after the number of polygons is reduced from that of the work model shown in FIG. 24B, and there are 50 polygons. Although the work model shown in FIG. 24B greatly reduces the number of polygons, the shape of the work model before the reduction of the number of polygons remains, and a reduction amount (data compression amount) of this level will be described later. This is preferable because the simulation result is highly reproducible.

  On the other hand, the work model shown in FIG. 25 has a shape that is far from the shape of the work model before the reduction of the number of polygons because the degree of reduction of the number of polygons is too large. This is not preferable because reproducibility is likely to be extremely low.

  In this embodiment, in order to prevent the shape of the work model from changing drastically due to excessive reduction of the number of polygons, the shape change is made to fall within a preset threshold range. In order to realize this, the data capacity compression unit 21 included in the robot setting device 100 shown in FIG. 3 is configured to be able to set the shape change allowable value between the work model before the compression process and the work model after the compression process. Yes. The user wants to reduce the number of polygons because the processing speed increases as the number of polygons in the CAD data used by the user is reduced. However, as described above, there is a possibility that the result of the simulation results in low reproducibility. There are limits to reducing the number of polygons. However, it is not easy for the user to determine how much the amount of polygon reduction should be reduced so that the reproducibility of the simulation results will be high. If so, the relevance between the simulation result and the picking moving image at the time of actual operation becomes low, and the meaning of performing the simulation is lost.

  In order to prevent this, the data capacity compression unit 21 has a preset initial setting value as a shape change allowable value that allows a change in the shape of the work model due to the reduction in the number of polygons. The initial set value can be stored in the storage unit 42 when the robot setting device 100 is shipped, or can be stored in the storage unit 42 when the use of the robot setting device 100 is started. The initial setting value can be changed after operation.

  For example, in the above-described QEM method, the amount of deformation when generating new vertices by reducing the edges in a plurality of adjacent faces and the vertices at both ends is used as a cost, and the edges are deleted in ascending order of the cost. . If the shape change allowance is suppressed to a small value on the order of 1%, for example, and the edges and vertices are deleted within the range where the accumulated deformation falls within the shape change allowance, it will be effective while preventing extreme shape change. Since the number of polygons can be reduced, the simulation result can be made highly reproducible while reducing the calculation load.

  For example, in the QEM method, for all edges (ridge lines) that make up polygon data, calculate the expected value of the deformation when deleting vertices at both ends and generating vertices at new positions (edge deletion) After sorting them in ascending order, the edges can be deleted in ascending order of expected deformation amount. When an edge is reduced, both ends of the edge move to a new vertex, and a new edge is generated between adjacent vertices. The expected value of the deformation amount of all edges generated in this order is added to the expected value of the deformation amount of the edge deleted by the previous operation, and the expected value of the deformation amount of all edges is again increased in ascending order. As a result, the edge newly generated by the operation of deleting a certain edge has a shape in which the deformation amount from the original shape up to that edge is accumulated, and is less likely to be reduced thereafter.

  It can also be configured such that the user can input the shape change allowable value. For example, the input screen for the shape change allowable value can be displayed on the display unit 3 by the display control unit 26, and the user can input the shape change allowable value using the operation unit 4. Similarly, the initial setting value can be changed.

  After compressing the data capacity of the work model, the process proceeds to step SA5 in the flowchart shown in FIG. 7 to execute a physical simulation. The physical simulation is performed by the physical simulation unit 22 included in the robot setting device 100 illustrated in FIG. 3. Specifically, the work is made of gravity using a plurality of work models in which the data capacity is compressed by the data capacity compression unit 21. The operation placed in the work space is simulated under the action, and a bulk state is generated by stacking a plurality of work models in the virtual work space. This is a physical simulation step. Performing physical simulations, such as excluding work models with a clearly unnatural posture, can generate bulk data according to the user's actual operating environment and perform the picking operation simulation described later, so actually prepare the work Therefore, it is possible to perform appropriate setting adjustment and verification in advance in a form close to actual operation. The physical simulation unit 22 may be configured to generate bulk data excluding a work model with a physically low probability of occurrence.

  The physical simulation can be executed using a conventionally well-known physical simulation engine. In the physical simulation, a work model is virtually arranged based on information on an action caused by a collision between works and information on an obstacle such as a floor. When the work WK is stored in the storage container BX as shown in FIG. 1 during actual operation, it is possible to set the physical simulation engine to perform a simulation in which the work model is input into the storage container BX from above. In the physical simulation, for example, when the workpieces collide with each other, it is shown how the workpiece model subsequently moves in consideration of a repulsion coefficient and a friction coefficient set in advance.

  As an example of the physical simulation, for example, a case in which a workpiece WK is dropped or thrown into the storage container BX from above to generate a bulk state inside the storage container BX can be given. “Throwing” and “throwing” are not limited to physically dropping from a high place. For example, in order to verify rolling, it is dropped while being pushed out from a predetermined height in the horizontal direction, or thrown in a parabolic shape. It is used in the sense that it includes dropping or placing the work model on the storage container BX or the floor surface or leaving it stationary. Furthermore, in the physical simulation, a work model can be sequentially input into the work space, and a plurality of work models can be simultaneously input, for example, an operation of dropping or standing can be simulated.

  After executing the physics simulation, the process proceeds to step SA6, and the bulk state data of the work model is generated based on the result of the physics simulation. Although this step is also a step included in the physical simulation step, the bulk data generation unit 23 included in the robot setting device 100 shown in FIG. 3 is executed.

  When an image is generated based on the bulk state data of the work model, the images shown in FIGS. 26 and 27 are obtained. FIGS. 26 and 27 show images displayed on the display unit 3. The left image in each drawing is a physical simulation image 57, and the right image is a height image 58 described later. The display control unit 26 generates an image indicating a state in which the work model is randomly inserted into the storage container BX based on the bulk state data of the work model, and displays the image on the display unit 3 as the physical simulation image 57. The work model in the physical simulation image 57 may be a work model before data capacity compression. In this case, by placing the work model before the compression processing by the data capacity compression unit 21 in accordance with the position and orientation information of the work model in the bulk state data of the work model, it is fine on the physical simulation image 57. Can be displayed as a work model. Instead of arranging the work model before the compression process, a work model having a data capacity larger than that of the work model used in the physical simulation unit 22 and close to the three-dimensional shape of the work may be arranged. In addition, when the user looks at the physical simulation image 57 and thinks that the result of the physical simulation is unsuitable for the picking operation simulation described later, the physical simulation result is discarded and the physical simulation is performed again. Can be executed.

  As shown in FIGS. 26 and 27, the physical simulation image 57 and the height image 58 can be made independent of the viewpoint, that is, the viewing angle. The arrangement of the physical simulation image 57 and the height image 58 is not limited to the arrangement shown in the figure, and can be arranged at arbitrary positions on one display screen. Further, the physical simulation image 57 may be displayed as a moving image from the start to the end of the physical simulation, for example. In this case, it can be seen that a plurality of work models are dropped onto the storage container BX from above the storage container BX, the work models collide with each other, and the work model collides with the wall of the storage container BX. The user can grasp how the bulk state is generated.

  After the work model bulk data is generated, the process proceeds to step SA7 shown in FIG. 7, where the work model in the work model bulk data is replaced with the CAD data before the polygon number reduction. That is, according to the position and orientation of each work model in the bulk data generated by the physics simulation unit 22, the data capacity is larger than that of the work model used by the physics simulation unit 22 and is close to the three-dimensional shape of the work. Arrange the work model. In this way, a data generation step for generating search processing data for executing search processing for detecting the position and orientation of each work model is performed. The search processing data is generated by the bulk data generation unit 23 of the robot setting device 100 shown in FIG.

  The work model having a larger data capacity than the work model used in the physical simulation unit 22 and close to the three-dimensional shape of the work WK has a low data compression amount and is more precise than the work model used in the physical simulation step. It is a work model with By replacing this work model, the reproducibility of the result of the picking operation simulation described later can be improved.

  The bulk data generation unit 23 arranges the work model before the compression processing by the data capacity compression unit 21 according to the position and orientation of each work model in the bulk data generated by the physical simulation unit 22. The search processing data can also be generated. By doing so, the result of the picking operation simulation described later becomes more reproducible.

  After the search processing data is generated in step SA7, the process proceeds to step SA8, and height image data representing the height of each part of the work model is generated. A height image obtained from height image data is an image having height information, and is also called a distance image, a three-dimensional image, or the like. Since the search processing data includes information on the height of the work model, the height image data can be obtained by a well-known method based on the information. As shown in FIGS. 26 and 27, the height image 58 that can be obtained based on the height image data generated in step SA8 is displayed on the display unit 3 simultaneously with the physical simulation image 57.

  After the height image is generated in step SA8 shown in FIG. 7, the process proceeds to step SA9, where search processing for detecting the position and orientation of each work model is performed on the search processing data generated by the bulk data generation unit 23. To do. This is a search processing step, which is performed by the search processing unit 24 included in the robot setting apparatus 100 shown in FIG.

Since the search process searches the search process data including the height information, it can be called a three-dimensional search. Specifically, a three-dimensional search for specifying the posture and position of each work model is performed using the search model registered by the search model registration unit 27. First, the position and orientation (X, Y, Z, R _X , R _Y , R _Z ) in which each feature point of the search model most closely matches is searched from the height image. Here, R _X , R _Y , and R _Z represent a rotation angle with respect to the X axis, a rotation angle with respect to the Y axis, and a rotation angle with respect to the Z axis, respectively. Various methods for expressing such a rotation angle have been proposed. Here, a ZYX Euler angle can be used. The matching position and orientation need not be one for each search model, and a plurality of matching positions and orientations may be detected.

  For example, as shown in FIG. 28, when a work model is detected based on the matching of feature points in the height image, the feature points of the search model are superimposed and displayed in a brighter color than the surroundings at the detection position. Can be made. As a search model used for 3D search in this way, the 3D search calculation process is simplified compared to the perspective view, etc., by using an image of the work as seen on each side, such as a hexagonal view. It is possible to obtain an advantage that the processing can be lightly loaded and speeded up. In addition, the displayed state is easy to see even when the search model is registered, and the user can easily understand it visually.

  Here, for example, a case where the work model constituting the height image used for the three-dimensional search is a work model compressed by the data capacity compression unit 21 will be described. FIG. 29 shows a case where the work model constituting the height image used for the three-dimensional search is before compression processing, while FIG. 30 shows the case where the work model constituting the height image used for the three-dimensional search is compressed. The case is after processing. Comparing FIG. 29B and FIG. 30B, FIG. 29B is in a state where the positions of the feature points are exactly matched, whereas FIG. 30B is in a state where the positions of the feature points are not exactly matched. It turns out that the score has fallen. Therefore, at the time of three-dimensional search, the accuracy of the three-dimensional search can be improved by using the work model before compression processing shown in FIG. 29B. Note that a search may be performed using the search model after the compression processing shown in FIG. 30B during the three-dimensional search, but it is preferable that the compression amount at that time is low.

  As shown in FIG. 31, each feature point of the search model can be superimposed and displayed at the position of the result of the three-dimensional search. The left side of FIG. 31 shows the case of two-dimensional display, and the right side shows the case of three-dimensional display. The search results are scored according to how many corresponding feature points exist in the height image (for example, the ratio of the number of feature points corresponding to the height image with an error of a certain distance or less). can do.

  As a result of the search processing unit 24 searching whether or not the search model registered by the search model registration unit 27 exists in the height image, if the search model does not exist in the height image, that is, a work model is detected. If it is not possible, NO is determined in step SA10 shown in FIG. 7 and the process proceeds to step SA11 to extract information on the remaining work model, and this flow ends.

  If the work model is detected in the height image as a result of the three-dimensional search in step SA9, YES is determined in step SA10, and the process proceeds to step SA12 to calculate interference determination and a gripping solution. Steps SA12 to SA15 are picking motion simulation steps performed by the picking motion simulation unit 25 that performs a simulation of the picking motion by the robot RBT on the work model successfully searched by the search processing unit 24. As a result of the three-dimensional search, a plurality of work models may be detected in the height image. The picking motion simulation unit 25 may have a physical simulation function and a height image generation function.

  As described above, the picking operation simulation unit 25 is a part for verifying the bulk picking operation of the work model in the virtual work space with respect to the generated bulk data. By performing a picking operation simulation, it is possible to perform in advance confirmation work and setting adjustments such as whether or not to detect the three-dimensional position and posture of the work in a form close to actual operation without actually preparing the work. it can.

  In particular, the physical simulation unit 22 grips one work model by the picking motion simulation unit 25 with a hand model, starts at least the extraction operation, and then performs physical simulation on the bulk data in the state after the work model is extracted. It can be configured to run again. The timing for executing the physical simulation again may be after the work model is held by the hand model and placed at the placement position, or may be in the middle of the work model take-out operation. That is, when a work model is held by a hand model and this work model is lifted, one work model is removed so that another work model around the work model can be moved. Therefore, there is no need to wait until the gripped work model is placed. Therefore, the re-execution timing of the physics simulation is the timing when one work model is gripped and lifted by the hand model, or after waiting until the other work model moves and stabilizes due to the action of gravity. It can be timing. For example, it may be after a lapse of a predetermined time (for example, after 10 seconds) from the time when the work model is gripped and lifted. After executing the physical simulation again, the bulk data is updated.

  Therefore, when one work model is taken out from the bulk data during the picking motion simulation, an operation such as the collapse of the work model group that is piled up according to this is also calculated by physical simulation, and the bulk data after this calculation is used. Since the picking motion simulation unit 25 executes the subsequent picking motion simulation, a more accurate bulk picking simulation is realized.

  In the interference determination, it is determined whether or not the three-dimensional point group indicated by the pixel data of each point of the height image interferes with the hand model input in advance by the hand unit shape input unit 35. The position where the end effector EET should be placed based on the position of this work model and the registered gripping posture of the work model for one work model detected by the three-dimensional search before the interference determination And calculate the posture. An image during the picking operation simulation is shown in FIG. Whether or not the end effector interferes with surrounding objects at the calculated position is determined using the cross-sectional model registered in the hand model registration procedure flowchart of FIG. In this interference determination, it is determined whether or not the three-dimensional point group interferes with the cross-sectional model using the cross-sectional model of the hand model. Specifically, as shown in FIG. 32A, when all three-dimensional points are separated from the cross-sectional model, it is determined that the three-dimensional point group, that is, the height image and the hand model do not interfere with each other, On the other hand, as shown in FIG. 32B, when there is even one three-dimensional point inside the cross-sectional model, or when one three-dimensional point is in contact with the edge of the cross-sectional model, the three-dimensional point group That is, it is determined that the height image and the hand model interfere with each other. If there is no interference, it means that the gripping solution has been calculated for this work model. In step SA13, the determination is YES and the process proceeds to step SA14. On the other hand, if the three-dimensional point cloud and the hand model interfere with each other, this means that a gripping solution cannot be calculated for this work model. In step SA13, NO is determined and the process proceeds to step SA11. This flow is finished.

  On the other hand, when the gripping solution can be calculated and the process proceeds to step SA14, the number of work models is reduced by one. The work model to be reduced in step SA14 is a work model for which a gripping solution can be calculated. After reducing the work model by one, the process proceeds to step SA15, and if there is another work model obtained by the three-dimensional search, the process proceeds to step SA12. If there is no other work model, NO is determined in step SA15, and the process proceeds to step SA11, and this flow is terminated.

  In this way, the presence / absence of a gripping solution that can grip the work model is determined for each detected work model. When a gripping solution is obtained for the work model, a simulation is performed in which the gripping candidate position of the work model is picked by the hand unit and conveyed to the outside of the storage container BX.

  The execution result of the picking operation simulation can be displayed on the display unit 3 as necessary. The user can determine whether the current setting is appropriate based on the execution result of the picking operation simulation displayed on the display unit 3. Various settings can be changed as necessary. For example, the installation position of the robot RBT, the position where the storage container BX is placed, the attitude of the sensor unit 2 and the like are adjusted. By re-inputting the changed setting as a simulation environment, executing a picking operation simulation again, and confirming its suitability, it is possible to determine appropriate setting conditions according to the user's environment.

  In addition, the physical simulation unit 22 can repeatedly perform the physical simulation, but is configured so that the work models can be randomly stacked so that the work models are not arranged in the same manner each time. Therefore, the physics simulation unit 22 cannot create the same state as the bulk stack obtained by the physics simulation performed previously. That is, since random bulk data is generated every time the physical simulation is performed, the result of the physical simulation is not reproducible.

  On the other hand, when the user reviews the picking operation environment settings and tries to confirm the effect, it is possible to reproduce the state that did not work in the previous picking operation simulation and check this bulk data. It is the most efficient. For example, when the gripping solution of the end effector model for a certain work model is not found and all the work models cannot be extracted, consider the case where the user increases the gripping candidate positions and gripping postures as a countermeasure. In this case, in order to determine whether the setting changed by the user was an effective solution to the problem that occurred last time, the state that did not go well last time or a state close to this was reproduced and confirmed. Is most effective.

  In addition, the picking operation simulation may be started not only from the state where the problem has occurred but also from the state where the problem has occurred. For example, if bulk data with a problem is configured by adding work models sequentially, whether or not it can be handled in a state where the number of work models is few several times before the stage where the problem occurred. It is possible to confirm.

  In order to realize these, in this embodiment, the bulk data in the state that the user wants to reuse, such as the initial state, the end state, or the intermediate state of the bulk data, is saved, and the environment is set again. Later, such stored bulk data can be recalled and verified. The bulk data can be stored in the storage unit 42 and can be called up by the user.

  In the above procedure, when a gripping solution is obtained with any of the work models, the processing for examining the gripping candidate position is terminated at that time, and the work model is gripped at the gripping candidate position corresponding to the obtained gripping solution. It is processed like so. However, the present invention is not limited to this method. For example, the user may determine which gripping candidate position is selected after obtaining all gripping candidate positions that can be gripped for one work model. In addition, paying attention to the height information where the work model is placed, the position where the work model is placed is high, in other words, among the stacked work models, the one located higher is the gripping candidate position Can also be selected.

  The picking operation simulation can be performed as described above, and the picking operation simulation is preferably performed a plurality of times. That is, it is determined whether or not each of the gripping candidate positions associated with the work model successfully searched by the search processing unit 24 can be gripped by the robot RBT, and the gripping candidate positions determined to be gripped are determined as the robot. The picking motion simulation unit 25 is configured to repeatedly execute a picking motion simulation held by the RBT. The physical simulation unit 22 is configured to be able to execute the simulation a plurality of times using a plurality of work models with compressed data capacity so that the picking operation simulation is performed a plurality of times. Upon completion, the physical simulation unit 22 automatically executes a new physical simulation, and a picking operation simulation is performed using the new physical simulation result.

  Further, the picking motion simulation unit 22 repeatedly executes a picking motion simulation in which each hand unit grips a work model successfully searched by the search processing unit 24 using the hand unit input by the hand unit shape input unit 35. It is configured as follows. When a plurality of hand part shapes are input according to the hand model registration procedure flowchart of FIG. 18, after performing a picking operation simulation using a hand model of one hand part, a hand model of another hand part The picking motion simulation is performed using After the picking operation simulation is completed for all the hand units, the picking operation simulation is performed again using the hand models of all the hand units. This can be repeated. Moreover, after performing the picking operation simulation a plurality of times using the hand model of one hand unit, the picking operation simulation can be performed a plurality of times using the hand model of the other hand unit. The picking operation simulation can be performed several hundred to 1000 times for each hand unit.

  As shown in the flowchart of FIG. 34, the picking operation simulation can be performed a plurality of times and the results can be accumulated. In step SG1 after the start of the flowchart of FIG. 34, a setting group to be a statistical target is acquired. This is a hand model of a plurality of hand units input by the hand unit shape input unit 35, a plurality of gripping candidate positions set in the search model, a plurality of search models, and the like.

  After acquiring the setting group to be a statistical target, the process proceeds to step SG2, and statistical accumulation is started. This can be initiated by a user instruction. When statistical accumulation is started, the process proceeds to step SG3, where one set of settings (one hand unit, one gripping candidate position, one search model) is acquired from the setting group acquired in step SG1, and the above-described picking is performed. Start motion simulation. The picking motion simulation unit 22 continues the picking motion simulation until it cannot be gripped by each hand unit.

  The picking operation simulation in step SG3 is executed a plurality of times. That is, the physical simulation unit 22 generates a bulk stack state in which a plurality of work models are stacked, obtains a height image based on this, performs a three-dimensional search, executes a picking motion simulation, and then performs a physical simulation. The unit 22 newly generates a bulk state, obtains a height image based on this, performs a three-dimensional search, and then executes a picking operation simulation. That is, each time the picking motion simulation by the picking motion simulation unit 25 is completed, a physical simulation is newly executed to update the data. This is repeated, and the results of the picking operation simulation for each height image are accumulated in step SG4.

  If the picking operation simulation is performed for the preset number of times, it is determined as YES in step SG5, and the process proceeds to step SG6. On the other hand, if the preset number is not reached, it is determined as NO in step SG5 Proceeding to step SG3, a picking operation simulation is executed for another height image.

  Thereafter, in step SG6, execution results for each picking operation simulation are accumulated. In step SG7, if there is a setting for which the picking operation simulation is not executed in the setting group acquired in step SG1, it is determined as NO, and the picking operation simulation is repeated for the setting for which the picking operation simulation has not been executed yet. Run. When the picking operation simulation is executed for all of the setting groups acquired in step SG1, it is determined as YES in step SG7 and the process is ended.

  FIG. 35 is a table in which picking operation simulation results are listed in the order of grip registration numbers. In the left column of this table, search models are described. In this example, there are three search models A to C. In the middle column in the left-right direction of the table, the gripping candidate positions are listed in the order of registration numbers. Three gripping candidate positions are registered in search model A, and five gripping candidate positions are registered in search model B. In this example, four gripping candidate positions are registered in the search model C. The number of times in the right column of the table is the number of times of successful gripping in the picking motion simulation, and is information indicating the frequency (usage frequency) used when gripping is successful in the picking motion simulation.

  The table shown in FIG. 35 is a table generated for the display control unit 26 to display on the display unit 3, and each search model A to C and each gripping candidate position (A−) when gripping is successful in the picking motion simulation. 001 to C-005), the information indicating the usage frequency is displayed in a comparable manner. Further, without displaying both the search models A to C and the gripping candidate positions (A-001 to C-005), only the search models A to C or the gripping candidate positions (A-001 to C- 005) only may be displayed. In this table, each search model not used for gripping and each gripping candidate position are also displayed, but only each search model that was not used for gripping or only the gripping candidate position that was not used for gripping can be displayed. May be. A gripping candidate position that has not been used for gripping is a gripping candidate position whose number of times is zero.

  FIG. 36 shows a case where the results shown in FIG. 35 are listed in descending order, and this table is also generated by the display control unit 26 for display on the display unit 3. Of course, it is possible to display the list in ascending order. The grip registration position A-002, the grip registration position B-002, and the grip registration position B-003 are not adopted as grip candidate positions at the time of the picking operation simulation, so there is a problem in actual bulk picking work even if they are deleted from the data I can say no. Further, even if the search model B having a significantly smaller number of grips than the search model A and the search model C is deleted from the data, it can be said that there is almost no problem in the actual bulk picking operation. Moreover, since it is not necessary to consider the result data using these settings from the robot control program side, the number of man-hours for setup can be reduced.

  FIG. 37 is a table in which the results of the picking operation simulation are displayed in a list according to the shape of the hand unit. This table is also generated by the display control unit 26 for display on the display unit 3. The figure shows the case where there are seven types of hand shapes from “hand shape 1” to “hand shape 7”. The number of remaining workpieces (average) is the number of workpieces (storage containers) that could not be gripped in one picking operation simulation. This is a value obtained by dividing the total number obtained by multiplying the number of workpieces remaining in BX by the number of picking operation simulations by the number of picking operation simulations. The number of remaining workpieces (maximum) is the number of workpieces in which the number of workpieces remaining in the entire picking motion simulation is the largest. The number of remaining workpieces (minimum) is the number of workpieces in which the number of workpieces remaining in the entire picking motion simulation is the smallest.

That is, since the display control unit 26 is configured to be able to display the number of work models remaining at the end of the picking motion simulation by the picking motion simulation unit 25 for each hand unit, the remaining number of workpieces for each shape of the hand unit. Can be compared. The remaining number of workpieces decreases when the success of gripping the work model by the hand unit is high, and conversely increases when the success of gripping the work model by the hand unit is low. This is information indicating the degree. Therefore, in this embodiment, information indicating the degree of success of gripping the work model by each hand unit obtained by the picking operation simulation can be displayed in a comparable manner. The degree of success of gripping the workpiece model can be displayed, for example, by the ratio of the workpieces that have been successfully gripped to the total workpieces. The step of displaying each table shown in FIGS. 35 to 37 on the display unit 3 is a display step.
(Exposure time setting)
As shown in FIG. 3, the shape measuring apparatus 400 includes an exposure control unit 40 for changing the exposure time (shutter speed) of the cameras CME1, CME2, CME3, and CME4 stepwise. The exposure control unit 40 is configured to increase the exposure time at an equal ratio, but is not limited to this. In this embodiment, the exposure time is increased by 1.5 times, but this magnification can be arbitrarily set.

The shape measuring apparatus 400 includes a height image data generation unit 33. The height image data generation unit 33 obtains the received light amount output from the cameras CME1, CME2, CME3, and CME4 at each exposure time changed in stages by the exposure control unit 40, and based on the obtained received light amount. Each pixel value is configured to generate height image data indicating the height of each part of the surface of the measurement object (the surface of the workpiece WK). This can be generated by a technique such as a phase shift method.
(Configuration of pixel determination unit)
The shape measuring apparatus 400 includes a pixel determining unit 34. The pixel determination unit 34 obtains the received light amount received by the light receiving unit during each exposure time changed in steps by the exposure control unit 40, and is generated by the height image data generation unit 33 based on the obtained received light amount. It is configured to determine whether each pixel of each height image is valid or invalid. When the projector PRJ is configured to emit pattern light having a periodic illuminance distribution, the pixel determination unit 34 has a contrast of the light amount of each pixel obtained by imaging with the cameras CME1, CME2, CME3, and CME4. A pixel that is equal to or greater than the threshold is determined to be valid. In addition, when using the light cutting method as a method for acquiring a three-dimensional shape, if the light amount of each pixel is too small or saturated, it is determined that the pixel is invalid, and other pixels Is determined to be valid. In other words, it is determined that pixels whose height cannot be displayed when a height image is generated (pixels that appear to be crushed black or pixels that fly white on the screen) are invalid. It is determined that a pixel that can display the size is valid.

  The pixel determination unit 34 can be configured to determine whether each pixel in the attention area set in the imaging range by the cameras CME1, CME2, CME3, and CME4 is valid or invalid. FIG. 39 shows an attention area setting screen 62 for allowing the user to set the attention area. The attention area setting screen 62 is a so-called user interface, which is generated by the display control unit 26 and displayed on the display unit 3. An image 60 displayed in the attention area setting screen 62 in FIG. 39 is a height image of a range imaged by the cameras CME1, CME2, CME3, and CME4, and the storage container BX and the work are included in the imaging range. WK is included. Since the storage container BX is not a picking target, it is meaningless to determine whether the pixels constituting the storage container BX are valid or invalid. In FIG. 39, a frame line 61 for setting the attention area is displayed in the height image 60 of the attention area setting screen 62, and the user sets the area surrounded by the frame line 61 as the attention area. Can do. The frame line 61 can be enlarged, reduced, or deformed by operating the operation unit 41, and the user can designate an arbitrary area as the attention area. When the attention area is set, it is determined whether each pixel in the area is valid or invalid, so that pixels in unnecessary areas are not used for setting the exposure time described later, and the exposure time is set to an appropriate time. It becomes possible to set.

42, 43, and 44 show one form of the attention area setting screen. In each figure, the attention area setting frame line 61 is shown in white. The display form of the frame line 61 is not limited to the form shown in each drawing. It is also possible to distinguish the attention area by changing the background color between the outside of the attention area and the inside of the attention area.
(Configuration of exposure time setting section)
The shape measuring apparatus 400 includes an exposure time setting unit 41. The exposure time setting unit 41 sets an exposure time corresponding to a height image in which the number of pixels determined to be valid by the pixel determination unit 34 satisfies a predetermined condition as a set value. The set value is an exposure time used when the robot system 1000 is actually operated. The exposure time can be adjusted manually, but in the case of manual operation, the process of searching for the optimal exposure time range while viewing the height image obtained by the user with the exposure time set by the user. I need it. In addition, when a search for narrowing down the optimal exposure time is performed manually, it is difficult to obtain an appropriate exposure time in various cases because of its personality and poor reproducibility. Furthermore, in the case of pixels constituting a height image, it is difficult to determine whether or not the pixel is an effective pixel, and it is particularly difficult to determine whether or not the exposure time determined manually is a reasonable time. It is difficult to verify and judge. Therefore, in this embodiment, an appropriate exposure time can be automatically set.

  FIG. 40 is a graph showing the relationship between the exposure time and the number of effective pixels, where the horizontal axis is a logarithmic scale and the exposure time (ms), and the vertical axis is the number of effective pixels (pixel). This graph captures a fixed workpiece while changing the exposure time of the cameras CME1, CME2, CME3, and CME4 in steps to obtain a height image, and exposes the number of effective pixels included in the height image. Plotted by time. The relationship between the number of effective pixels and exposure time is roughly shown in this graph. Within a certain exposure time range, even if the exposure time changes, the number of effective pixels hardly changes. The number of pixels decreases rapidly. In the example of this graph, a height image with a large number of effective pixels can be obtained if the exposure time is in the range of 0.1 ms to 6 ms. However, if the exposure time is less than 0.1 ms and exceeds 6 ms, the effective pixels The number of images becomes extremely small. When the number of effective pixels is extremely reduced, black portions increase in the height image of FIG. 39, and the accuracy of the three-dimensional search is reduced.

  In this embodiment, as shown in detail in FIG. 41, an exposure time at which the effective pixel is 95% of the maximum value is set as a set value. That is, the exposure time setting unit 41 is configured to set, as a set value, a predetermined time that is equal to or less than the exposure time in which the number of pixels determined to be valid by the pixel determination unit 34 is the largest. The reason is that the exposure time at which the number of effective pixels takes the maximum value often exists in a portion near the flat portion of the graph shown in FIG. 41, and the exposure time selected as the setting value tends to fluctuate in the time axis direction. Since it is likely to become unstable, not the exposure time at which the number of effective pixels takes the maximum value, but the exposure time at which the number of effective pixels is slightly smaller than the maximum value is set as the set value. In this embodiment, the exposure time at which the effective pixel is 95% of the maximum value is set as the set value. However, the present invention is not limited to this, and the effective pixel has a range of 90% to 99% of the maximum value. The exposure time can be set as a set value.

  In this embodiment, the user can adjust the exposure time automatically set by the exposure time setting unit 41. FIG. 42 shows an exposure time setting screen 65 in which the exposure time is at the recommended value. On the exposure time setting screen 65, a height image display region 65a generated by the height image generation unit 33, a graph display region 65b for displaying a graph indicating the relationship between the exposure time and the number of effective pixels, and a user A setting area 65c that can be set is provided.

  An example of a height image generation method in the height image generation unit 33 will be described. A height image is an image in which a height value is stored in each pixel value constituting the image. Typically, it is an image in which the height is expressed by a luminance value. As a method for generating a height image, the Z buffer method can be suitably used.

  That is, a two-dimensional height image is drawn by rendering that removes shadows based on three-dimensional bulk data composed of a work model, which is CAD data of a work. As a rendering method, a Z buffer method, a Z sort method, a scan line method, a ray tracing method, or the like can be used. Here, a plurality of work models are drawn on a height image using the Z buffer method. In the Z buffer method, it is realized as Z buffer = height image. In addition, the Z buffer is not initialized at infinity, but is initialized to the minimum with information on the floor and the box. In the process of the Z-buffer method, a normal image that holds normal information of the surface of the work model for each pixel is also created at the same time. Here, the normal line data is also updated for the pixel in which the Z buffer is overwritten. In the present specification, the Z buffer method is exemplified as a conversion method to a height image, but the present invention is not limited to the Z buffer method, and other methods can be appropriately used as described above. In this way, the height image is redrawn from the three-dimensional bulk data.

  The graph displayed in the graph display area 65b can be generated by the display control unit 26 and is the same as the graph shown in FIG. 40 or FIG. The horizontal axis of this graph, that is, the exposure time axis, is displayed logarithmically (equal ratio), but is not limited to this display form and can be arbitrarily displayed.

  In the setting area 65c, a threshold value setting unit 65d for setting an adjustment threshold value (%), a range setting unit 65e for setting an adjustment range for exposure time, a ratio setting unit 65f, and a set value for exposure time (recommended) And an exposure time display area 65g for displaying (value).

  The threshold value setting unit 65d is a part for setting the exposure time at which the effective pixel is the number of pixels of the maximum value as a set value or setting the value as a percentage. Can be set in 1% increments. The range setting unit 65e can set the maximum value (maximum exposure time) and the minimum value (minimum exposure time) of the exposure time adjustment range separately. The ratio setting unit 65f is a part for setting a ratio for increasing the exposure time by an equal ratio.

  The set value of the exposure time can be displayed as a white vertical line 65h in the graph displayed in the graph display area 65b. In addition, a value obtained by multiplying the value set by the threshold setting unit 65d by the maximum value of the effective pixel can be displayed as a white horizontal line 65i in the graph displayed in the graph display area 65b.

  The exposure time can be adjusted by the user inputting the exposure time displayed in the exposure time display area 65g by operating the operation unit 4. FIG. 43 shows a case where the exposure time is shorter than the recommended value, and FIG. 44 shows a case where the exposure time is longer than the recommended value. As shown in FIGS. 43 and 44, when the user changes the exposure time to a time deviating from the set value, the height image captured at the changed exposure time is displayed in the height image display area 65a. Can do. The height images shown in FIG. 43 and FIG. 43 are wider than the height image in the case where the exposure time shown in FIG. 42 is at the recommended value, and the exposure time is inappropriate. .

  Further, the height image captured during the exposure time shown in FIG. 42 and the height image captured during the exposure time shown in FIGS. 43 and 44 may be displayed on the display unit 3 at the same time. That is, the display control unit 26 is configured so that a height image corresponding to the first exposure time and a height image corresponding to a second exposure time different from the first exposure time can be displayed simultaneously. be able to.

Although it is desirable to set the exposure time before starting the actual operation, the exposure time can also be set for the purpose of adjusting the exposure time after the start of the actual operation.
(3D measurement unit)
The three-dimensional measuring unit 30 can be provided in the image processing apparatus 300. The three-dimensional measuring unit 30 is configured to three-dimensionally measure a work space in which a plurality of workpieces are stacked during actual operation and output the measurement data. The sensor unit 2 can be a component of the three-dimensional measurement unit 30. The measurement method is as described above, and is not particularly limited. The three-dimensional measuring unit 30 may be provided in the sensor unit 2 and integrated.
(Search processing part)
The search processing unit 24 is used in the above-described picking operation simulation, but can be used even during actual operation. The search processing unit 24 can be configured separately from that used in the picking operation simulation.

  That is, the search processing unit 24 uses the plurality of search models registered in the search model registration unit 27 to detect the positions and orientations of the plurality of workpieces included in the measurement data output from the three-dimensional measurement unit 30. Perform search processing. When performing the search process, a height image is generated by the height image generation unit 33 based on the measurement data output from the three-dimensional measurement unit 30, and is registered in the height image by the search model registration unit 27. Search whether there is a new search model. This method can be the same as the method performed in the picking motion simulation.

  Further, the search processing unit 24 can detect the positions and orientations of a plurality of workpieces included in the measurement data output from the three-dimensional measurement unit 30 using a plurality of feature points representing the shape of the search model. it can. That is, a height image is generated based on the measurement data output from the three-dimensional measurement unit 30, and a shape that matches a plurality of feature points representing the shape of the search model is searched in the height image.

In addition, the search processing unit 24 uses the associated height image data registered in the search model registration unit 27, and positions of a plurality of workpieces included in the measurement data output from the three-dimensional measurement unit 30. And a posture can be detected.
(Interference judgment unit)
The interference determination unit 31 can be provided in the image processing apparatus 300. The interference determination unit 31 determines whether or not the gripping candidate position set on the workpiece surface associated with the search model that has been successfully searched by the search processing unit 24 can be gripped without the robot RBT interfering with surrounding objects. . The determination method by the interference determination unit 31 can be the same as the method performed in the picking operation simulation.

  The interference determination unit 31 is configured to estimate a workpiece surface that cannot be measured by the three-dimensional measurement unit 30 based on a search model that has already been successfully searched. FIG. 45A shows a point group in which a height image generated based on the measurement data obtained by the three-dimensional measuring unit 30 is displayed in a three-dimensional manner from a slightly oblique direction and a feature point of the search model superimposed on the position of the search result. In this figure, a surface that is nearly vertical is missing from the measurement data, and the bottom of the back is visible, or there are many blackened portions. The black-crushed part in the height image is a blind spot due to the blind spot of the camera CME or the projection of the projector PRJ, a highly glossy metal workpiece, or the like due to multiple reflections with surrounding objects. The data is invalid. If there are many black-crushed portions in the height image in this way, there is a problem that the three-dimensional search cannot be performed sufficiently, or it is erroneously determined that there is no interference even though it actually interferes.

45B, on the other hand, in addition to FIG. 45A, the workpiece surface that cannot be measured by the three-dimensional measuring unit 30 is estimated using the relevance with other surfaces, and the feature points of the estimated search model of another surface are also shown. The figure superimposed and displayed is shown. That is, in the search model registration unit 27, a plurality of height image data obtained by viewing the work model from different directions are registered as search models, and each height image is stored in a state having the relationship information thereof. For example, if the search model that saw the work from above is successfully searched in actual operation, the search model viewed from the left and the search model viewed from above should be arranged where and how the work will be shaped Can be used to compensate for missing portions of measurement data. Therefore, in FIG. 45B, a portion that cannot be three-dimensionally measured, such as a surface that is nearer vertical than that in FIG. 45A, is complemented with the feature points of the search model. In addition to the interference determination for the three-dimensional point group described in the explanation of the interference determination in the physical simulation, the interference determination for the feature points of the search model including other surfaces estimated based on the search result is also performed. In this way, it is possible to realize interference determination that also considers a portion that cannot be measured in three dimensions.
(Gripping position determination unit)
The grip position determination unit 32 can be provided in the image processing apparatus 300. The gripping position determination unit 32 is configured to determine a gripping candidate position determined by the interference determination unit 31 as capable of gripping without interfering with surrounding objects as an actual gripping position.

(Procedure for actual operation)
Next, the procedure during actual operation will be described based on the flowchart shown in FIG. In step SH1 after the start, as shown in FIG. 1, the sensor W2 is used to three-dimensionally measure the workpiece WK stacked in the storage container BX. In step SH <b> 1, a height image is generated by the height image generation unit 33 based on the measurement data output from the three-dimensional measurement unit 30.

  Thereafter, in step SH2, the above-described three-dimensional search is performed on the height image generated in step SH1, and the position and posture of the workpiece are detected. In the subsequent step SH3, the position and posture where the hand unit should be arranged are calculated based on the position of the work model and the gripping posture of the registered work model.

  In step SH4, it is determined using the cross-sectional model registered in the hand model registration procedure flowchart of FIG. 18 whether or not the hand unit does not interfere with surrounding objects at the calculated position. Specifically, whether or not the 3D point group indicated by the pixel data of each point of the height image interferes with the hand model input in advance by the hand unit shape input unit 35 is determined. Judgment is made using a cross-sectional model. If it is determined as NO in step SH5 and the hand unit does not interfere with surrounding objects, the process is terminated as having a gripping solution. This gripping solution is transmitted to the robot controller 6, and the robot controller 6 controls the arm part ARM and the hand part to pick and convey the work WK.

  If it is determined as YES in step SH5 and the hand unit interferes with surrounding objects, the process proceeds to step SH6, and it is determined whether there is another registered gripping candidate position. If YES in step SH6 and there is another gripping candidate position, the process returns to step SH3. If it is determined as NO in step SH6 and there is no other gripping candidate position, the process proceeds to step SH7. In step SH7, it is determined whether there are other workpieces detected in step SH2. If it is determined as YES in step SH7 and there is another workpiece, the process returns to step SH3. If it is determined NO in step SH7 and there is no other workpiece, the process ends with no gripping solution. As described above, when a plurality of workpieces are detected by the three-dimensional search, it is determined whether or not the detected workpieces can be gripped one by one.

  The flowchart shown in FIG. 46 shows another procedure 1 during operation. Steps SI1 to SI5 in the flowchart shown in FIG. 46 are the same as steps SH1 to SH5 in the flowchart shown in FIG. In step SI6 of the flowchart shown in FIG. 46, it is determined whether or not the hand unit does not interfere with other detected workpieces at the same position and posture as the result calculated in step SI3. Specifically, the cross-sectional model of the hand is used to determine whether or not the feature points of the search model determined from the position and orientation of each search result interfere with the hand model input in advance by the hand unit shape input unit 35. Judgment. Further, using the relationship between the surfaces, it is determined whether or not interference is similarly caused with a feature point group of a search model of another surface estimated from each search result.

  If it is determined NO in step SI7, it is determined that there is a gripping solution. On the other hand, if YES is determined in step SI7, the process proceeds to step SI8. Step SI8 and step SI9 are the same as step SH6 and step SH7 of the flowchart shown in FIG.

  FIG. 47A shows a case where interference determination is performed only on a three-dimensional point group obtained by actually performing three-dimensional measurement without using the relevance of the surface, and the right side of the workpiece WKA to be grasped. It can be seen that the shape of the side surface of the workpiece WKB that is close to is missing in the measurement data. On the other hand, FIG. 47B shows a case where interference is determined for a feature point of a search result of another workpiece or a feature point in a search model of another surface to be estimated using the relevance of the surface. Since the work WKB has succeeded in searching the search model viewed from above, and the shape of the side surface of the work WKB is estimated using the relationship with the search model viewed from above, the side of the work WKB is also Feature points are displayed. Thus, by utilizing the relevance of the surface, it is possible to estimate and compensate for the missing portion in the measurement data.

  In the case of FIG. 47A, the shape of the side surface of the workpiece WKB is missing. Therefore, when trying to grip the workpiece WKA to be gripped by the hand portion, the tip of the hand portion actually interferes with the side surface of the workpiece WKB. However, it is erroneously determined that there is no interference. On the other hand, in the case of FIG. 47B, the shape of the side surface of the work WKB is estimated based on the relationship information of the surface, in addition to the interference determination with respect to the three-dimensional point group indicated by the pixel data of each point of the height image. Since the interference determination is performed including the estimated feature points of the search model, it is determined that the tip of the hand part interferes with the side surface of the work WKB when trying to grip the work WKA to be gripped by the hand part. A determination result can be obtained.

  As shown in FIG. 48, the interference between the workpiece and the hand unit can be determined for each gripping candidate position. As for the cause of the interference, on the screen shown in FIG. 48, for example, when the feature point and the hand part interfere with each other, “point cloud interference” is displayed, and the inclination angle of the hand part falls outside the specified range. Is displayed as “inclination angle”.

Also, as shown in FIG. 49, when cube-shaped workpieces are stacked in bulk, the shape of all six surfaces of the workpiece is the same, so only one surface is registered as a search model and that surface is gripped from directly above. Only one gripping posture to be registered is registered. As a result, searching and gripping can be performed on all six surfaces.
(Effect of embodiment)
According to this embodiment, a plurality of work models with compressed data capacities are used during physical simulation, and according to the position and orientation of each work model in the bulk data generated by physical simulation during picking motion simulation. Since the search processing data in which the work model having a large data capacity is arranged is used, it is possible to execute the bulk picking simulation with high reproducibility while reducing the time required for the physical simulation.

  Further, it is possible to register a plurality of search models and set a plurality of gripping candidate positions in association with the search models for each search model. A search process that detects the position and orientation of the work model is executed using the registered search model, and the robot can grip each of a plurality of gripping candidate positions associated with the work model that has been successfully searched. In some cases, a picking operation for gripping the gripping candidate position by the robot is simulated. Information indicating the use frequency of each search model and / or each gripping candidate position when the gripping is successful by repeating this simulation can be displayed in a comparable manner. Therefore, it is possible to easily compare a gripping position with a low contribution rate to the workpiece removal success rate with other gripping positions, and to select a gripping position based on objective data.

  Also, a plurality of hand part shapes of the robot RBT are input by the hand part shape input part 35, and a picking operation simulation is executed using each of the input hand parts to indicate the degree of success of gripping the work model by each hand part. Since information can be displayed in a comparable manner, the shape of the hand part, which has a low contribution rate to improving the success rate of workpiece removal, can be easily compared with the shape of other hand parts. Can be selected.

  Further, the workpiece surface that cannot be measured by the three-dimensional measuring unit 30 is estimated based on a search model that has already been successfully searched, and the robot RBT including the estimated workpiece surface does not interfere with surrounding objects. Determine whether gripping is possible. This makes it possible to estimate the shape of the workpiece where the height data is missing, using the part where the height data exists, so even if there is a missing piece in the height data The presence or absence of interference between surrounding objects and the robot RBT can be accurately determined.

  Further, when setting the exposure time, the amount of received light can be obtained at each exposure time changed in stages by the exposure control unit 40. Based on the amount of light received during each exposure time, it is determined whether each pixel that generates height image data is valid or invalid. Since the exposure time corresponding to a height image in which the number of pixels judged to be valid satisfies a predetermined condition is automatically set, height image data with a large number of effective pixels can be obtained without adjusting the exposure time manually. Can be obtained. Thereby, the personality is lost, the reproducibility is improved, and an appropriate exposure time can be obtained in various cases.

  The above-described embodiment is merely an example in all respects and should not be interpreted in a limited manner. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

  As described above, the present invention can be applied to, for example, a robot system that picks workpieces stacked in bulk.

2 Sensor unit 3 Display unit 4 Operation unit 6 Robot controller 7 Robot operation tool 20 Work model setting unit 21 Data capacity compression unit 22 Physical simulation unit 23 Bulk data generation unit 24 Search processing unit 25 Picking motion simulation unit 26 Display control unit 27 Search Model registration unit 28 Gripping position setting unit 30 Three-dimensional measurement unit 31 Interference determination unit 32 Gripping position determination unit 33 Height image generation unit 34 Pixel determination unit 35 Hand shape input unit 40 Exposure control unit 41 Exposure time setting unit 42 Storage unit 100 Robot setting device 200 Robot simulation device 300 Image processing device 400 Shape measuring device 1000 Robot system BX Storage container CME1-4 Camera (light receiving unit)
PRJ projector (projecting part)
RBT Robot WK Work

Claims (7)

A robot simulation device for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially taken out by a robot,
A work model setting unit for setting a work model obtained by modeling the three-dimensional shape of the work;
A bulk data generation unit for generating data in a bulk state in which a plurality of work models set in the work model setting unit are stacked in a virtual work space;
A search model registration unit for registering a plurality of search models used for search processing for detecting the position and orientation of a work model for the bulk data generated by the bulk data generation unit,
A gripping position setting unit capable of setting a plurality of gripping candidate positions gripped by the robot in association with each search model registered by the search model registration unit;
A search processing unit that executes a search process for detecting the position and orientation of the work model using the search model registered by the search model registration unit from the bulk data generated by the bulk data generation unit;
For each of the gripping candidate positions associated with the work model that has been successfully searched by the search processing unit, it is determined whether or not gripping by the robot is possible, and the gripping candidate positions determined to be gripping are gripped by the robot. A picking motion simulation unit that repeatedly executes a picking motion simulation;
A robot simulation apparatus, comprising: a display control unit that displays information indicating the use frequency of each search model and / or each gripping candidate position when gripping is successful in the picking motion simulation. A robot simulation device for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially taken out by a robot,
A work model setting unit for setting a work model obtained by modeling the three-dimensional shape of the work;
A bulk data generation unit for generating data in a bulk state in which a plurality of work models set in the work model setting unit are stacked in a virtual work space;
A search model registration unit for registering a plurality of search models used for search processing for detecting the position and orientation of a work model for the bulk data generated by the bulk data generation unit,
A hand part shape input unit capable of inputting a plurality of shapes of the robot hand part;
A search processing unit that executes a search process for detecting the position and orientation of the work model using the search model registered by the search model registration unit from the bulk data generated by the bulk data generation unit;
A picking operation simulation unit that repeatedly executes a picking operation simulation by using the hand unit input by the hand unit shape input unit by using the hand unit shape input unit, and a hand model that has been successfully searched by the search processing unit;
A robot simulation apparatus comprising: a display control unit configured to display, in a comparable manner, information indicating a degree of success of gripping a work model by each hand unit obtained by the picking operation simulation. The robot simulation apparatus according to claim 1 or 2,
The bulk data generation unit is configured to execute a physical simulation for simulating an operation in which a workpiece is placed in the work space under the action of gravity using a plurality of workpiece models, and the picking operation simulation unit A robot simulation apparatus characterized in that each time a picking motion simulation is completed, the physical simulation is newly executed to update data. The robot simulation apparatus according to claim 1,
The robot controller according to claim 1, wherein the display control unit is configured to display each search model and / or each gripping candidate position that is not used for gripping. The robot simulation apparatus according to claim 2,
The picking motion simulation unit is configured to continue the picking motion simulation until it cannot be gripped by each hand unit,
The robot control apparatus according to claim 1, wherein the display control unit is configured to be able to display the number of work models remaining at the end of the picking motion simulation by the picking motion simulation unit for each hand unit. A robot simulation method for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially taken out by a robot,
A work model setting step for setting a work model obtained by modeling the three-dimensional shape of the work;
A bulk data generation step for generating data in a bulk state in which a plurality of work models set in the work model setting step are stacked in a virtual work space;
A search model registration step for registering a plurality of search models used for search processing for detecting the position and orientation of the work model for the bulk data generated in the bulk data generation step,
A gripping position setting step for setting a plurality of gripping candidate positions gripped by the robot in association with each search model registered in the search model registration step;
A search processing step for executing a search process for detecting the position and orientation of the work model from the bulk data generated in the bulk data generation step, using the search model registered in the search model registration step;
For each of the gripping candidate positions associated with the work model that has been successfully searched in the search processing step, it is determined whether or not gripping by the robot is possible, and the gripping candidate positions determined to be gripping are gripped by the robot. A picking motion simulation step for repeatedly executing the picking motion simulation;
A robot simulation method comprising: a display step for displaying, in a comparable manner, information indicating the frequency of use of each search model and / or each gripping candidate position when gripping is successful in the picking motion simulation. A robot simulation method for simulating a bulk picking operation in which a plurality of workpieces stacked in a work space are sequentially taken out by a robot,
A work model setting step for setting a work model obtained by modeling the three-dimensional shape of the work;
A bulk data generation step for generating data in a bulk state in which a plurality of work models set in the work model setting step are stacked in a virtual work space;
A search model registration step for registering a plurality of search models used for search processing for detecting the position and orientation of the work model for the bulk data generated in the bulk data generation step,
A hand portion shape input step for inputting a plurality of shapes of the hand portion of the robot;
A search processing step for executing a search process for detecting the position and orientation of the work model from the bulk data generated in the bulk data generation step, using the search model registered in the search model registration step;
A picking motion simulation step for repeatedly executing a picking motion simulation for gripping the work model successfully searched in the search processing step by the hand portion input in the hand portion shape input step;
A robot simulation method, comprising: a display step for displaying information indicating a degree of success of gripping a work model by each hand unit obtained by picking motion simulation in a comparable manner.