JP7128933B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing apparatus, for example, measuring the three-dimensional shape of a plurality of workpieces piled up in a workspace to be picked by a sensor section, and gripping the workpieces by an end effector provided at the tip of an arm section of a robot. The present invention relates to an image processing apparatus for controlling a robot that performs a bulk picking operation that sequentially takes out.

By combining a manipulator with a robot vision, the target workpiece is imaged with the robot vision and height information is acquired. ) are being developed. Using such a robot device, a large number of workpieces placed in a returnable box, called bulk picking, are imaged by the cameras and sensors that make up the robot vision, the posture is grasped, and the appropriate gripping position is grasped. After that, the robot arm is moved to this gripping position, gripped by an end effector such as a hand provided at the tip of the arm, and placed at a predetermined position outside the returnable container. It is

In bulk picking using this kind of robot vision, the positional relationship between the workpiece and the robot when gripping the workpiece is registered as the gripping position at the time of setting. Calculate the position and move the robot to the calculated position for picking.

In such bulk picking, the end effector may interfere with surrounding obstacles such as other works and boxes when gripping the work. In order to avoid this, it is conceivable to calculate the position of the end effector at the gripping position, determine whether it interferes with surrounding objects, and output a solution that does not interfere as an effective solution. Here, the surrounding objects include a three-dimensional point cloud obtained by measurement by the sensor unit, a pre-registered box, a floor, and the like.

When creating such an end effector interference determination model, the method of reading the three-dimensional CAD data of the end effector is considered to be the least troublesome. For example, a method of reading three-dimensional CAD data and determining interference has been proposed (Patent Document 1). In this method, the three-dimensional CAD data of the end effector is read, and the interference judgment is performed using the end effector model. However, if you want to add an interference area, you need to edit the three-dimensional CAD data using a three-dimensional CAD program or the like to create the shape of the interference area and add it, which is a problem that requires complicated work. rice field.

A method of creating an end effector model with a rectangular parallelepiped and substituting it has also been proposed (Patent Document 2). In this method, in order to imitate the end effector, rectangular parallelepipeds are combined to create an end effector substitute model. However, it is not easy to create a model that is close to the actual end effector with only a combination of rectangular parallelepipeds, and there is a problem in terms of reproducibility.

JP 2015-9314 A Japanese Patent No. 3782679

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and one of the objects thereof is to provide an image processing apparatus that facilitates determination of interference between end effectors.

Means for solving the problem and effects of the invention

According to the image processing apparatus according to the first aspect of the present invention, the three-dimensional shape of a plurality of works piled up in the working space is measured by the sensor section, and the end provided at the tip of the arm section of the robot is measured by the robot controller. An image processing device for controlling a robot that performs a picking operation gripped by an effector, wherein an input image representing the three-dimensional shape of a plurality of workpieces from a sensor unit that measures the three-dimensional shape of workpieces arranged in a work space. a search model registration unit for registering a search model used in a three-dimensional search for specifying the position and orientation of the workpiece from the input image; and a tertiary of the end effector controlled by the robot controller. An end effector model registration unit for registering an end effector model that virtually represents the original shape, and a work model that virtually represents the three-dimensional shape of the work in a three-dimensional shape in a virtual three-dimensional space. a holding position specifying unit for specifying a holding position where the end effector model holds the work model displayed on the display unit; and the search model from the input image. a three-dimensional search unit that performs a three-dimensional search for specifying the position and orientation of one work using , the position and orientation of the one work specified by the three-dimensional search unit, and the work model corresponding to the one work specified by the gripping position specifying unit a three-dimensional pick determination unit that determines whether or not the one workpiece can be gripped by the end effector model based on the gripping position, and the three-dimensional pick determination unit performs a three-dimensional an interference determination unit that determines whether or not the end effector interferes with a surrounding object when the originally searched one workpiece is gripped by the end effector model; and an angle determination unit that determines whether the one work is within the tilt angle range, and the display unit displays the end effector model when gripping the one work on the input image. is displayed, and when the three-dimensional pick determination unit determines that the one workpiece cannot be gripped by the end effector model, the reason why the workpiece cannot be gripped can be displayed , and , the interference determination unit and the corner Different gripping impossibility factors are configured to be able to be displayed according to the determination result of the strength determination unit .

Further, according to the image processing apparatus according to the second embodiment, the three-dimensional shape of a plurality of works piled up in the work space is measured by the sensor section, and the end effector provided at the tip of the arm section of the robot is controlled by the robot controller. An image processing device for controlling a robot that performs a picking operation by gripping a workpiece with an input image representing the three-dimensional shape of a plurality of workpieces from a sensor unit that measures the three-dimensional shape of workpieces arranged in a work space. an input image acquisition unit for acquiring; a search model registration unit for registering a search model used in a three-dimensional search for specifying the position and orientation of the workpiece from the input image; and a three-dimensional end effector controlled by the robot controller. An end effector model registration unit for registering an end effector model that virtually represents a shape, and a work model that virtually represents the three-dimensional shape of the work in a three-dimensional shape in a virtual three-dimensional space. a display unit for displaying; a holding position specifying unit for specifying a holding position at which the work model displayed on the display unit is held by the end effector model; a three-dimensional search unit that performs a three-dimensional search for specifying the position and orientation of one work using a three-dimensional search unit; a three-dimensional pick determination unit that determines whether or not the one work can be gripped by the end effector model based on the gripping position specified by the gripping position specifying unit for the work model, The display unit displays the end effector model when gripping the one workpiece on the input image, and the three-dimensional pick determination unit determines that the one workpiece cannot be gripped by the end effector model. In the case where the one workpiece is not grippable, it is configured to be able to display the cause of the inability to grip the one workpiece, and furthermore, the display mode of the end effector model is changed and displayed according to the determination result of the three-dimensional pick determination unit. configured as possible.

Furthermore, according to the image processing apparatus according to the third embodiment, the three-dimensional shape of a plurality of works stacked in the work space is measured by the sensor unit, and the end effector provided at the tip of the arm of the robot is controlled by the robot controller. An image processing device for controlling a robot that performs a picking operation by gripping a workpiece with an input image representing the three-dimensional shape of a plurality of workpieces from a sensor unit that measures the three-dimensional shape of workpieces arranged in a work space. an input image acquisition unit for acquiring; a search model registration unit for registering a search model used in a three-dimensional search for specifying the position and orientation of the workpiece from the input image; and a three-dimensional end effector controlled by the robot controller. An end effector model registration unit for registering an end effector model that virtually represents a shape, and a work model that virtually represents the three-dimensional shape of the work in a three-dimensional shape in a virtual three-dimensional space. a display unit for displaying; a holding position specifying unit for specifying a holding position at which the work model displayed on the display unit is held by the end effector model; a three-dimensional search unit that performs a three-dimensional search for specifying the position and orientation of one work using a three-dimensional search unit; a three-dimensional pick determination unit that determines whether or not the one work can be gripped by the end effector model based on the gripping position specified by the gripping position specifying unit for the work model, The three-dimensional search unit searches for the position and position of the one workpiece from the input image based on the feature points on the contour representing the outline of the workpiece and the feature points on the surface representing the surface shape included in the input image. The display unit is configured to execute a three-dimensional search for identifying a posture, and the display unit displays the end effector model when gripping the one workpiece on the input image, and performs the three-dimensional pick determination. When the part determines that the one workpiece cannot be gripped by the end effector model, it is configured to be able to display the reason why the one workpiece cannot be gripped.

Furthermore, according to the image processing apparatus of the fourth aspect, the three-dimensional shape of a plurality of works piled up in the work space is measured by the sensor section, and the end provided at the tip of the arm section of the robot is measured by the robot controller. An image processing device for controlling a robot that performs a picking operation gripped by an effector, wherein an input image representing the three-dimensional shape of a plurality of workpieces from a sensor unit that measures the three-dimensional shape of workpieces arranged in a work space. a search model registration unit for registering a search model used in a three-dimensional search for specifying the position and orientation of the workpiece from the input image; and a tertiary of the end effector controlled by the robot controller. An end effector model registration unit for registering an end effector model that virtually represents the original shape, and a work model that virtually represents the three-dimensional shape of the work in a three-dimensional shape in a virtual three-dimensional space. a holding position specifying unit for specifying a holding position where the end effector model holds the work model displayed on the display unit; and the search model from the input image. a three-dimensional search unit that performs a three-dimensional search for identifying the position and orientation of one work using a three-dimensional pick determination unit that determines whether or not the one workpiece can be gripped by the end effector model based on the gripping position specified by the gripping position specifying unit for the work model to be picked. The display unit displays the end effector model when the one workpiece is gripped on the input image, and the three-dimensional pick determination unit determines that the one workpiece cannot be gripped by the end effector model. When it is determined, the cause of the inability to grip the one work can be displayed, and the display unit includes a three-dimensional search button for accepting the selection of the three-dimensional search and the It is possible to display a function selection screen in which a three-dimensional pick button for accepting selection of three-dimensional pick determination is arranged side by side, and when the selection of the three-dimensional search button is accepted from the function selection screen, the search model is registered. an icon for displaying a screen on the display unit; an icon for displaying a screen for setting a search area for performing the three-dimensional search on the display unit; an icon for displaying a screen for setting conditions for executing a search on the display unit in this order, and the selection of the three-dimensional pick button is received from the function selection screen. , the position and orientation calculated in the coordinate system of the vision space, which is a virtual three-dimensional space displayed on the display unit, and the position and orientation of the coordinate system in the robot space in which the robot controller operates the end effector; An icon for displaying a screen for selecting calibration information to be converted on the display unit, an icon for displaying a screen for registering the gripping position on the display unit, and conditions for detecting the gripping position are set. Icons for displaying the screen on the display section are arranged in this order and displayed on the display section.

Furthermore, according to the image processing apparatus according to the fifth aspect, in any one of the configurations described above , an input interface for acquiring the input image from the sensor unit; a display interface that outputs display data for displaying an end effector model; and a robot controller that, when the three-dimensional pick determination unit determines that the one workpiece can be gripped, causes the robot controller to pick the one workpiece. and a robot interface for outputting information necessary for controlling the robot connected to the robot controller.

Furthermore, according to the image processing apparatus according to the sixth aspect, in any one of the configurations described above, the interference determination unit receives a setting of an extension amount of the end effector model when performing interference determination, and the end effector It is configured to perform interference determination based on the model and the extension amount.

Furthermore, according to the image processing apparatus according to the seventh aspect, in any one of the configurations described above, an evaluation index calculation unit for calculating an evaluation index for the result of the three-dimensional search is further provided, and the evaluation index calculation unit is configured to calculate the evaluation index according to the existence ratio of feature points corresponding to the search model registered by the search model registration unit.

Furthermore, according to the image processing apparatus according to the eighth aspect, in any one of the configurations described above, the gripping priority determination unit is provided for determining the priority of the workpieces to be gripped by the end effector, and the gripping priority determination unit is , the priority is determined based on the evaluation index and the position of the workpiece in the Z direction.

FIG. 4 is a schematic diagram showing how a robot system performs a bulk picking operation; 1 is a block diagram of a robot system; FIG. It is a perspective view which shows an example of a sensor part. FIG. 4A is a schematic cross-sectional view showing an example in which workpieces are randomly put into a storage container and stacked, FIG. 4B is a schematic cross-sectional view showing an example in which workpieces are stacked on the floor, and FIG. 4C is a tray with workpieces in a fixed posture. It is a perspective view which shows the state arranged above. FIG. 5A is a schematic diagram showing an example of gripping a workpiece with an end effector, FIG. 5B is a schematic diagram showing an example of gripping a hollow workpiece from the inside, and FIG. 5C is a schematic diagram showing an example of gripping a plate-shaped workpiece by suction. It is a schematic diagram. 1 is a block diagram showing a robot system according to Embodiment 1; FIG. FIG. 2 is a perspective view showing a work model constructed from three-dimensional CAD data; FIG. 8 is an image diagram showing a state in which an origin of a search model is set with respect to the work model of FIG. 7; 9A is a basic direction image of the work model of FIG. 7 viewed from the positive direction of the X axis, FIG. 9B is a basic direction image of the work model viewed from the positive direction of the Y axis, and FIG. 9C is a basic direction image of the work model viewed from the positive direction of the Z axis. , and FIG. 9D is a cardinal direction image viewed from the negative direction of the Z-axis. FIG. 10 is an image diagram showing a search model registration screen for registering a basic direction image as a search model for three-dimensional search; FIG. 8 is an image diagram showing actual measurement data obtained by imaging a workpiece corresponding to the workpiece model of FIG. 7 from the positive direction of the X axis; 12A is an image diagram showing a state in which feature points are extracted from the image viewed from the X-axis direction in FIG. 9A, and FIG. 12B is an image diagram showing a state in which the image in FIG. 12A is three-dimensionally displayed. 13A is an input image in which a work group is displayed two-dimensionally, FIG. 13B is an input image in which FIG. 13A is displayed three-dimensionally, FIG. 13C is the result of performing a three-dimensional search on FIG. 13A, and FIG. 4A and 4B are image diagrams each showing a three-dimensional display state; FIG. FIG. 10 is an image diagram of a user interface screen showing a gripping registration screen; FIG. 11 is an image diagram showing a gripping posture addition screen; FIG. 16A is an image diagram showing a state in which a plurality of cubic works are randomly stacked, FIG. 16B is a state in which this work model is registered in the search model, and FIG. 16C is a state in which the gripping posture is registered for the work model in FIG. is. FIG. 7 is a functional block diagram showing a robot system according to Embodiment 2; FIG. 10 is an image diagram showing a state before the end effector model is rotated according to the ZYX system Euler angles; 19 is an image diagram showing a state rotated by 90° around the Z-axis from FIG. 18. FIG. FIG. 20 is an image diagram showing a state rotated by 90° around the Y-axis from FIG. 19; 21 is an image diagram showing a state rotated by 90° around the X-axis from FIG. 20. FIG. FIG. 22 is an image diagram showing rotation axes of R _Y and R _Z in the state of FIG. 21; FIG. 23 is an image diagram showing a state in which the _RZ correction rotation axis is displayed in the state of FIG. 22; FIG. 23 is an image diagram showing a state in which a correction rotation axis of _RY is displayed in the state of FIG. 22; FIG. 23 is an image diagram showing a state in which a correction rotation axis of R _X is displayed in the state of FIG. 22; FIG. 11 is a block diagram showing a robot system according to Embodiment 3; 4 is a flow chart showing the procedure of teaching work performed before actual operation. 4 is a flow chart showing a procedure for registering three-dimensional CAD data as a search model; 4 is a flow chart showing a procedure for registering an end effector model; FIG. 11 is a flow chart showing a procedure for registering an end effector model to which additional regions are added; FIG. FIG. 11 is a block diagram showing a robot system according to Embodiment 4; FIG. 11 is an image diagram showing an additional area setting screen; FIG. 4 is an image diagram showing a basic figure editing screen; 4 is a flow chart showing a procedure for registering a gripping position and posture for gripping a workpiece with an end effector; FIG. 35A is an image diagram of a work model composed of three-dimensional CAD data, and FIG. 35B is an image diagram showing an XY designation screen. FIG. 10 is an image diagram showing a ZR _Z designation screen; FIG. 11 is an image diagram showing an _RY designation screen; FIG. 11 is an image diagram showing an R _X designation screen; FIG. 10 is an image diagram showing a position parameter designation screen; FIG. 10 is an image diagram showing an XYZ designation screen; FIG. 35B is an image diagram showing how a gripping position is specified in a state where the work model of FIG. 35B is projected from an oblique direction; FIG. 10 is an image diagram showing how a gripping position is designated on the 3D image viewer. FIG. 3 is an image diagram showing a flat plate-like work; 44A is a plan view of a workpiece model representing the workpiece in FIG. 43, FIG. 44B is a bottom view, FIG. 44C is a front view, FIG. 44D is a rear view, FIG. 44E is a right side view, and FIG. 44F is a left side view. FIG. 44 is an image diagram showing the result of performing a three-dimensional search on point cloud data obtained by imaging a work group in which a plurality of works of FIG. 43 are randomly stacked. FIG. 4 is a schematic diagram showing an example of expressing the posture of a workpiece using the Euler angles of the ZYZ system; FIG. 11 is an image diagram showing a search model registration screen; FIG. 10 is a flow chart showing a registration procedure of a search model with attitude restrictions; FIG. It is an image figure which shows an inclination-angle / rotation-angle setting screen. FIG. 50A is an image diagram showing the orientation of the work at the time of registration, FIG. 50B is an orientation of the input image, and FIG. 50C is an image diagram showing the tilt angle. FIG. 51A is an image diagram showing the orientation of the workpiece at the time of registration, FIG. 51B is an orientation of the input image, and FIG. 51C is an image diagram showing the rotation angle. 4 is a flow chart showing a procedure for obtaining an inclination angle and a rotation angle from a three-dimensional posture; FIG. 53A is an image diagram showing the orientation of the workpiece at the time of registering the search model, FIG. 53B is the orientation of the input image, and FIG. FIG. 54A is an image diagram showing a state in which the search model in FIG. 53C shows the rotation axis, and FIG. 54B is an image diagram showing a state in which the Z′-axis in FIG. 54A is three-dimensionally rotated so as to match the Z-axis. FIG. 55A is an image diagram showing the posture of the workpiece at the time of registration, FIG. 55B is an image diagram showing how the rotation angle when viewed from the Z vertical direction in FIG. 55B is obtained as the rotation angle. FIG. 5 is an image diagram showing an end effector mounting position setting screen; FIG. 11 is an image diagram showing a gripping position specifying screen; FIG. 11 is an image diagram showing a multiple gripping position selection screen; FIG. 11 is an image diagram showing an end effector mounting position calibration screen; 4 is a flow chart showing a procedure for automatically calibrating an end effector mounting position; FIG. 10 is an image diagram showing an end effector imaging screen; 4 is a flow chart showing a procedure for registering measured data as a search model; 7 is a flowchart showing a procedure for determining whether or not there is a gripping solution for each workpiece during actual operation; FIG. 49 is a flow chart showing a procedure during actual operation in which a picking operation is performed with a search model registered in the procedure of FIG. 48; FIG. 4 is a flow chart showing a procedure for determining interference using a cross-sectional model; FIG. 4 is an image diagram showing an end effector model and its basic axis and cross-sectional position; 67A is an image diagram showing the cross section SS1 of FIG. 66, FIG. 67B is the cross section SS2, FIG. 67B is the cross section SS3, FIG. 67D is the cross section SS4, and FIG. 68A is an image diagram showing the three-dimensional point and the basic axis of the end effector model of FIG. 66, FIG. 68B is a state in which the projection point of the three-dimensional point in FIG. FIG. 10 is an image diagram showing a state of being held. 9 is a flow chart showing a procedure for collision determination using an additional model; FIG. 11 is a block diagram showing a robot system according to Embodiment 7; FIG. 14 is a flowchart showing a procedure for determining whether or not there is a gripping solution for each workpiece during actual operation according to Embodiment 7; FIG. It is a perspective view which shows an example of a workpiece|work. 73A is a height image A of the work in FIG. 72 viewed from the positive direction side of the Z axis, FIG. 73B is a height image B viewed from the negative direction side of the Z axis, and FIG. 73C is a height image B viewed from the positive direction side of the X axis. FIG. 73D is a height image D viewed from the negative direction of the X axis, FIG. 73E is a height image E viewed from the positive direction of the Y axis, and FIG. 73F is viewed from the negative direction of the Y axis. It is an image figure which shows the height image F which was seen, respectively. FIG. 11 is an image diagram showing a work selection screen; FIG. 11 is an image diagram showing a gripping solution candidate display screen in which gripping OK is selected; FIG. 11 is an image diagram showing a gripping solution candidate display screen in which gripping NG is selected; FIG. 11 is an image diagram showing still another example of a gripping solution candidate display screen in which gripping NG is selected; FIG. 11 is an image diagram showing an example of a gripping solution candidate display screen for search model C; FIG. 79 is an image diagram showing an example of a gripping solution candidate display screen in which another gripping position candidate is selected in FIG. 78; FIG. 11 is an image diagram showing an example of a gripping solution candidate display screen in which a gripping position is added and gripping is OK. FIG. 12 is a block diagram showing a robot system according to Embodiment 8; FIG. 16 is a flow chart showing a search model registration procedure according to an eighth embodiment; FIG. 83A shows the state of registering the gripping posture in the basic direction image of FIG. 73A, FIG. 83B shows the state of registering the gripping posture in the basic direction image of FIG. 73B, and FIG. 83C shows the state of registering the gripping posture in the basic direction image of FIG. 73C. 83D is a state of registering the gripping posture in the basic direction image of FIG. 73D, FIG. 83E is a state of registering the gripping posture in the basic direction image of FIG. 73E, and FIG. 83F is a state of registering the gripping posture in the basic direction image of FIG. 73F. 4A and 4B are image diagrams showing respective states; FIG. FIG. 11 is an image diagram showing a work selection screen; FIG. 11 is an image diagram showing a gripping solution candidate display screen in which gripping posture C-000 is selected; FIG. 11 is an image diagram showing a gripping solution candidate display screen in which a gripping posture A-000 is selected; FIG. 21 is a flow chart showing a procedure for performing a three-dimensional search and graspability determination during actual operation according to the eighth embodiment; FIG. FIG. 22 is a block diagram showing a robot system according to Embodiment 9; FIG. 11 is an image diagram showing a function selection screen; FIG. 10 is an image diagram showing a search model registration method selection screen; FIG. 10 is an image diagram showing an actual measurement data imaging screen; FIG. 11 is an image diagram showing a search model exclusion area setting screen; FIG. 11 is an image diagram showing a rotational symmetry setting screen; FIG. 11 is an image diagram showing a search model registration screen; FIG. 10 is an image diagram showing a search model registration screen to which a search model has been added; FIG. 11 is an image diagram showing a search area setting screen; FIG. 11 is an image diagram showing a search area setting screen displaying a floor specification dialog; FIG. 10 is an image diagram showing a search area setting screen displaying floor surface information; FIG. 10 is an image diagram showing a search parameter setting screen; FIG. 10 is an image diagram showing a search parameter setting screen displaying a detailed detection condition setting dialog; FIG. 10 is an image diagram showing a search parameter setting screen to be specified for each search model; FIG. 10 is an image diagram showing a search model registration method selection screen; FIG. 10 is an image diagram showing a CAD data reading screen; FIG. 4 is an image diagram showing a search model registration screen based on three-dimensional CAD data; FIG. 10 is an image diagram showing a search model registration screen displaying a rotational symmetry designation dialog; FIG. 11 is an image diagram showing a model list display screen; FIG. 11 is an image diagram showing a search result display screen; FIG. 11 is an image diagram showing a search result display screen; FIG. 11 is an image diagram showing a search result display screen; FIG. 11 is an image diagram showing a search result display screen; FIG. 11 is an image diagram showing a search result display screen; FIG. 11 is an image diagram showing a search result display screen; FIG. 11 is an image diagram showing a search result display screen; FIG. 11 is an image diagram showing a model editing screen; FIG. 11 is an image diagram showing a model area detail setting screen; FIG. 11 is an image diagram showing a detailed feature setting screen; FIG. 10 is an image diagram showing a three-dimensional pick initial setting screen; FIG. 11 is an image diagram showing an end effector model setting screen; FIG. 11 is an image diagram showing an end effector model editing screen; FIG. 11 is an image diagram showing a parts addition screen; FIG. 11 is an image diagram showing a CAD position/orientation setting screen; FIG. 11 is an image diagram showing a CAD position/orientation setting screen; FIG. 11 is an image diagram showing a CAD position/orientation setting screen; FIG. 11 is an image diagram showing an end effector model editing screen; FIG. 11 is an image diagram showing an additional area position/orientation setting screen; FIG. 11 is an image diagram showing an end effector model editing screen; FIG. 11 is an image diagram showing an end effector model editing screen; FIG. 11 is an image diagram showing an end effector model editing screen; FIG. 11 is an image diagram showing an end effector model editing screen; FIG. 11 is an image diagram showing an end effector model editing screen; FIG. 11 is an image diagram showing a gripping registration screen; FIG. 11 is an image diagram showing a grip setting dialog; FIG. 10 is an image diagram showing a setting method selection dialog; FIG. 11 is an image diagram showing an XY designation screen; FIG. 10 is an image diagram showing a ZR _Z designation screen; FIG. 10 is an image diagram showing a ZR _Z designation screen; FIG. 10 is an image diagram showing a ZR _Z designation screen; FIG. 11 is an image diagram showing an _RY designation screen; FIG. 10 is an image diagram showing an _RY designation screen; FIG. 11 is an image diagram showing an R _X designation screen; FIG. 11 is an image diagram showing an R _X designation screen; FIG. 11 is an image diagram showing a gripping registration screen; FIG. 11 is an image diagram showing a gripping registration screen; FIG. 11 is an image diagram showing a condition verification screen; FIG. 11 is an image diagram showing a verification dialog; It is an image figure which shows a detection result display dialog. It is an image figure which shows a detection result display dialog. FIG. 11 is an image diagram showing a detection condition detail setting screen; FIG. 11 is an image diagram showing a place setting screen; 4 is a flow chart showing a procedure for manually performing deviation correction. FIG. 11 is an image diagram showing an example of a deviation correction screen when there is a deviation that cannot be ignored; FIG. 11 is an image diagram showing an example of a deviation correction screen when the deviation can be ignored; FIG. 10 is an image diagram showing an example of a deviation correction screen having manual deviation correction and automatic deviation correction functions;

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below with reference to the drawings. However, the embodiments shown below are examples for embodying the technical idea of the present invention, and the present invention is not limited to the following. In addition, this specification does not in any way specify the members shown in the claims as the members of the embodiment. Unless otherwise specified, the dimensions, materials, shapes, relative arrangements, etc. of the components described in the embodiment are not meant to limit the scope of the present invention, but are merely illustrative examples. It's nothing more than Note that the sizes and positional relationships of members shown in each drawing may be exaggerated for clarity of explanation. Furthermore, in the following description, the same names and symbols indicate the same or homogeneous members, and detailed description thereof will be omitted as appropriate. Furthermore, each of the elements constituting the present invention may be configured with the same member so that a single member can serve as a plurality of elements, or conversely, the function of one member can be It can also be realized by sharing.
(Embodiment 1)

As Embodiment 1, FIG. 1 shows a configuration example of a robot system 1000 for picking a work to be picked. In this example, a plurality of works WK piled up in a work space are picked up one by one by using the robot RBT and placed at a predetermined position. The robot RBT is also called a manipulator or the like, and includes an arm portion ARM and an end effector HND provided at the tip of the arm portion ARM. The arm part ARM has a plurality of movable parts, and moves the end effector EET to a desired position according to the angle formed by the two arms and the rotation of the arm fulcrum. The end effector EET can grip or release the work WK.

A robot controller 6 controls the operation of the robot RBT. The robot controller 6 controls the movement of the arm portion ARM and the opening/closing operation of the end effector EET. Further, the robot controller 6 acquires information necessary for controlling the robot RBT from the image processing device 100 . For example, the three-dimensional shape of a workpiece WK, which is a large number of parts randomly placed in the storage container BX, is acquired by the sensor unit 2 such as a three-dimensional camera and illumination, and the workpiece is determined by the calculation unit 10 of the image processing device 100. It detects the position and posture and sends the information to the robot controller 6. - 特許庁The robot controller 6 grips the works WK one by one with the end effector EET provided at the tip of the arm portion ARM of the robot RBT, and arranges them on a predetermined position on the stage STG, for example, on a conveyor belt.

A functional block diagram of the robot system 1000 is shown in FIG. A robot system 1000 shown in this figure includes an image processing device 100 , a sensor section 2 , a display section 3 , an operation section 4 , a robot body 5 , a robot controller 6 , and a robot operation tool 7 .

The operation unit 4 is used to make settings related to image processing. Also, the sensor unit 2 acquires a three-dimensional shape by capturing an image of the workpiece. Further, the display unit 3 is used to confirm settings and operating conditions. Furthermore, the calculation unit 10 performs three-dimensional search, interference determination, calculation of grasping solutions, and the like. On the other hand, the robot controller 6 controls the robot according to the result of the calculation section 10 . Further, the robot operating tool 7 sets the operation of the robot. In addition, in the example of FIG. 2, the operation part 4 and the robot operation tool 7 are separate members, but they may be a common member.

The sensor unit 2 is a member called a robot vision or the like that captures an image of a work space or a work. Three-dimensional shape data representing the three-dimensional shape of the randomly stacked workpieces is acquired from the image captured by the sensor unit 2 . Methods for obtaining a three-dimensional shape include a pattern projection method, a stereo method, a lens focus method, a light section method, an optical radar method, an interferometry method, a TOF method, and the like. In this embodiment, the phase shift method is used among the pattern projection methods.

The configuration of the sensor unit 2 is determined according to the three-dimensional shape measurement technology. This sensor unit 2 includes a camera, lighting, a projector, or the like. For example, when measuring the three-dimensional shape of a workpiece by the phase shift method, as shown in FIG. Note that the sensor unit may be composed of a plurality of members such as a camera and a projector, or may be integrally composed of these members. For example, a 3D imaging head formed by integrating a camera and a projector into a head shape can be used as the sensor section.

The generation of the three-dimensional shape data itself can also be performed on the sensor unit side. In this case, an image processing IC or the like for realizing the function of generating three-dimensional shape data is provided on the sensor unit side. Alternatively, the image processing device does not generate the three-dimensional shape data, and the raw image captured by the sensor unit is processed by the image processing device to generate three-dimensional shape data such as a three-dimensional image. good too.

Furthermore, by performing calibration described later based on the image captured by the sensor unit 2, the actual position coordinates of the work WK (coordinates of the movement position of the end effector EET) and the coordinates of the movement position of the end effector EET are displayed on the display unit 3. can be linked with the position coordinates on the image.

The image processing apparatus 100 performs three-dimensional search, interference determination, grasping solution calculation, etc., based on the three-dimensional shape data of the workpiece thus obtained. As the image processing apparatus 100, a general-purpose computer installed with a dedicated image processing program, a dedicated image processing controller, a robot vision device, or the like can be used. Although the example of FIG. 2 shows an example in which the sensor unit 2, the robot controller 6, and the like are configured by separate members from the image processing apparatus 100, the present invention is not limited to this configuration. The device can be integrated, or the robot controller can be incorporated into the image processing device. Thus, the division of members shown in FIG. 2 is an example, and a plurality of members can be integrated. For example, the operation unit 4 that operates the image processing apparatus 100 and the robot operation tool 7 that operates the robot controller 6 may be a common member.

However, the sensor unit 2 is separate from the robot main body 5 . That is, the present invention is intended for a form called an off-hand type in which the sensor section 2 is not provided on the arm section ARM of the robot body 5 . In other words, an aspect called an on-hand type or the like in which the sensor section is provided in the end effector is not included in the present invention.

The display unit 3 is a member for displaying the three-dimensional shape of the workpiece acquired by the image processing apparatus 100, and for confirming various settings and operating conditions, and can use a liquid crystal monitor, an organic EL display, a CRT, or the like. . The operation unit 4 is a member for performing various settings such as image processing, and can use input devices such as a keyboard and a mouse. Further, by using a touch panel as the display section 3, the operation section and the display section can be integrated.

For example, when the image processing apparatus 100 is configured by a computer in which an image processing program is installed, a graphical user interface (GUI) screen of the image processing program is displayed on the display section 3 . Various settings can be made on the GUI displayed on the display unit 3, and processing results such as simulation results can be displayed. In this case, the display section 3 can be used as a setting section for performing various settings.

A robot controller 6 controls the motion of the robot based on information captured by the sensor unit 2 . The robot operation tool 7 is a member for setting the operation of the robot main body 5, and a pendant or the like can be used.

The robot body 5 includes a movable arm portion ARM and an end effector EET fixed to the tip of the arm portion ARM. The robot main body 5 is driven by the robot controller 6 to operate the arm part ARM, pick up one work WK, move it to a desired position, place it, and then release it. For this reason, an end effector EET for gripping the work WK is provided at the tip of the arm portion ARM. The placement position for placing the work WK may be, for example, on a tray or on a conveyor.

As shown in FIG. 1, a plurality of works WK are randomly stored in a storage container BX such as a returnable box. A sensor unit 2 is arranged above such a working space. The sensor section 2 has a camera and illumination, and can measure the three-dimensional shape of the work WK with this sensor section 2 . Based on the three-dimensional shape of the work WK measured by the sensor unit 2, the robot controller 6 specifies the work WK to be gripped from among a plurality of works, and controls the robot to grip this work WK. do. Then, while holding the work WK, the arm part ARM is operated to move it to a predetermined placement position, for example, the stage STG, and the work WK is placed in a predetermined posture. In other words, the robot controller 6 identifies the work WK to be picked by the sensor unit 2, grips this work WK with the end effector EET, and places the gripped work WK in a predetermined reference posture at a position to be placed. The operation of the robot is controlled so that it is placed at a certain placement position and the end effector EET is released.

Here, in this specification, bulk picking means that workpieces WK placed in a storage container BX as shown in FIG. An example of gripping and placing works WK stacked in a predetermined area without using a storage container as shown in FIG. 4B, or works stacked in a predetermined posture as shown in FIG. 4C. It is used in the sense that it includes an example of sequentially gripping and placing WKs. In addition, it is not always necessary that the works are stacked on top of each other, and works that are randomly placed on a plane without overlapping each other are also referred to as random stacking in this specification (sequential (This is the same reason why picking is still called bulk picking, even when there is no overlap between works at the end of picking.) It should be noted that the present invention is not necessarily limited to picking in bulk, but can also be applied to pick up works that are not in bulk.

In addition, in the example of FIG. 1, the sensor unit 2 is fixed above the work space, but any fixed position such as oblique, lateral, or downward is sufficient as long as it can capture an image of the work space. However, a mode in which the sensor section is arranged at a movable and indefinite position such as on the arm section ARM is excluded. Furthermore, the number of cameras and illuminations included in the sensor unit 2 is not limited to one, and may be plural. Furthermore, the connection with the sensor unit 2, the robot, and the robot controller 6 is not limited to wired connection, and may be wireless connection.

Grasping of the work means pinching the outside of the work WK as shown in FIG. 5A, and inserting the claws of the end effector EET2 into the interior of the work WK2 having a cavity as shown in FIG. 5B to expand the work WK. It is used in the sense of including an example of holding by and an example of an end effector EET3 that sucks and holds a plate-shaped work WK3 as shown in FIG. 5C. In the following, as an example of gripping the work, a mode of gripping the outer surfaces of the work from both sides will be described. Also, as shown in FIG. 1, a large number of works are stored in a storage container BX and piled up at random. Next, the setting of gripping positions (teaching work) in the bulk picking operation in which the work of placing the objects at the mounting positions is repeated will be described below.

When the robot system 1000 performs the bulk picking operation, teaching including settings for performing the bulk picking operation is performed in advance. Specifically, which part of the work is gripped by the end effector in what posture, gripping positions, and the like are registered. Such settings are performed using a robot operation tool 7 such as a pendant.
(teaching work)

FIG. 6 shows a functional block diagram of a robot system including an image processing device that realizes the gripping position teaching function described above. A robot system 1000 shown in this figure includes an image processing device 100, a display section 3, an operation section 4, a sensor section 2, and a robot RBT.

The sensor unit 2 three-dimensionally measures the three-dimensional shape of the work placed at the working position. The sensor section 2 is controlled by a sensor control section 2b. In this example, the sensor control section 2b is integrated with the sensor section 2, but they may be provided separately. The robot section includes an arm section ARM and an end effector EET. This robot is controlled by the image processing apparatus 100 to grip the workpiece at the gripping position and grip it at the gripping position. Here, the state in which the workpiece is held in the reference posture and placed in the reference posture is imaged by the sensor unit 2 and registered. Here, the reference orientation includes the position and orientation of the workpiece.
(Display section 3)

The display unit 3 displays a work model that virtually represents the three-dimensional shape of the work, and an end effector model that is composed of three-dimensional CAD data that virtually represents the three-dimensional shape of the end effector. Each is displayed three-dimensionally in the original space. A height image is an image having height information, and is also called a distance image or the like. Further, the display unit 3 has a hexahedral view display area 3a for displaying the basic direction image of the work model as a hexahedral view. As a result, each posture of the workpiece model can be displayed in a hexahedron view, and the gripping position can be set with respect to the basic direction image, which makes it easy to set the gripping position. can be easily performed.

The operation unit 4 is a member for performing various settings such as image processing, and can use input devices such as a keyboard and a mouse. Further, by using a touch panel as the display section 3, the operation section and the display section can be integrated.
(Image processing device 100)

The image processing apparatus 100 of FIG. 6 includes an input image acquisition section 2c, a storage section 9, a calculation section 10, an input/output interface 4b, a display interface 3f, and a robot interface 6b.

The input image acquisition unit 2c acquires an input image having a three-dimensional shape from images including a plurality of workpieces measured by the sensor unit 2 and surrounding objects. An input image having a three-dimensional shape may be constructed by the sensor unit or the sensor control unit, or may be constructed by the image processing device (for example, the input image acquisition unit). Also, in the example of FIG. 6, an input interface is configured for the image processing apparatus 100 to acquire an input image representing a three-dimensional shape acquired by the sensor unit 2 . However, the configuration is not limited to this configuration, and a configuration in which an input image captured in advance and held in a storage unit such as a recording medium is read and acquired may be employed.

The storage unit 9 is a member for holding various settings, and can use a nonvolatile memory, a hard disk, a storage medium, or the like. The storage unit 9 functions as a gripping position storage unit for storing gripping positions of the work model and the end effector model.

The input/output interface 4b is connected to input devices such as a keyboard and a mouse, and accepts input of data.

The display interface 3f constitutes an output interface with the display section and controls the display section constructed by the calculation section 10 to display image data.

The robot interface 6 b constitutes a communication interface with the robot controller 6 .
(Calculation unit 10)

The calculation unit 10 includes a positioning unit 8c, a basic direction image generation unit 8e′, a basic direction image selection unit 8e, a search model selection unit 8i, a work model registration unit 8t, a search model registration unit 8g, an end effector, and an end effector. It includes a model registration section 8u, a gripping position specifying section 8d, an end effector mounting surface setting section 8f, a rotation angle limiting section 8h, a three-dimensional search section 8k, and a three-dimensional pick determination section 8l.

The positioning unit 8c is a member for adjusting the position and orientation of the work model displayed on the display unit 3 in the virtual three-dimensional space.

The basic direction image generation unit 8e' provides a view of the work model positioned in the virtual three-dimensional space by the positioning unit 8c from each of three axial directions orthogonal to each other in the virtual three-dimensional space, at least It is a member for generating three height images as basic direction images. By automatically generating the basic orientation image in this way, the user does not need to manually change the orientation of the workpiece to individually acquire the basic orientation image, which has the advantage of saving labor in the gripping position registration work. .

The basic direction image selection unit 8e selects at least three basic direction images displayed on the display unit 3 from among a plurality of basic direction images different in appearance from other basic direction images. It is a member. In this way, by deleting surfaces that have a common appearance, it is possible to eliminate unnecessary display of basic direction images and the like, thereby simplifying the setting work. For example, one of the basic direction images having the same appearance, such as the top surface and the bottom surface of a cylindrical workpiece, is deleted. The basic direction image selection unit 8e can be configured to allow the user to manually select from at least three basic direction images displayed on the display unit 3 . Alternatively, the basic direction image selection unit may be configured to automatically extract and select a basic direction image having a common appearance from at least three basic direction images.

The search model selection unit 8i is a member for selecting a basic direction image to be registered as a search model. As will be described later, by sharing the search model used for the three-dimensional search and the model for specifying the gripping position, the search model selection unit 8i, the basic direction image selection unit 8e, and the common image selection unit 8j can be configured. .

The work model registration unit 8t is a member for registering a work model that virtually represents the three-dimensional shape of the work. Here, for example, the work model registration unit 8t uses three-dimensional point cloud data obtained by imaging an actual work as a work model. In this case, the three-dimensional point cloud data acquired by the sensor unit 2 and the input image acquisition unit 2c are registered as a work model by the work model registration unit 8t. Alternatively, three-dimensional CAD data representing the shape of a separately created work is read and registered as a work model. In this case, the three-dimensional CAD data input via the input/output interface 4b is registered as a work model in the work model registration section 8t. Alternatively, three-dimensional CAD data simulating the workpiece may be created and registered. In this case, the work model registration unit 8t implements a simple three-dimensional CAD function.
(Search model registration unit 8g)

The search model registration unit 8g is used when performing a three-dimensional search for specifying the posture and position of each work in a plurality of works included in the input image acquired by the input image acquisition unit 2c. This is a member for registering a search model that virtually represents the original shape. By providing the search model registration unit 8g in this way, the search model for performing the three-dimensional search can be registered in common with the basic direction image for designating the gripping position of the work model, thereby saving the user's setting work. . In addition, even during actual operation, by registering the gripping position of a workpiece for each basic orientation image for searching for grippable workpieces, waste such as searching for a basic orientation image with no gripping position can be eliminated. Since it is possible to examine whether or not the object can be gripped at the gripping position set in the basic direction image, it is possible to perform processing efficiently.

The search model registration unit 8g is preferably configured to select whether or not to use the basic direction image selected by the basic direction image selection unit 8e as a search model for three-dimensional search. As a result, it is possible to select whether or not to use the basic direction image as a search model for the three-dimensional search. By eliminating the basic direction image that may be mistakenly detected as the image viewed from the side, it is possible to set up so that the plate-like workpiece is upright in the 3D search, which practically limits the pose of the workpiece model. It is possible to set easily.

The search model registration unit 8g and the work model registration unit 8t can be provided separately or integrated. For example, in the robot system 1000 of FIG. 6, the search model registration unit 8g and the workpiece model registration unit 8t are integrated as a common model registration unit 8g'. As a result, by registering a model related to one workpiece, it becomes possible to share the registration of the gripping position and the registration of the search model for three-dimensional search, thereby obtaining the advantage of simplifying the setting.

The end effector model registration unit 8u is a member for registering an end effector model in which the three-dimensional shape of the end effector is virtually represented by three-dimensional CAD data.

The end effector mounting surface setting unit 8f causes the display unit 3 to display the end effector model and the mounting surface for mounting the end effector model to the tip of the arm of the robot, and causes the display unit 3 to display the mounting surface. and is a member for defining the posture of the end effector model with respect to this mounting surface.

The rotation angle limiting unit 8h registers one of the basic direction images as a search model for performing a three-dimensional search for identifying the posture and position of each of a plurality of randomly stacked workpieces. A search model registration unit 8g and a member for individually setting the range of allowable rotation angle for rotation of the work model for each selected search model.
(Grip position specifying unit 8d)

The gripping position specifying unit 8d provides at least three positions of the work model positioned in the virtual three-dimensional space by the positioning unit 8c, viewed from each of three axial directions orthogonal to each other in the virtual three-dimensional space. It is a member for specifying a gripping position for gripping the work model with the end effector for at least one height image in a state where the basic direction image is displayed on the display unit 3 . The gripping position specifying section 8d includes a workpiece side gripping point specifying section 8d1, an end effector side gripping setting section 8d2, and a relative position setting section 8d5.

The work-side gripping point designation unit 8d1 displays a plurality of basic direction images selected by the basic direction image selection unit 8e on the display unit 3, and selects one of the basic direction images. It is a member for designating the gripping position when the end effector model grips the workpiece model shown. The work-side gripping point designation unit 8d1 is configured to be able to register a plurality of gripping positions for each of a plurality of basic direction images.
(End effector side gripping setting portion 8d2)

The end effector-side gripping setting unit 8d2 is a member that performs settings related to gripping of the work model for the end effector model. The end effector-side gripping setting portion 8d2 can include a gripping reference point setting portion 8d3 and a gripping direction setting portion 8d4. The gripping reference point setting unit 8d3 defines gripping reference points corresponding to positions at which the work model is gripped for the end effector model. On the other hand, the gripping direction setting section 8d4 defines the gripping direction for gripping the work model with the end effector model. The gripping reference point setting portion 8d3 and gripping direction setting portion 8d4 may be a common member or may be configured individually. The gripping reference point setting unit 8d3 and the gripping direction setting unit 8d4 can set the gripping reference point and the gripping direction to predetermined values, respectively. For example, the gripping reference point is the center between claws provided at the tip of the end effector model and sandwiching the workpiece. The gripping direction is any one of the coordinate axes forming the tool coordinate axes that define the end effector model. For example, by using the Z-axis direction, the end effector model is moved along the Z-axis direction, i.e., the height direction, to approximate the action of approaching the work model for gripping, which makes it easier for the user to intuitively grasp. Alternatively, the gripping reference point setting section 8d3 and the gripping direction setting section 8d4 may allow the user to adjust the gripping reference point and the gripping direction.
(Relative position setting unit 8d5)

The relative position setting unit 8d5 moves the end effector model displayed on the display unit 3 until it interferes with the work model, and automatically adjusts the gripping state to a position separated by a predetermined amount from the position where the interference position is reached. It is a member for specifying. As a result, the user does not have to manually move the end effector model to contact the work, but automatically moves the end effector model to contact the work to indicate the gripping position. The advantage is that the work can be significantly labor-saving.
(Registration of grip position)

In the teaching work, the positional relationship between the workpiece and the end effector when gripping the workpiece is registered as the gripping position. In the following, as a representative example of gripping, an example of gripping a workpiece using a hand as an end effector will be described. With the gripping positions registered, during actual operation of robot picking using robot vision, each workpiece is detected by robot vision from a group of randomly stacked workpieces. The grip position on the end effector side is calculated with respect to the posture, and the robot is operated so that the end effector is positioned at the calculated position for picking. Here, as a grip position registration method, there is a method of actually operating a robot for registration, and a method of operating an end effector model in a virtual three-dimensional space using three-dimensional CAD for registration. However, the method of registering gripping positions by actually operating the robot requires time-consuming registration work while the robot is actually moving, requiring a large-scale verification environment, and the problems of trial and error taking time. there were. On the other hand, the method of registering a robot by virtually operating it in a three-dimensional CAD space has the advantage of being able to register without the need for a robot. It was necessary to accurately match the three-dimensional posture of the effector model, and it was difficult to make settings such as alignment of the three-dimensional coordinates. In addition, since this method requires three-dimensional CAD data of the workpiece and the end effector, it was not possible to set up without the three-dimensional CAD data at hand.

Therefore, in this embodiment, a plurality of height images of the three-dimensional CAD model viewed from each axial direction are displayed as basic direction images, and from among these, a desired basic direction image is selected by the user, and the selected basic direction image is displayed. By setting the gripping position with respect to the directional image, it is possible to easily perform gripping registration in the virtual three-dimensional space. As a result, when operating and registering the end effector model in the virtual three-dimensional space in which the work model of the three-dimensional CAD data simulating the work is arranged, the basic direction viewed from each axial direction defining the virtual three-dimensional space By selecting an image and performing grip registration, it is possible to register the posture of the end effector model viewed from a direction substantially perpendicular to each basic direction image as a reference, so grip settings can be easily performed. . Further, according to this method, even if there is no three-dimensional CAD data, data obtained by three-dimensional measurement of an actual workpiece can be used as the basic direction image. Therefore, even if there is no three-dimensional CAD data, grip registration can be easily performed in the virtual three-dimensional space by the same procedure.
(Three-dimensional pick determination unit 8l)

The three-dimensional pick determination unit 8l performs gripping such that the gripping direction defined by the gripping direction setting unit 8d4 is perpendicular to the work plane representing the orientation of the work model displayed in the image display area and along this gripping direction. A member for adjusting the relative positions of the end effector model and the workpiece model so as to position the reference point and the gripping position. As a result, when simulating a gripping state in which a workpiece is gripped by the end effector, it is possible to easily set the position where the workpiece model is gripped by the end effector model. In particular, the gripping direction is orthogonal to the work plane, and the gripping reference point and the gripping position are positioned on the axis of the gripping direction. Since it can be adjusted, there is an advantage that the work load on the user side is greatly reduced. Conventionally, the user manually moves the end effector and visually adjusts it to the posture of gripping the workpiece. There are many parameters and a large degree of freedom in how to adjust the position and posture of the end effector. Therefore, it was an extremely complicated task. On the other hand, a gripping reference position and a gripping direction are defined in advance on the end effector model side, and the gripping reference point of the end effector model and the gripping position of the workpiece are set so that the gripping direction is orthogonal to the workpiece plane and on the axis of the gripping direction. The moving direction of the end effector model is specified by setting the position of the end effector model so that the user can obtain the gripped state by adjusting only the distance between the end effector model and the work model. As a result, it can be expected that the work of adjusting the gripping positions of the end effector model and the work model, which has conventionally been extremely complicated, can be greatly reduced.

The operation of adjusting the relative positions of the end effector model and the workpiece model so that the gripping direction is orthogonal to the workpiece plane and the gripping reference point and gripping position are positioned on the axis of the gripping direction is performed by the three-dimensional pick determination unit. It is preferred to have it done automatically by 8l. As a result, the gripping reference position and the gripping direction are defined in advance on the end effector model side, and the gripping reference point of the end effector model and the gripping position of the workpiece are aligned so that the gripping direction is orthogonal to the workpiece plane and on the axis of the gripping direction. By automatically adjusting the position, the moving direction of the end effector model is specified, and the user can obtain the gripping state by adjusting only the distance between the end effector model and the workpiece model. As a result, it can be expected that the work of adjusting the gripping positions of the end effector model and the work model, which has conventionally been extremely complicated, can be greatly reduced.

However, it is not necessarily limited to the automatic adjustment by the three-dimensional pick determination unit. The three-dimensional pick determination unit may be configured to support such adjustment work when adjusting the holding position of the . For example, as a first step, the work plane and the gripping direction are displayed in the image display area, and an auxiliary line is shown to the user so that they are perpendicular to each other. Please adjust the end effector model as follows." may be displayed to prompt the user to perform adjustment work. Furthermore, as a second step, an extension line of the gripping direction is displayed in the image display area so that the gripping reference point of the end effector model and the gripping position of the workpiece are positioned on the axis of the gripping direction. Please adjust so that the gripping reference point of the end effector model and the gripping position of the workpiece are positioned."

A fitting function can also be realized by the three-dimensional pick determination unit 8l. For example, the three-dimensional pick determination unit 8l determines the relative positions of the end effector model and the workpiece model in a state in which the gripping direction is orthogonal to the workpiece plane and the gripping reference point and gripping position are positioned on the axis of the gripping direction. The effector model can be moved along the gripping direction until it interferes with the workpiece model, and the gripping state can be automatically defined in a posture separated from the interference position by a predetermined amount. As a result, it becomes possible to automatically adjust the position and orientation of the end effector model to grip the work model, which is advantageous in that the work load on the user side can be further reduced.
(Three-dimensional search section)

The three-dimensional search section is a member for performing a three-dimensional search for identifying the posture and position of each workpiece from the input image using the search model registered by the search model registration section. Prior to the three-dimensional search, the search model registration unit performs a three-dimensional search for specifying the orientation and position of each workpiece included in the input image from the input image showing the state in which a plurality of workpieces are randomly stacked. A search model that virtually expresses the three-dimensional shape of the work used for the work is registered in advance. In this state, the three-dimensional pick determination unit 8l determines whether the end effector is positioned at the gripping position specified for the work by the work-side gripping position specifying unit for the search result in the input image searched by the three-dimensional search unit. to determine whether it can be grasped. For example, an input image acquisition unit acquires an input image having a three-dimensional shape from images of a plurality of workpieces measured by a sensor unit, and from this input image, a three-dimensional search unit registers a search model in a search model registration unit. A three-dimensional search for specifying the posture and position of each workpiece is performed using the search model. As a result, it becomes possible to perform a three-dimensional search on an input image obtained by actually imaging a workpiece, and to perform gripping determination that is more in line with the actual situation.
(search model)

In bulk picking, it is necessary to first extract each workpiece from among a plurality of randomly stacked workpieces in order to determine a grippable workpiece. Here, the shape of a workpiece to be searched is registered in advance as a workpiece model for the shape of a workpiece group having height information obtained by the sensor unit, and a three-dimensional search is performed using this workpiece model. Detects the position and orientation of the
(basic direction image)

A search model for three-dimensionally searching a workpiece is created using a height image when the workpiece is viewed from a specific direction. For the height image used as the search model, three-dimensional CAD data, which is a work model expressing the work three-dimensionally, or actual measurement data obtained by actually imaging the work with the sensor section can be used. Here, an example of registering three-dimensional CAD data as a search model will be described. For example, consider a case where a workpiece model CWM constructed from three-dimensional CAD data as shown in FIG. 7 is used as a search model. From this workpiece model CWM, six height images viewed from the positive direction and negative direction of each of the three axes (for example, X-axis, Y-axis, Z-axis) orthogonal to each other in the virtual three-dimensional space, that is, Six views are acquired as cardinal direction images. For example, the basic direction image generating section 8e' in FIG. The six basic direction images are, for example, "top", "bottom", "left", "right", "front", and "back" of the work model CWM, that is, a plan view, a bottom view, a left side view, and a right side view. The basic direction image generating unit 8e' automatically calculates from the origin of the workpiece model CWM (details will be described later) and the surfaces forming the workpiece model CWM so as to obtain a view, a front view, and a rear view. Here, "top" is the height image seen from the positive direction (+ side) of the Z-axis, "bottom" is the height image seen from the negative direction (- side) of the Z-axis, and "left" is the negative direction of the X-axis. Height image seen from the direction, "Right" is the height image seen from the positive direction of the X-axis, "Front" is the height image seen from the negative direction of the Y-axis, "Back" is the height image seen from the positive direction of the Y-axis Viewed height images are shown, respectively. However, these are only examples, and different coordinate systems may be used, and the positive and negative directions of each axis are based on a coordinate system that is orthogonal to this axis, with the straight line of X=Y on the XY plane as an axis. You may use the height image seen from. In addition, when generating a height image from three-dimensional CAD data, it is necessary to start from a direction orthogonal to the CAD axis ("up", "down", "left", "right", "front", "back"). The height image does not have to be the viewed height image. For example, the posture (viewpoint) of the work model may be arbitrarily changed, and the height image may be generated from the changed viewpoint.
(Origin of work model)

Here, the origin of the work model is automatically determined by the image processing apparatus from the coordinate information of the three-dimensional CAD data. For example, for the three-dimensional CAD data of the work model CWM in FIG. 7, a virtual rectangular parallelepiped IBX circumscribing the work model CWM is defined as indicated by the dashed line in FIG. set as
(Basic direction image selection unit 8e)

Note that the number of basic direction images does not necessarily have to be six, and at least a plurality of images will suffice. For example, if the opposing surfaces have the same shape, such as a rectangular parallelepiped, a basic direction image viewed from one of the surfaces will suffice. In other words, the processing load of the three-dimensional search can be reduced by excluding basic direction images that have the same shape. Such a function of deleting a basic direction image whose appearance is common to any of the basic direction images is realized by the basic direction image selection section 8e. As an example, basic direction images obtained from the work model CWM of FIG. 7 are shown in FIGS. 9A-9D. In these figures, FIG. 9A is a height image of the work model CWM of FIG. 7 viewed from the positive direction of the X axis, FIG. 9D is a height image viewed from the positive direction of the axis, and FIG. 9D is a height image viewed from the negative direction of the Z axis. Here, as the height image, a height image expressing height information as a luminance value is used so that the higher the point, the brighter the point, and the lower the point, the darker the point.

Here, the match/mismatch of appearance is a total of six height images viewed from the top and bottom (± directions of the Z axis), front and back (± directions of the Y axis), and left and right (± directions of the X axis). is generated and checked by matching. In this case, rotation matching is confirmed at increments of 90°, and surfaces that appear to match other surfaces are excluded from search model registration targets. In the workpiece model CWM of FIG. 7, the height image viewed from the negative direction of the X-axis and the height image viewed from the negative direction of the Y-axis are in a state of appearance corresponding to FIGS. 9A and 9B, respectively. Therefore, it is excluded from search model generation. In this way, search models are generated for the number of faces with different appearances. The basic direction image from which the unnecessary height image is excluded in this way is displayed in the hexahedral view display area 3a. It should be noted that the six-view display area is not limited to its name, and it is not necessary to display all six views of the workpiece model. In this specification, it is called a six-view display area, including the mode of displaying in .
(Search model registration screen 130)

Also, such exclusion can be manually performed by the user, automatically performed by the image processing apparatus, or a combination thereof.

For example, in the search model registration screen 130 for registering a basic direction image BDI as a search model for three-dimensional search shown in FIG. and displayed in the six-view display area 3a of the display unit 3. At this time, if there is a common basic direction image, the user is urged to exclude it from the registration target of the search model. Here, as a search model selection section 8i for setting whether or not to register a search model, a "registration target" check box 131 is provided for each basic direction image BDI that is a search model candidate. When the user turns on the "registration target" check box 131, this basic direction image BDI can be registered as a search model, and when the user turns off the "registration target" check box 131, this basic direction image can be excluded from the search model. can.

At this time, the search model registration screen 130 is displayed to the user with the "registration target" check box 131 unchecked in advance by the basic direction image selection unit 8e for the basic direction images having a common appearance. After confirming that the basic direction image BDI to be registered in the search model and the basic direction image to be excluded from the search model are correctly selected as an initial state, the user approves this selection, or if necessary can be modified or replaced. In this manner, by selecting a basic direction image to be registered as a search model for a three-dimensional search by default and by presenting candidates for a basic direction image to be excluded from registration, the search model registration work by the user can be labor-saving. can.
(actual measurement data MWM)

An example of using three-dimensional CAD data as a search model for three-dimensional search has been described above. However, as described above, the present invention does not necessarily limit the search model to three-dimensional CAD data. It is also possible to use the captured actual measurement data as a search model. As an example, FIG. 11 shows actual measurement data MWM obtained by imaging a workpiece corresponding to the workpiece model CWM of FIG. 7 from the positive direction of the X axis. In this way, if there is no CAD data for the work, it is possible to use data obtained by three-dimensionally measuring the actual work. As shown in FIG. 10, the number of measured data MWM required for the three-dimensional search is imaged and registered.

When registering an actual work, the information on the bottom surface where the work is placed (for example, the shape of the floor around the work) is also three-dimensionally measured. Therefore, it is preferable to exclude unnecessary information on the bottom surface by, for example, cutting out only portions having a certain height or more from the bottom surface by threshold processing. This makes it possible to register only the shape portions necessary for three-dimensional search.
(Extraction of feature points)

Next, in a state in which the height images corresponding to the respective surfaces of the workpiece to be the search model are registered in this manner, a search model of the registered surfaces is generated. Here, feature points necessary for three-dimensional search are extracted for each registered surface. Here, an example using two types of feature points, that is, feature points representing the outline of the shape (feature points on the contour) and feature points representing the surface shape (feature points on the surface) will be described. FIG. 12A shows a state in which two types of feature points are extracted from the search model SM of the height image (corresponding to FIG. 9A) viewed from the X-axis direction. A perspective view of the search model SM is shown in FIG. 12B. Here, the feature points SCP on the surface and the feature points OCP on the contour are preferably displayed in different display modes. For example, the feature points SCP on the surface are displayed in white, and the feature points OCP on the contour are displayed in light blue. Alternatively, other color schemes may be used, such as displaying in purple to make it easier to distinguish from the white of the feature points on the surface. By displaying the feature points in different colors in this manner, the user can easily visually distinguish the meaning of each feature point.

The feature points SCP on the surface are extracted, for example, at regular intervals on the surface of the work model. On the other hand, for feature points OCP on the contour, for example, edge extraction is performed on locations where the height changes, and the thinning-processed locations are extracted as feature points at regular intervals. Thus, each feature point is a feature representing the three-dimensional shape of the surface.
(Three-dimensional search method)

A three-dimensional search is performed using a search model from which feature points are extracted in this way. Here, a method of performing a three-dimensional search for extracting each work from a state in which a work group in which a plurality of works are randomly stacked as shown in FIGS. 13A and 13B is imaged as an input image and a three-dimensional shape is acquired. will be explained. First, the input image is searched for the position and orientation (X, Y, Z, R _X , R _Y , R _Z ) that best match each feature point of the search model. Here, R _X , R _Y , and R _Z represent the rotation angle with respect to the X axis, the rotation angle with respect to the Y axis, and the rotation angle with respect to the Z axis, respectively. Various methods for expressing such a rotation angle have been proposed, and ZYX system Euler angles are used here (details will be described later). Also, the number of matching positions and orientations does not have to be one for each search model, and a plurality of matching positions and orientations above a certain level may be detected.

Here, as an input image, a three-dimensional search is performed with respect to a two-dimensional display as shown in FIG. 13A, or a work group such as that shown in FIG. As a result, a two-dimensional display as shown in FIG. 13C or a searched image as shown in FIG. 13D, which is three-dimensionally displayed, is obtained as a three-dimensional search result. As shown in FIGS. 13C and 13D, it can be seen that the feature points of the search model are retrieved from the input image, and the workpiece corresponding to the search model is detected. FIG. 13D shows how the search results of search model A and search model B are obtained. These search models A and B correspond to the search models A and B displayed in the search model selection column of FIG. 14, which will be described later. For the work WK on the right side of FIG. 13D, two search results of search models A and B are obtained for the same work. Therefore, if the workpiece can be gripped at the gripping positions registered in each search model, a plurality of gripping positions are obtained for this workpiece.

As a search model used for 3D search in this way, by using an image of the work viewed from each side like a 6-sided view, the arithmetic processing of 3D search is simplified compared to the case of using a perspective view, etc. It has the advantage of lightening the processing load and speeding up processing. In addition, the displayed state becomes easy to see even in the registration work of the search model, and the user can visually understand it easily.
(Evaluation index for three-dimensional search results)

Furthermore, it is also possible to set an evaluation index for the three-dimensional search results. For example, in the examples shown in FIG. 13C and FIG. 13D, how many corresponding feature points exist in the input image , score the 3D search result by deducting the error amount of the feature point according to the prescribed calculation formula as a penalty, etc. With this method, when there is a large amount of invalid data (invalid pixels) that could not be measured in 3D, The score becomes lower.In this way, the score can be used as an index representing the reliability of the three-dimensional search result.For example, it can be set so that workpieces are preferentially gripped in descending order of score. 3D search results with scores below the score may be determined to be highly likely to be erroneously detected, and may be set to be excluded from gripping targets for workpieces.For example, in the image processing apparatus 100 shown in FIG. An evaluation index calculation unit 8q is provided to calculate an evaluation index for the search results based on a predetermined criterion, thereby setting priority in descending order of the search results with the highest evaluation index, and gripping the workpiece according to this priority. For example, it can be set so that among the results with a certain score or higher, the one with the highest position in the Z direction is preferentially gripped. , it is difficult to interfere with other workpieces. Therefore, by setting the workpiece to be gripped with priority from the highest position in the Z direction, the effect of reducing the processing load for interference determination can be obtained.

By detecting each workpiece from the randomly stacked workpiece group, the image processing apparatus can recognize the workpiece to be gripped. Next, in order to grip a workpiece with an end effector, it is necessary to recognize which position of the workpiece and in what posture to grip each workpiece. Therefore, the workpiece gripping position is registered. In this specification, the terms "gripping position" and "gripping orientation" are used to include the position at which the workpiece is gripped and the orientation at that time. One or more workpiece gripping positions can be registered with respect to the workpiece. Moreover, it is preferable to register the gripping position for each surface of the workpiece in terms of easiness of the gripping registration work and recognition of the gripping position. That is, the gripping position is registered after the posture of the workpiece is defined as a posture in which a specific surface is the upper surface.
(Grip registration screen 140)

FIGS. 14 and 15 show examples of user interface screens for gripping registration for registering a gripping position where the end effector model grips the work model. On the gripping registration screen 140, the surfaces of the work model to be registered are specified, and the gripping posture is registered for each surface. The example of FIG. 14 shows a user interface screen for designating four types of surfaces (each surface of the search model) and registering the gripping posture. Here, search model "C" is selected from search models A to D, and three gripping postures are displayed in this search model C. FIG.

In the grip registration screen 140 of FIG. 14, an image display field 141 for displaying an image is provided on the left side, and an operation field 142 for performing various operations is provided on the right side. The image display field 141 displays the work model CWM and the end effector model EEM. By dragging the screen of the image display column 141, the viewpoint can be changed. Thus, it functions as a positioning unit 8c that adjusts the position and orientation of the work model displayed in the display area in the virtual three-dimensional space. Also, the display mode of the image display field 141 can be switched between two-dimensional display and three-dimensional display. In order to show the current display mode in an easy-to-understand manner, a three-dimensional reference coordinate axis BAX is superimposed on the image display column 141 and displayed. In the example of FIG. 14, the image display column 141 displays the grasping orientation registered in the grasping orientation 001 selected in the operation column 142, that is, the manner in which a part of the work model CWM is grasped by the end effector model EEM. ing. In this example, the entire end effector model EEM is displayed, but it is not necessary to display the entire end effector model at the time of gripping registration. is enough.

The operation field 142 is also provided with a search model selection field 143 for selecting a search model, and a grip posture display field 144 indicating the grip posture registered for the search model selected in the search model selection field 143. It is When a gripping posture listed in the search model selection field 143 is selected, the corresponding registered gripping posture is displayed in the image display field 141 . Also, by pressing an edit button 145, it becomes possible to correct the registered gripping posture. Further, the operation field 142 is provided with an add button 146 for adding a gripping posture and a delete button 147 for deleting a registered gripping posture. When the delete button 147 is pressed, the selected gripping posture is deleted.
(Gripping posture addition screen 150)

If you want to add a new gripping posture, press the add button 146 . As a result, a gripping posture addition screen 150 is displayed as shown in FIG. 15, and a gripping posture can be added. The image display field 141 displays the end effector model EEM and the work model CWM. The operation field 142 also displays gripping posture coordinate information 153 that defines the gripping posture. Here, the position parameters X, Y, Z, R _X , R _Y , and R _Z displayed as the gripping orientation coordinate information 153 represent the position and orientation data of the end effector model EEM with respect to the origin of the search model. ing. The origin of the search model can be the center of gravity of the work model CWM or the central coordinates of the CAD data, as described above.

When the values of X, Y, Z, R _X , R _Y , and R _Z of the gripping posture coordinate information 153 are changed, the position and posture of the end effector model EEM three-dimensionally displayed in the image display field 141 are changed accordingly. updated accordingly. Conversely, by dragging and moving the end effector model EEM in the image display field 141, the display content of the gripping posture coordinate information 153 in the operation field 142 is updated to the gripping posture coordinates after movement. As a result, the user can register the gripping position and orientation while confirming the end effector model EEM in the image display field 141 and the gripping attitude coordinate information 153 in the operation field 142 . Also, in the image display field 141, the three-dimensional reference coordinate axis BAX can be superimposed and displayed.

When registering the gripping position, for the end effector model EEM displayed in the image display field 141, the gripping reference point corresponding to the position at which the work model CWM is gripped and the gripping direction in which the work model CWM is gripped by the end effector model EEM. stipulate. It is preferable that the gripping reference point and gripping direction are set as default values on the image processing apparatus 100 side. Then, on the gripping posture addition screen 150 of FIG. 15, the position of the end effector model EEM is changed according to the position and posture of the work model CWM so that the gripping direction is orthogonal to the work plane representing the posture of the work model CWM. and adjust posture automatically. This adjustment can be made, for example, by the three-dimensional pick determination unit 8l. Alternatively, the user may manually adjust the position and orientation of the end effector model EEM using the positioning unit 8c so that the gripping direction of the end effector model EEM is orthogonal to the work plane of the work model CWM. good too.

Furthermore, when registering the gripping position, in the initial state where the gripping posture addition screen 150 of FIG. The initial value is the state below . In this way, the gripping posture can be determined by lowering the end effector model EEM so as to contact the workpiece model CWM. It is possible to reduce the problem of troublesome alignment. That is, basically, the grip orientation can be easily registered simply by aligning the end effector model EEM in the X, Y, and Z directions and setting the rotation about the Z axis.

Furthermore, by setting a gripping position to be gripped by the end effector model EEM with respect to the work model CWM, the position of the end effector model EEM is automatically adjusted so that this gripping position is located on an axis extending from the gripping direction. It can also be configured to adjust to

Note that the registration of the gripping posture is performed based on the image of the workpiece. For this reason, it is not always necessary to use CAD data, and as described above, it is possible to register the gripping posture with respect to actual measurement data obtained by actually imaging the workpiece.
(fit function)

Furthermore, when specifying the gripping position, not only the user manually moves the end effector model to the gripping position of the workpiece model, but also a fitting function for automatically performing this can be provided. Conventionally, when specifying gripping positions X, Y, Z and gripping postures R _X , R _Y , R _Z at which a work model is gripped by an end effector model, the end effector model is dragged in the image display field 141 of FIG. 15, for example. In the operation field 142, the user inputs a numerical value for Z in the height direction. However, it is troublesome for the user to visually move the end effector model to match the posture of gripping the workpiece model. It's hard to tell if it's good or not. Therefore, a fitting function is provided to automatically position the end effector model at the gripping position of the work model.

Here, if the work model or the end effector model has height information such as three-dimensional CAD data, a position on the work model to be gripped is specified by clicking the mouse, etc., and the height of this position is specified. A direction, that is, a Z coordinate is obtained, and a position obtained by adding an offset to the Z coordinate is set as the Z coordinate after movement of the end effector model. This eliminates the need for the user to manually move the end effector model to the gripping position and manually input position parameters such as the Z coordinate, and allows the gripping position to be specified accurately. As an example of such a fit function, a "fit" button 154, which is one aspect of the relative position setting section 8d5, is arranged in the operation field 142 shown in the gripping posture addition screen 150 of FIG. When the "Fit" button 154 is pressed, the end effector model EEM is virtually moved in the Z direction, stopped at the position where it contacts the work model CWM, and returned from this interference position by the offset amount in the opposite direction. , the gripping position. The offset amount is a numerical value for slightly separating the tip of the end effector model EEM from the work model CWM to interfere with or damage the work model CWM.

In the above example, a method of carrying out gripping registration using a height image used when generating a three-dimensional search model for performing a three-dimensional search has been described. By standardizing the search model for extracting workpieces and the model for registering gripping positions in this way, the user can perform three-dimensional search and gripping position settings for common workpieces, and can perform operations. A sense of unity is obtained and it becomes easy to understand intuitively. However, the present invention does not necessarily require that the search model for three-dimensional search and the target model for gripping registration match. If the correspondence relationship between the three-dimensional search model and the model used for gripping registration is known, it is not necessary to use the model used as the search model for gripping registration.

In addition, after the height image is generated from the three-dimensional CAD data, it is not necessary to limit the confirmation of matching of appearances to only the confirmation of the positive and negative directions of each axis. For example, a workpiece such as a cube looks the same when viewed from any of the ± directions of the X axis, ± directions of the Y axis, and ± directions of the Z axis. Good luck. Similarly, it suffices to register gripping positions by designating gripping for one surface. With this alone, it is possible to grip all surfaces (± directions of the X axis, ± directions of the Y axis, and ± directions of the Z axis). For example, for a randomly stacked cubic workpiece WKC as shown in FIG. 16A, one face of the cube is model-registered as shown in FIG. 16C, three-dimensional search for all six faces and grasping by the end effector model EEM is possible.
(Calibration)

In the gripping registration screen 140 described above, the relative position and orientation of the end effector model EEM when gripping a workpiece are registered with respect to the origin of the search model. On the other hand, when picking a workpiece with an actual end effector, the robot controller 6 uses the vision coordinates, which are the coordinates of the three-dimensional space (vision space) in which the workpiece is imaged by the sensor unit, to actually drive the robot. Need to convert to robot coordinates. Specifically, the position and orientation of the workpiece obtained as a result of the three-dimensional search are determined by the position (X, Y, Z) and the orientation (R _X , R _Y , R _Z ) in the vision space (the orientation ( R _X , R _Y , and R _Z ) indicate attitudes expressed by ZYX system Euler angles, which will be described later). Similarly, the orientation of the end effector that grips the end effector can also be obtained as the position (X, Y, Z) and orientation (R _X , R _Y , R _Z ) in the virtual three-dimensional space of the image processing device. In order for the robot controller 6 to drive the robot based on such a position and orientation in the vision space, the position (X', Y', Z') and orientation (R _X ', R _Y ') in the robot space are , R _Z ′). The process of calculating a transformation formula for coordinate transformation of the position and orientation calculated in the coordinate system displayed by such an image processing device into the position and orientation of the coordinate system in which the robot controller operates the end effector is called calibration. called a session.
(Embodiment 2)

An example of a robot system having such an image processing device (machine vision device) and robot calibration function is shown in the functional block diagram of FIG. 17 as a second embodiment. A robot system 2000 shown in this figure includes an image processing device 200, a display unit 3, an operation unit 4, a sensor unit 2, a robot controller 6, and a robot RBT. In the robot system shown in this figure, the same reference numerals are assigned to the members common to those shown in FIG.
(Image processing device 200)

The image processing apparatus 200 includes an input image acquisition section 2c, a storage section 9, a calculation section 10, an input/output interface 4b, a display interface 3f, and a robot interface 6b.

The input image acquisition unit 2c acquires an input image having a three-dimensional shape from the image including the end effector measured by the sensor unit. When the input image acquisition unit 2c acquires the input image of the end effector, it is preferable to capture the image so as to include the attachment position on the flange surface at the tip of the arm of the robot. In addition, an input image is captured in a posture that increases the area of the end effector. For example, the user operates the robot so that the end effector is set horizontally and an input image is captured from a sideways orientation.

The calculation unit 10 includes an end effector model registration unit 8u, a work model registration unit 8t, a calibration unit 8w, a search model registration unit 8g, a three-dimensional search unit 8k, a conversion unit 8x, and end effector mounting position calibration. A portion 8y and a gripping position specifying portion 8d.

The end effector model registration unit 8u is a member for registering an end effector model, which is three-dimensional CAD data that virtually expresses the three-dimensional shape of the end effector. For example, the end effector model registration unit 8u reads three-dimensional CAD data representing the shape of an end effector created separately and registers it as an end effector model. In this case, the three-dimensional CAD data input via the input/output interface 4b is registered as an end effector model in the end effector model registration section 8u. Alternatively, three-dimensional point cloud data obtained by imaging an actual end effector can be used as the end effector model. In this case, the three-dimensional point cloud data acquired by the sensor unit 2 and the input image acquisition unit 2c are registered as an end effector model by the end effector model registration unit 8u. Alternatively, three-dimensional CAD data simulating an end effector may be created and registered. In this case, the end effector model registration unit 8u implements a simple three-dimensional CAD function.

The calibration unit 8w converts the position and orientation calculated in the coordinate system of the vision space, which is a virtual three-dimensional space displayed on the display unit, to the position in the coordinate system of the robot space in which the robot controller operates the end effector. and a member for acquiring calibration information for coordinate transformation into posture.

For a plurality of position coordinates, the calibration unit 8w calculates a conversion formula between the actual position coordinates of the end effector EET of the robot and the position coordinates on the image displayed on the image processing device. The coordinate transformation method is not particularly limited, and known methods such as three-dimensional affine transformation can be used as appropriate.

The storage unit 9 stores calibration information from the calibration unit 8w.

The three-dimensional search unit 8k uses the end effector model as a search model from the input image acquired by the input image acquisition unit 2c to perform a three-dimensional search for specifying an image region corresponding to the position and orientation of the end effector. It is a member.

The conversion unit 8x converts the vision coordinates into robot coordinates based on the calibration information obtained by the calibration unit 8w. The conversion unit 8x reads the calibration information stored in the storage unit 9. FIG.

The end effector mounting position calibration unit 8y uses the information obtained by converting the position and orientation of the end effector in the vision space searched by the three-dimensional search unit into the position and orientation in the robot space by the conversion unit 8x. , calibrate the error between the position and orientation of the end effector in the vision space and the position and orientation in the robot space. As a result, by performing a three-dimensional search using the end effector model, it is possible to correct errors in the virtual end effector model according to the actual mounting state of the end effector, and perform more accurate setting work. become.

The gripping position specifying unit 8d specifies one or more gripping positions at which the work model registered by the work model registration unit 8t is gripped by the end effector model.
(ZYX system Euler angles)

Here, the ZYX system Euler angles will be described. Conventionally, ZYX system Euler angles are used for positioning the posture of a work model or an end effector model based on three-dimensional CAD data in order to define a gripping position for gripping a work by an end effector. In this case, the position and orientation of the end effector model with respect to the work model are expressed by six position parameters ( _X , Y, _Z , RX, _RY , RZ). Here, X, Y, and Z are orthogonal coordinate axes that define a three-dimensional space, and R _X , R _Y , and R _Z indicate rotation angles around the X, Y, and Z axes, respectively.

Here, an example of rotating the end effector model according to the ZYX system Euler angles will be described with reference to FIGS. 18 to 21. FIG. First, FIG. 18 shows a state in which the end effector model EEM and the workpiece model WM11 are displayed in the image display area of the display section 3. As shown in FIG. In this example, in order to make the explanation easier to understand, the reference coordinate axes BAX (XYZ) that define the virtual three-dimensional space and the three-dimensional space coordinate axes after the rotation object to be rotated (here, the end effector model EEM) is rotated. The rotated coordinate axes RAX (XYZ) are displayed respectively.

In the example of FIG. 18, the work model WM11 side is stationary and the end effector model EEM side is rotated. and posture, respectively. The origin of the rotated coordinate axis RAX on the end effector model EEM side coincides with the gripping reference point HBP, which is the gripping position of the end effector model EEM (details will be described later).

Furthermore, in the state before rotation, the rotation angles R _X , R _Y , and R _Z of the XYZ axes are respectively R _X =0°, R _Y =0°, and R _Z =0°, and the reference coordinate axis BAX and the rotated coordinate axis RAX are consistent. In this state, when the end effector model EEM is rotated 90 degrees counterclockwise around the Z axis, it becomes as shown in FIG. In the state of FIG. 19, the rotation angles R _X , R _Y , and R _Z are respectively R _X =0°, R _Y =0°, and R _Z =90°. The coordinate axis RAX is updated to the posture after rotation. Further, when the state shown in FIG. 19 is rotated 90 degrees clockwise about the Y-axis, the state shown in FIG. 20 is obtained. In this state, the rotation angles R _X , R _Y , and R _Z are R _X =0°, R _Y =90°, and R _Z =90°, respectively. is updated to Further, when the state of FIG. 20 is _{rotated clockwise} about the _{X -} axis by ₉₀ °, the state of FIG. _Z = 90°.

Conventionally, such ZYX system Euler angles have been used to express the generation and positions of workpiece models and end effector models. However, since the number of operable position parameters is as large as six, it is not easy to adjust these position parameters to achieve the position and posture expected by the user. In addition, since the XYZ axes used for moving the position while maintaining the posture are different from the R _X , R _Y , and R _Z axes used for rotation, it is intuitive how the position parameters will change. There was a problem that it was difficult to understand. This situation will be described with reference to FIG.

Now, in the state of FIG. 21, the rotated coordinate axes RAX, which are the orthogonal coordinate axes after the rotation angles R _X =90, R _Y =90, and R _Z =90, are displayed in the image display area. If R _Y or R _Z is changed from this state, the rotation will be performed around an axis different from the displayed rotated coordinate axis RAX. For example, the rotation of R _Y is not the _Y -axis indicated by the rotated coordinate axis RAX in FIG. will rotate around. Also, R _Z does not rotate around the Z-axis indicated by the rotated coordinate axis RAX in FIG. 22, but the Z-axis before the R _X and R _Y rotation indicated by broken lines (the Z-axis indicated by the rotated coordinate axis RAX in FIG. 18). will rotate around.

This is because the ZYX system Euler angles are defined on the premise that they are rotated in the order of the Z-axis, Y-axis, and X-axis. According to the ZYX system Euler angles, the Z axis is the reference, and the rotation of R _Z around the Z axis causes the rotation axes of R _X and R _Y to rotate. Here, the rotary shaft around the Z axis does not move, while the rotary shafts of R _X and R _Y are rotated by the rotation R _Z around the Z axis. Rotation by R _Y about the Y-axis thus determined causes the X-axis to rotate accordingly. Here, since the rotation axis around the Y axis is determined by the rotation R _Z around the Z axis, even if it is rotated by R _X around the X axis, both the rotation axis around the Y axis and the rotation axis around the Z axis change. do not do. In other words, in the ZYX system Euler angles, R _Z has an independent axis of rotation, R _Y has an axis of rotation dependent on R _Z , and R _X has an axis of rotation dependent on R _Y. It can be said that

In this way, in the ZYX system Euler angles generally used in conventional robot control, the three rotation axes are related to each other, and the axis after rotation about the other rotation axis is Since it rotates, it is extremely difficult for the user to grasp which axis is the rotation axis, and it is not easy to rotate in the intended direction.
(Display of correction rotation axis)

On the other hand, in the method according to the present embodiment, the rotation axes R _X , R _Y , and R _Z are displayed based on the actual rotation axes. This axis is different from the conventional display of three-dimensional space using the ZYX system Euler angles (Figs. 18 to 21). The actual rotation axis corrected in consideration of is displayed as the corrected rotation axis. For example, let us consider an example of displaying the correction rotation axis of R _Z in the state of FIG. 22 . In this case, as shown in FIG. 23, the Z-axis when calculated with R _X and R _Y at 0° becomes the correction rotation axis. The corrected rotational Z-axis AXZ displayed here is the Z-axis when R _X =0°, R _Y =0°, and R _Z =90°.

On the other hand, in order to display the corrected rotation axis of R _Y in the state of FIG. 22, the Y axis when calculated with R _X being 0° becomes the actual rotation axis, as shown in FIG. The corrected rotation Y-axis AXY displayed here is the Y-axis when R _X =0°, R _Y =90°, and R _Z =90.

Furthermore, in order to display the correction rotation axis of R _X in the state of FIG. 22, the X axis should be displayed as it is. FIG. 25 shows an example in which the X axis of correction rotation is displayed as the correction rotation axis. The corrected rotation X-axis AXX displayed here is the X-axis when R _X =90°, R _Y =90°, and R _Z =90°.

The starting point of these correction rotation axes is preferably the center of rotation of the end effector model. For example, the intermediate position between the pair of claws, which is the gripping position of the end effector model EEM.
(X-Y-Z Euler angles)

Even when the order of rotation is changed, the correction axis of rotation can be displayed with the same concept. For example, in the above-described examples of FIGS. 23 to 25, since the ZYX system Euler angles are adopted, rotation is performed in the order of R _Z , R _Y , and R _X . On the other hand, if the XYZ Euler angles are adopted, the rotation is performed in the order of R _X , R _Y , and R _Z , so the axis calculated as 0° is different from the ZYX system Euler angles. For example, in order to obtain the corrected rotation axis in XYZ Euler angles, assuming that the rotation is performed in the order of R _X , R _Y , and R _Z as in FIGS. 18 to 21, the Z axis is displayed as the correction rotation axis. To do so, display the Z-axis after rotation. To display the Y-axis as the correction rotation axis, the Y-axis when calculated with R _Z being 0° is displayed. Furthermore, in order to display the X-axis as a correction rotation axis, the X-axis when calculated with R _Z and R _Y set to 0° is displayed.

In this way, when the user sets the gripping position and posture using the Euler angles, the rotation axis can be displayed in an easy-to-understand manner, and the operation can be intuitively performed without understanding the complicated concept of the Euler angles. can.
(Embodiment 3)

It is not easy for the user to set the position and orientation of the end effector and workpiece defined by multiple position parameters such as Euler angles. Therefore, it is also possible to guide the setting procedure of the positional parameters so that the user is allowed to specify limited positional parameters among the plurality of positional parameters, and to sequentially set the necessary positional parameters according to the guidance. FIG. 26 shows such an example as a robot system 3000 according to the third embodiment. A robot system 3000 shown in this figure includes an image processing device 300, a display section 3B, an operation section 4, a sensor section 2, a robot controller 6, and a robot RBT. In the robot system shown in this figure, members common to those in FIG. 6 and the like are denoted by the same reference numerals, and detailed description thereof will be omitted as appropriate.
(Image processing device 300)

The image processing apparatus 300 includes an input image acquisition section 2c, a calculation section 10, a storage section 9, an input/output interface 4b, a display interface 3f, and a robot interface 6b. The storage unit 9 has a gripping position storage unit 9b. The gripped position storage unit 9b is a member for storing the gripped position of the work model or the end effector model specified by the gripped position specifying unit 8d.

The calculation unit 10 includes a gripping position specifying unit 8d, a gripping position copying unit 8d8, a relative position setting unit 8d5, a search model registration unit 8g, a three-dimensional search unit 8k, a three-dimensional pick determination unit 8l, and a cross-sectional model. and a generator 8s.

The gripping position specifying unit 8d specifies the position and orientation of either or both of the work model and the end effector model displayed in the virtual three-dimensional space on the display unit. The six position parameters of X, Y, and Z coordinates, which are coordinate positions, and R _X rotation angle, R _Y rotation angle, and R _Z rotation angle, which are rotation angles about the X axis, Y axis, and Z axis. , are members for specifying each.

The gripping position copying unit 8d8 is a member for reading the gripping position of the work model or the end effector model stored in the gripping position storage unit 9b, changing the gripping position, and registering it as a new gripping position. As a result, when registering a plurality of gripping positions, the gripping positions that have already been registered can be read out, the gripping positions can be changed based on this, and the gripping positions can be registered as new gripping positions. Gripping positions can be added more easily than in the case of registration, and labor saving in registration work can be achieved.

The three-dimensional pick determination unit 8l determines whether or not the workpiece model can be gripped by the end effector at the gripping position specified by the gripping position specifying unit 8d for each search result searched by the three-dimensional search unit 8k. It is a member for determining whether

The cross-sectional model generating unit 8s generates cross-sections obtained by cutting the end effector model along a linear basic axis set in one direction with respect to the end effector model on a plane perpendicular to the basic axis, and orthogonal cross-sections including each cross-section. A cross-sectional model composed of a plurality of pairs of cross-sections and cross-sectional positions is generated with cross-sectional positions set at points where the plane intersects the basic axis. Polygon data or the like is used for the end effector model for creating this cross-sectional model.
(Grip position specifying unit 8d)

The grip position specifying portion 8d shown in FIG. 26 includes a first specifying portion 8d9, a second specifying portion 8d6, and a third specifying portion 8d7. The first designating portion 8d9 is a member for which at least one of the X coordinate, Y coordinate, and Z coordinate can be designated, and other positional parameters cannot be designated. The second specifying unit 8d6 specifies at least the R _X rotation angle, the R _Y rotation angle, and the R _Z rotation angle with respect to the work model or the end effector model in a state in which some of the position parameters are specified by the first specifying unit 8d9. It is a member that can specify either one and cannot specify other position parameters. The third specifying unit 8d7 specifies the R _X rotation angle, the R Y rotation angle, the R X rotation angle, the R _Y rotation angle, and the It is a member that can specify a position parameter that is not specified by the second specifying section 8d6, and cannot specify other position parameters, of at least one of the R _Z rotation angles.

As a result, the gripping position is divided into a plurality of specification sections each limiting the parameters that can be specified, instead of allowing all six position parameters to be specified on a single screen, and the position parameters are specified individually in each section. This avoids a situation in which multiple position parameters are intertwined with each other, making it difficult to grasp the position and orientation. Further, by sequentially applying the position parameters, the information necessary for specifying the position and orientation can be set. becomes possible. In particular, the initial position of the gripping position in the first designating section 8d9 can be specified by clicking a mouse or the like on any part of the work model that is displayed in a plane on the display section. This makes it possible for the user to specify the work model in a visually easy-to-understand manner, making it much easier than the conventional complicated and troublesome tasks such as determining the posture and specifying numerical values. can. It should be noted that the work model is not necessarily displayed two-dimensionally in the first specifying section 8d9, and may be displayed three-dimensionally.
(Procedure for creating settings)

Here, the setting work for operating the robot system will be described with reference to the flow chart of FIG. 27 for the procedure of the teaching work performed before the actual operation.

First, in step S2701, a search model for three-dimensionally searching a workpiece is registered. Here, the search model can be registered as a work model, here three-dimensional CAD data, as described above. Alternatively, measured data obtained by actually imaging a workpiece with a sensor unit may be registered as a search model.

Next, in step S2702, the end effector model of the robot is registered. Also here, three-dimensional CAD data can be registered as an end effector model. Next, in step S2703, the surface of the work model to be gripped is selected from the height image. Next, in step S2704, the position and orientation of the robot when gripping the selected surface are registered. Next, in step S2705, it is determined whether or not the necessary number of positions and orientations have been registered. If not, the process returns to step S2703 to repeat the above processing. Then, when the required number of registrations has been completed, the processing ends.
(Procedure for registering three-dimensional CAD data in the search model)

Here, in step S2701, an example of a procedure for registering a search model of a workpiece when three-dimensional CAD data is used as the search model will be described with reference to the flow chart of FIG. First, in step S2801, a three-dimensional CAD data model of a workpiece is read. Next, in step S2802, the center of the circumscribed cuboid of the three-dimensional CAD data model is corrected to the origin of the three-dimensional CAD data. Further, in step S2803, height images viewed from each direction of "up", "down", "left", "right", "front", and "back" are generated. Here, when generating a height image from three-dimensional CAD data, the height image is generated so that the origin of CAD is at the center of the height image.

Next, in step S2804, height images having the same appearance are deleted from the generated height images. Finally, in step S2805, a search model is registered using the generated height image.

In this way, the user can register the search model used for the three-dimensional search according to the guidance.
(Registration of end effector model)

Next, details of the procedure for registering the end effector model in step S2702 of FIG. 27 will be described with reference to the flowchart of FIG. Here, it is assumed that the three-dimensional CAD data constituting the end effector model is composed of polygon data.

First, in step S2901, the polygon data of the end effector model are read. Next, in step S2902, the direction for creating the cross section is determined. Furthermore, in step S2903, a cross-sectional model is created. A cross-sectional model is created by the cross-sectional model creating unit 8s shown in FIG. The details of creating the cross-sectional model will be described later. Thus, the end effector model is registered in the storage unit 9. FIG.
(additional area)

Also, when registering the end effector model, an additional region can be added to the original end effector model. Such a procedure will be described based on the flowchart of FIG. First, in step S3001, an end effector model is registered. For example, an end effector model composed of three-dimensional CAD data such as STL data is read.

Next, in step S3002, an additional area is set. For example, when determining interference (details will be described later), the additional area adds a shape that imitates the shape of the actual end effector and the attached cover, harness, etc. to the surface of the end effector model. Accuracy of interference determination can be improved.
(Embodiment 4)

An additional model creation function is used to set such an additional region. An example of a robot system to which an additional model creation function is added is shown in the block diagram of FIG. 31 as a fourth embodiment. A robot system 4000 shown in this figure includes an image processing device 400, a display section 3, an operation section 4, a sensor section 2, a robot controller 6, and a robot RBT. In the robot system shown in this figure, members common to those in FIG. 6 and the like are denoted by the same reference numerals, and detailed description thereof will be omitted as appropriate.
(Image processing device 400)

The image processing apparatus 400 includes an input image acquisition section 2c, a calculation section 10, a storage section 9, an input/output interface 4b, a display interface 3f, and a robot interface 6b.

The calculation unit 10 includes a workpiece model registration unit 8t, an end effector model registration unit 8u, an additional model creation unit 8v, a gripping position identification unit 8d, a search model registration unit 8g, a three-dimensional search unit 8k, and an interference determination unit. A section 8m and a section model generation section 8s.

The additional model creation unit 8v is a member for creating an additional model by adding an additional region represented by one or more predetermined three-dimensional figures to the surface of the end effector model. As a result, without editing the 3D CAD data, interference judgment areas can be added to the end effector model with simple shapes such as cuboids and cylinders, and collision judgments can be made according to each area. Interference can be detected. Note that the three-dimensional figure includes not only basic figures prepared in advance by the image processing apparatus, but also figures that can be freely designed by the user.

The gripping position specifying unit 8d is a member for specifying one or more gripping positions at which the work model registered by the work model registration unit 8t is gripped by the end effector model.

The three-dimensional search unit 8k uses the search model registered by the search model registration unit 8g to specify an image area corresponding to the position and orientation of each workpiece from the input image acquired by the input image acquisition unit 2c. It is a member for performing a three-dimensional search.

The interference determination unit 8m determines whether or not there is interference with another object that may interfere with the operation of the end effector when the additional model created by the additional model creation unit 8v is used. It is a member for performing When moving the end effector to the position to grip any of the plurality of works included in the input image acquired by the input image acquisition unit 2c, the interference determination unit 8m detects that an object around the work is an end effector. In order to determine whether or not the object interferes with the effector, interference determination is performed on the image of the object surrounding the work within the input image. Thereby, interference can be determined by comparing the end effector model with the input image. For example, the interference determination unit 8m determines the position and orientation of each workpiece searched from the input image by the three-dimensional search unit 8k, and identifies the workpiece specified by the gripping position specifying unit 8d in order to grip the workpiece. When arranging the end effector model at the gripping position, it is determined whether or not there is interference with an object existing around the workpiece.
(Additional area setting screen 310)

The additional regions are added to the end effector model constructed from 3D CAD data. FIG. 32 shows an additional area setting screen 310 for adding an additional area to the end effector model as one aspect of the additional model creating section 8v. This additional area setting screen 310 can read an end effector model EEM, which is three-dimensional CAD data of an end effector, and add graphics to it. The additional area setting screen 310 has an image display field 141 and an operation field 142 . The image display field 141 displays the read three-dimensional CAD data of the end effector model EEM. A basic figure display field 311 is provided in the operation field 142 . A list of selectable basic figures is displayed in the basic figure display field 311 . Basic figures include rectangular parallelepipeds, cylinders, cones, triangular pyramids, hexagonal prisms, and the like. The user can select a desired basic figure from the basic figure display field 311 and press the "Add" button 312 to add the basic figure to the specified position on the end effector model EEM. Each line displayed in the basic figure display column 311 is provided with an "edit" button 313. By pressing this button, the settings of the selected basic figure can be changed. For example, basic graphic parameters such as the length of one side of the bottom surface, the radius, and the height can be adjusted.

In the example of FIG. 32, a rectangular parallelepiped is selected as the basic figure to be added, and is highlighted in the basic figure display field 311 . When the "edit" button 313 is pressed in this state, the basic figure editing screen 320 of FIG. 33 is displayed. The operation column 142 of the basic figure editing screen 320 is provided with a basic image parameter adjustment column 321, from which the size of the basic figure and the position and orientation of the end effector relative to the reference position can be changed. When the basic graphic parameters are adjusted from the basic image parameter adjustment column 321, the display contents of the image display column 141 are updated accordingly.

In the example of FIG. 32, the additional area ADA is inserted between the flange surface FLS at the tip of the arm and the end effector model EEM. As a result, it is possible to reproduce a state in which a mounting jig or the like is interposed between the arm portion and the end effector. In addition, when the flange surface at the tip of the arm portion deviates from the surface set for calculation on the image processing device side, it can be used to offset the amount of deviation.

You can also add multiple shapes. In the example of FIG. 32, various basic figures can be added by selecting a desired basic figure from the basic figure display field 311 and pressing the "Add" button 312. FIG. This makes it possible to express various shapes by combining a plurality of basic figures.

In this way, by providing the function of setting an additional region for the end effector model, it is possible to create a basic model prepared in advance without changing the shape by directly editing the shape of the three-dimensional CAD data as in the past. It is possible to easily add a shape just by adding a figure. As a result, when the end effector is placed at the gripping position for gripping the workpiece, in the interference judgment for confirming in advance whether or not there will be interference with objects around the workpiece, the end effector model does not actually exist. If there are any members that are similar to each other, it is possible to improve the accuracy of the determination result by using a shape that imitates them. For example, when an end effector is connected to the tip of an arm via a joint, the entire end effector may be offset and the tip may protrude slightly. Also, there may be additional elements such as a cover for preventing contact on the outside of the end effector, or a cable extending from the end effector. Such additional elements are often not included in the three-dimensional CAD data of the end effector model, and CAD data are rarely prepared separately. On the other hand, it is troublesome to process the three-dimensional CAD data of the end effector model and edit it so as to change the shape according to these additional elements. Further, even if a complicated shape can be represented by CAD data, if three-dimensional CAD data of such a complicated shape is used for interference determination, arithmetic processing becomes complicated and processing time becomes long.

Therefore, by expressing such additional elements graphically, it is possible to obtain an end effector model that is close to the actual situation with a simple additional area without going through troublesome editing work. It becomes possible to easily express the shape of the object. Also, with this method, since the interference determination area is added without changing the form of the CAD data, an increase in the processing time can be suppressed.
(Procedure for setting gripping position)

Next, in step S2704 of FIG. 27, the procedure for registering the gripping position and posture of gripping the workpiece with the end effector will be described with reference to the flowchart of FIG. 34 and FIGS. 35A to 38. FIG. Here, using a work model WM11 and an end effector model EEM configured by three-dimensional CAD data, the end effector model EEM for the work model WM11 is used to grip the work model WM11 while the work model WM11 is fixed. Six position parameters that define the position and orientation (hereinafter collectively referred to as "gripping positions"), that is, gripping positions X, Y, Z and gripping orientations R _X , R _Y , R _Z are shown in FIG. The settings are made in order according to the procedure shown in the flow chart.
(XY designation part)

First, in step S3401, the X and Y coordinates of the gripping position are designated. Here, a two-dimensional planar image obtained by projecting the workpiece onto a certain plane is displayed by the XY specifying section, and the grip position is specified on this planar image. FIG. 35B shows an XY designation screen 230 as one form of such an XY designation section. The XY designation screen 230 shown in this figure displays the three-dimensional CAD data of the work shown in FIG. In this state, the user is asked to specify which position of the workpiece model WM11F displayed in plan view is to be gripped by the end effector model. As shown in FIG. 35B, a desired position can be specified in an easy-to-understand manner without being conscious of the posture or rotation angle by selecting a gripping position on the plan view with a pointing device such as a mouse. In particular, by clicking to specify the position to be gripped on the work model WM11F displayed in a plan view, there is an advantage that it is easier to specify than adjusting the position by inputting numerical values. Then, the X and Y coordinates of the gripping position can be determined from the position manually specified by the user and the scale of the image. Note that an X, Y coordinate display column for displaying the X, Y coordinates as numerical values may be provided on the XY designation screen.
(Z-R _Z designation part)

Next, in step S3402, the gripping position Z and gripping orientation R _Z are designated from the ZR _Z designating section. FIG. 36 shows a _ZRZ designation screen 240 as one form of the _ZRZ designation section. A ZR _Z designation screen 240 shown in this figure comprises an image display field 141 and an operation field 142 . The operation field 142 is provided with a Z coordinate designation field 241 for designating the Z coordinate of the gripping position and an R _Z rotation angle designation field 242 for designating the R _Z rotation angle of the gripping posture. At this stage, since the X and Y coordinates have already been determined in step S3401, it is sufficient to display only the Z axis. Therefore, while the end effector model EEM and the work model WM11 are displayed in the image display column, only the corrected rotation Z-axis AXZ is superimposed and displayed as the corrected rotation axis. This makes it easier for the user to imagine how the end effector model EEM displayed in the image display field 141 will be rotated when the rotation angle is changed, and facilitates the position adjustment work of the end effector model EEM. You get the benefits you can get. In particular, by not displaying other rotation axes, the user can visually grasp the rotation direction without being confused. In addition, by making it impossible to specify other position parameters other than the Z coordinate and R _Z rotation angle, it eliminates the possibility of changing other position parameters by mistake, allowing the user to concentrate on setting only the specifiable position parameters. You can focus on only the necessary settings while eliminating confusion such as misunderstandings and misconfigurations.

As for the rotation of the end effector model EEM, when a numerical value is entered in the _RZ rotation angle specifying field 242, the end effector model EEM in the image display field 141 is automatically rotated and displayed accordingly. Further, when the end effector model EEM displayed in the image display field 141 is dragged and rotated, the value of the R _Z rotation angle displayed in the R _Z rotation angle specification field 242 is also changed accordingly.
( _RY designation part)

Furthermore, in step S3403, the gripping orientation R _Y is specified. Here, the R _{Y rotation angle is designated from the R Y rotation} angle designation field 251 using the R _Y designation screen 250 shown in FIG. In this example, the postures of the end effector model EEM and the work model WM11 are represented by ZYX system Euler angles. Therefore, by using the value of R _Z specified in step S3402, the direction of the rotation axis of R _Y can be corrected. The corrected rotational Y-axis AXY, which is the calculated corrected rotational axis, is displayed in the image display field 141 in the same manner as described above. By displaying the rotation axis in this way, the user can easily imagine how the end effector model EEM will change when the R _Y rotation angle is adjusted, which facilitates setting.
(R _X designation part)

Finally, in step S3404, the grip orientation R _X is designated. Here, the R _{X rotation angle is designated from the R X rotation} angle designation field 261 using the R _X designation screen 260 shown in FIG. 38 . Here, similarly, the corrected rotation X-axis AXX indicating the direction of the rotation axis of R _X is displayed in the image display field 141 .

In this manner, the user can designate six position parameters X, Y, Z, R _X , R _Y , and R _Z as the grip position for gripping the workpiece with the end effector. In particular, by limiting the adjustable position parameters for each screen and limiting the displayed rotation axis to display only the rotation axis related to the specified position parameter, the user can set according to such guidance. , the necessary positional parameters can be defined in turn. As a result, it is possible to smoothly perform the work of designating a grip position designating a three-dimensional position and orientation, which has been difficult in the past. Further, when setting the rotation angles R _X , R _Y , and R _Z of the Euler angles, the rotation axis is calculated and displayed in the image display field 141 as the correction rotation axis, so that the position parameter being set by the user can be calculated. By changing the , you can easily imagine how it works.

Furthermore, by dividing the gripping position registration into a plurality of steps and limiting the position parameters that can be adjusted in each step, it becomes possible to easily imagine how the robot will move when the position parameters are moved. In particular, by providing a designation screen for each step and registering different position parameters on different designation screens, the position parameters to be set can be presented to the user without confusion, and the setting work can be guided. In the examples of FIGS. 36 to 38, examples of providing different designation screens have been described, but the present invention is not limited to this example. For example, as shown in a positional parameter designation screen 270 in FIG. 39, it is possible to limit the positional parameters that can be designated for each step while using a common designation screen for designating a plurality of positional parameters. In the example of FIG. 39, only the X and Y coordinates can be designated from the X and Y coordinate designation field 271, and the other position parameters are grayed out so that they cannot be designated and cannot be selected. After specifying the X and Y coordinates, the user presses the "Next" button 272, and as the next step, another position parameter, for example, only the Z coordinate and the R _Z rotation angle is specified as the Z coordinate and the R _Z rotation angle. Other position parameters, including the specified X and Y coordinates, can be specified from the column 273, and are all grayed out or hidden so that they cannot be selected. Then, after specifying the Z coordinate and the R _Z rotation angle, by pressing the "Next" button 272 again, as the next step, another position parameter, for example, only the R _X , R _Y rotation angle, can be set to the R _X , R _Y rotation. It is possible to specify from the corner specification field 274, and other position parameters cannot be selected. Even with such a configuration, it is possible to realize a guidance function that restricts and presents the user with position parameters that can be specified and causes the user to sequentially set them.

Furthermore, the number and division of steps for sequentially defining position parameters are not limited to the above example. For example, the Z coordinate and the R _Z rotation angle may be set in separate steps, or the R _X rotation angle and the R _Y rotation angle may be set in the same step. For example, as an example of dividing the position step into a plurality of steps, the Z-R _Z designating part is composed of a Z designating part for designating the _Z coordinate and an R Consists of the _Z specification part. Alternatively, the R _X -R _Y designating part may be replaced by an R _X designating part for designating the R _X rotation angle about the X axis and an R _Y designating part for designating the R _Y rotation angle about the Y axis. It can also consist of parts. Alternatively, FIG. 37 and FIG. 38 may be integrated to form an R _X -R _Y designating section in which the R _X rotation angle and the R _Y rotation angle can be designated on one screen.

In the above example, the position and orientation of the end effector model are adjusted while the work model is fixed. and posture. Alternatively, the positions and orientations of both the end effector model and the work model may be adjusted.

In the above example, the six position parameters X, Y, Z, R _X , R _Y , and R _Z are set as X and Y coordinates in the XY designation section, and as Z and R _Z coordinates in the ZR _Z designation section. An example has been described in which the R _X and R _Y rotation angles are specified in order by the rotation angle and R _X -R _Y specifying section, but the order and combination of specifying each position parameter are not limited to the above configuration. For example, prepare an XYZ designation part, an R _Z designation part, and an R _X -R _Y designation part. Next, specify the R _Z rotation angle around the Z axis in the R _Z specification section, and finally specify the R _X rotation angle around the X axis and the Y axis in the R _X -R _Y specification section. You may specify the _RY rotation angle around . In this case, the XYZ designation unit can use, for example, an XYZ designation screen 280 as shown in FIG. , Y and Z coordinates can be specified from the X, Y and Z coordinate specification fields 281 . In this way, the image for first designating the gripping position is not limited to the planar image projected onto the XY plane as shown in FIG. 35B. It is also possible to specify using a perspective view WM11P projected from a direction.

Further, the gripping position may be specified in a non-projected state as shown in FIG. 35A without being limited to the method of specifying the gripping position from the image once projected onto the plane as in FIG. 35B described above. For example, while the workpiece model WM11 is displayed three-dimensionally on a three-dimensional image viewer 290 as shown in FIG. do.
(Modification)

Furthermore, in the above example, the procedure for specifying the six positional parameters X, Y, Z, R _X , R _Y , and R _Z according to the ZYX system Euler angles was explained, but the present invention can be applied to this aspect. Other aspects that define the position and orientation of the end effector and workpiece, such as XYZ system Euler angles, XZY system Euler angles, YXZ system Euler angles, YZ -X system Euler angles, ZXY system Euler angles, XYX system Euler angles, XZX system Euler angles, YXY Y system Euler angles, YZY system Euler angles , ZXZ system Euler angles, ZYZ system Euler angles, roll/pitch/yaw angle expressions, rotation axis/rotation angle expressions, etc. .

In the above example, only the correction rotation axis related to the position parameter to be specified is displayed as the correction rotation axis. not. For example, the same effect can be achieved by displaying three orthogonal axes including other correction rotation axes and emphasizing the rotation axis related to the rotation angle to be specified more than the other correction rotation axes. Obtainable. As an example, the rotation axis for which the rotation angle is specified is displayed in bold, highlighted by coloring or blinking, and conversely, the rotation axis that cannot be specified is grayed out or displayed with a thin line. , to distinguish them visually. Especially in 3D display of images, if there are orthogonal coordinate axes, the user can intuitively understand that 3D display is in progress. The distinction makes it difficult for users to get confused. Furthermore, it is preferable that the origin when displaying such three axes should be the center of rotation. For example, the intermediate position between the pair of claws, which is the gripping position of the end effector model EEM described above.
(Grip position copy function)

Furthermore, when registering a plurality of gripping positions, a gripping position copy function may be provided that reads the gripping positions that have already been registered, changes the gripping position based on this, and registers it as a new gripping position. . In general, a plurality of gripping positions are often registered for one workpiece. This is because if a plurality of gripping positions are registered, the optimum solution can be selected from a plurality of gripping solutions. This is because if there is a candidate, the possibility of being determined to be grippable increases. When registering a plurality of such gripping solutions, if the gripping registration is performed from the beginning each time, the number of steps increases and the work becomes troublesome when registering the same gripping position. Therefore, by copying an already registered gripping position, changing some of the position parameters set at this gripping position, and saving it as a new gripping position, it is possible to save time and effort and create a plurality of gripping positions. can be easily registered. Similarly, it is also possible to read the existing grip position, correct the position parameter as appropriate, and overwrite and save it.
(Grip position copy unit 8d8)

Such gripping position copying function and gripping position editing function are realized by the gripping position copying section 8d8 shown in FIG. The gripping position copying unit 8d8 reads the gripping positions of the workpiece model and the end effector model stored in the gripping position storage unit 9b, and changes and registers the position parameters that constitute the gripping positions. Also, when the gripping position is changed by the gripping position copying unit 8d8, if the order is registered according to the guidance as described above, it becomes easier to understand which position parameter should be changed in which step. Grasp registration becomes easier.

Furthermore, as described above, a fitting function may be added to automatically move the end effector model to the gripping position of the workpiece model when specifying the gripping position. For example, by providing a "fit" button 154 in the operation field 142 on the ZR _Z designation screen 240 in FIG. 36, the fit function can be executed to automatically set the Z coordinate.
(Embodiment 5)

Furthermore, in order to improve the accuracy of the three-dimensional search, it is also possible to limit the poses and angles that the search model can take. For example, let us consider detecting the position and posture of a work by three-dimensional search from a work group in which a plurality of plate-like works WK9 as shown in FIG. 43 are randomly stacked. In this case, the surfaces of the work model WM9 representing the work WK9 of FIG. 43 as three-dimensional CAD data are as shown in FIGS. 44A to 44F. 44A is a plan view of the workpiece model WM9, FIG. 44B is a bottom view, FIG. 44C is a front view, FIG. 44D is a rear view, FIG. 44E is a right side view, and FIG. 44F is a left side view. Duplicate faces are excluded from these basic direction images and registered as search models. Here, it is assumed that the four surfaces shown in FIGS. 44A, 44B, 44C, and 44E are registered. In this state, the group of randomly piled workpieces is imaged to obtain point group data having three-dimensional information, and a three-dimensional search is performed. As a result, the work may be erroneously detected as if it were in a vertically placed posture as indicated by X in FIG. Actually, it is difficult to imagine that a thin plate-like work exists in an upright position. 44A and 44B, for example, some of the shapes shown in FIGS. 44A and 44B, resulting in erroneous detection.

On the other hand, it is conceivable to perform a three-dimensional search by restricting the postures that the workpiece can take, but setting the conditions for restricting the postures is extremely difficult. For example, it is known that any posture of the workpiece can be represented by defining the rotation angle ranges of the Z-axis, Y-axis, and Z-axis using the ZYZ system Euler angles as shown in FIG. However, it is not easy to define the rotation angle of each axis by this method so that the posture range is desired by the user.

Therefore, in the fifth embodiment, the user can easily set possible postures of the workpiece without using such a troublesome concept. Specifically, when registering the search model, by excluding attitudes that are unlikely to occur when bulk loading, such surfaces are prevented from being detected. For such setting, the search model registration unit 8g shown in FIG. 6 is used. A search model registration unit 8g registers a group of randomly stacked works from any of the basic direction images as a search model for performing a three-dimensional search for specifying the position and orientation of each work. Select a cardinal direction image. The difference between the search model registration unit 8g and the basic direction image selection unit 8e is that the basic direction image selection unit 8e selects one of the basic direction images having the same appearance. On the other hand, the search model registration unit 8g is for excluding the basic direction image, which is not to be searched in the three-dimensional search, from the search model registration target. Preferably, the search model registration unit 8g excludes unnecessary basic direction images from search model registration targets in a state in which the basic direction image selection unit 8e excludes basic direction images that overlap in appearance. Alternatively, select a basic direction image to be searched. As a result, the number of candidates can be reduced and the search model registration work can be performed more easily by allowing the user to select basic direction images that are candidates for search model registration in advance.
(Search model registration screen 130B)

FIG. 47 shows an example of a search model registration screen 130B for registering a search model for the work of FIG. 43 described above. In the search model registration screen 130B shown in this figure, from the work model WM9 corresponding to the work WK9 in FIG. 44D and 44F in the example of FIG. 44, the remaining four basic direction images are displayed on the display unit 8e. It is displayed in the plan view display area 3a. In this state, as one form of the search model registration section 8g for setting whether or not to register the search model, a selection check box 131B for "Register" displayed on each basic direction image in the search model registration screen 130B is provided. there is If the user checks the selection check box 131B, this basic direction image is registered as a search model. Conversely, if the user unchecks the selection check box 131B, this basic direction image is excluded from search model registration targets. The user can look at the basic direction images displayed in the hexahedral view display area 3a, and can eliminate postures that are unlikely to occur in an actual random stacking state. In the example of FIG. 47, the selection check boxes 131B of model C and model D are turned off, and such a posture in which the side can be seen, that is, a posture in which the plate-shaped work stands upright can be excluded from the target of the three-dimensional search. can be done. As a result, the user can eliminate images that do not need to be searched or select only necessary images while viewing the basic direction image that shows the posture of the workpiece, without going through troublesome tasks such as angle calculation and range designation of the workpiece. As a result, it is possible to easily limit the posture of the workpiece to be searched in three dimensions.

Note that the search model registration unit 8g is not limited to the mode in which the user manually selects a search model to be registered or excluded. may be automatically extracted and eliminated.

Alternatively, a combination of automatic calculation and manual selection, for example, as shown in FIG. The selected basic direction image may be displayed with the selection check box 131B turned off as an initial state. The user can refer to the result of the automatic determination by the search model registration unit 8g, manually correct the selection as necessary, and press OK, so that the search model can be properly registered more reliably.

In FIG. 47, an example in which selection by the basic direction image selection unit 8e is automatically calculated has been described, but automatic determination by the basic direction image selection unit or manual selection can be similarly combined. For example, as a result of the determination by the basic direction image selection unit, one image that is determined to be a basic direction image having a common appearance is grayed out and displayed in the six-view display area so that the user can see the basic direction image. The result of automatic determination by the selection unit can be confirmed. If the determination result is wrong, the user can manually select the cardinal direction image and leave it as the cardinal direction image or, on the contrary, exclude it. In this manner, by displaying the result of automatic calculation and allowing the user to confirm it, it is possible to further improve the accuracy of image selection. In addition, by setting a state in which selection/non-selection is set in advance based on the result of automatic determination as an initial state, the user can approve the result of selection by automatic determination by pressing OK if the result is correct. Labor can be minimized.

Moreover, the purpose of excluding unnecessary surfaces from the three-dimensional search target can be applied to other than the upright state of the workpiece. For example, in a situation where a group of workpieces is given as an input image in such a manner that only a specific surface appears, in addition to an aspect in which workpieces are stacked completely randomly, erroneous detection can be avoided by excluding surfaces that do not appear. . In particular, it is effective for workpieces whose front and back surfaces are similar in shape and whose back surface is likely to be erroneously detected. In this way, by simply excluding the search model of the back side, which cannot actually be seen, from the target of the three-dimensional search, it is possible to obtain the same effect as restricting the posture of the work so that the back side of the work is not detected. .
(Procedure for registering a search model)

The procedure for registering the search model with the attitude restriction described above will be described with reference to the flow chart of FIG. Here, a procedure for registering three-dimensional CAD data of a workpiece as a search model will be described.

First, in step S4801, three-dimensional CAD data of a work is read. Next, in step S4802, the center of the circumscribed cuboid of the three-dimensional CAD data model is corrected to the origin of the three-dimensional CAD data. Further, in step S4803, height images viewed from each direction of "up", "down", "left", "right", "front", and "back" are generated. Here, when generating a height image from three-dimensional CAD data, the height image is generated so that the origin of CAD is at the center of the height image. Furthermore, in step S4804, height images that look the same are deleted from the generated height images.

Next, in step S4805, a search model to be used for three-dimensional search is selected from the remaining height images. Here, the search model selection unit 8i eliminates unnecessary basic direction images and selects necessary basic direction images.

Finally, in step S4806, the selected height image is registered as a search model. Here, the search model is registered by the search model registration unit 8g. In this way, by excluding basic direction images of unnecessary postures, it becomes possible to register a search model in a state in which posture restrictions are substantially provided.
(Embodiment 6)

The above is an example of limiting the three-dimensional search target using actual images instead of numerical values. However, the present invention is not limited to this aspect, and instead of or in addition to this, it is also possible to limit the orientation by the tilt angle or rotation angle of the search model. This makes it possible to appropriately set the three-dimensional search conditions according to how the workpieces are actually stacked, thereby reducing erroneous search detections. FIG. 49 shows an example of the tilt angle/rotation angle setting screen 160 for restricting the posture based on the tilt angle and the rotation angle as the sixth embodiment. The inclination angle/rotation angle setting screen 160 shown in this figure includes an attitude restriction search model selection field 161 for selecting a search model for which attitude restriction is to be performed, and an inclination angle upper limit setting field for specifying the upper limit of the angle to which tilt is allowed. 162, it has a rotation angle range setting field 163 for defining the angle range in which rotation is permitted.
(Tilt angle/rotation angle setting screen 160)

This tilt angle/rotation angle setting screen 160 does not require an esoteric method of designating a three-dimensional posture of the workpiece, for example, with R _X , R _Y , and R _Z indicating the rotation angles of the coordinate axes. As parameters that are easy to understand even without them, the "tilt angle" and "rotation angle" from the posture at the time of registration of the search model are specified. Here, the basic direction image registered as the search model is used as the registered orientation, and the tilt angle and rotation angle are specified from this state (details will be described later).

Specifically, in the tilt angle/rotation angle setting screen 160 of FIG. 49, in the attitude restriction search model selection field 161, the search model to which the attitude restriction is applied is selected. In the example of FIG. 49, the search model of model A registered on the search model registration screen 130B of FIG. 47 is selected.

In addition, in the tilt angle upper limit setting field 162, it is set whether or not the tilt angle within the registered posture is to be output as the result of the three-dimensional search.

Further, in the rotation angle range setting field 163, a reference angle and an angle range are set. First, in the reference angle setting field, the rotation angle with respect to the posture at the time of registration is entered. The rotation angle set here is the reference angle for designating the rotation angle range. Furthermore, in the range setting field, a rotation angle range to be output as a result of the three-dimensional search is set with respect to the reference angle set in the reference angle setting field.
(Position restriction method by tilt angle/rotation angle)

Here, a method of obtaining the tilt angle and the rotation angle from the three-dimensional posture of the search model will be described. In the method of limiting the surface to be registered as a search model in the above-described three-dimensional search, posture restrictions are automatically added. On the other hand, in order to add orientation restrictions to a plane registered as a search model, the fact that the plane is registered is used.

Here, the attitude is restricted by two angles, the "tilt angle" with respect to the registered attitude and the "rotational angle" with respect to the registered attitude viewed from directly above. With these two angles, the concept is easy to understand, and the angle can be limited relatively easily.
(tilt angle)

First, as shown in FIGS. 50A to 50C, the tilt angle is defined by the angle of the attitude of the work model at the time of registration as the search model with respect to the Z vertical direction. FIG. 50A shows the posture of the work model WMR at the time of registering the search model and the Z-axis defined as the vertical direction of this work model. On the other hand, it is assumed that the posture of the work WKI obtained as an input image and the Z'-axis defined as the vertical direction of this work WKI are in the state shown in FIG. 50B. In this state, the "tilt angle" is defined as the tilt angle between the Z-axis and the Z'-axis, that is, the vertical tilt of the workpiece, as shown in FIG. 50C. As a result, regardless of the direction of inclination, the inclination from the registered state of the search model can be represented by one angle. In addition, there is an advantage that both the user and the concept can be easily grasped.
(rotation angle)

On the other hand, the rotation angle is defined by the rotation angle when viewed from the vertical direction at the time of registration, that is, the Z-axis. Here, the rotation angle will be explained with reference to FIGS. 51A to 51C. In these figures, FIG. 51A shows the work model WMR at the time of registering the search model and the Y axis that defines the XY plane. On the other hand, it is assumed that the posture of the workpiece WKI obtained as an input image and the Y'-axis defining the XY plane are in the state shown in FIG. 51B. In such a state, as shown in FIG. 51C, the "rotation angle" is defined as the rotation angle when viewed from the vertical direction, that is, the Z-axis. In this example, the rotation angle is defined as the Y-axis angle viewed from the Z-axis direction, with the Y-axis coordinate axis that defines the workpiece model and the workpiece as a reference. By defining in this way, the same concept as the angle in the conventional two-dimensional pattern search can be obtained with the advantage that it is easy for the user to comprehend.

A known algorithm can be used to define the origin and the XYZ axes for the workpiece. For example, as shown in FIG. 8, a figure circumscribing the workpiece, such as a rectangular parallelepiped or a sphere, is calculated from the workpiece shape information possessed by the workpiece model or the input image, and the center of gravity is calculated to set this as the origin of the workpiece. Define an axis. As for the method of defining the XYZ axes, when CAD data is used, the directions of the coordinate axes of the original CAD data may be used as the directions of the respective XYZ axes. When using actual measurement data obtained by actually capturing a workpiece in 3D, the vertical direction of the actual measurement data is the Z axis, the Y axis is the upper direction of the actual measurement data in plan view, and the X axis is the right direction of the actual measurement data in plan view. and should be. As for the origin, the center of gravity of the workpiece model or the central coordinates of the CAD data may be used as the origin.
(Procedure for obtaining the tilt angle and rotation angle from the three-dimensional posture)

Here, a procedure for obtaining the tilt angle and the rotation angle from the three-dimensional posture will be described with reference to the flowchart of FIG. First, in step S5201, the tilt angle is obtained. For example, when the posture of the workpiece WKI obtained as the input image is as shown in FIG. The tilt angle is obtained as the tilt of the direction.

Next, in step S5202, the input image is three-dimensionally rotated so that the Z'-axis is aligned with the Z-axis. In other words, the tilt is eliminated. For example, as shown in FIG. 54A, the input image WKI of FIG. 53C is rotated around the outer product vector VP of the Z'-axis vector and the Z-axis vector so that the Z'-axis overlaps with the Z-axis. As a result, a rotated input image WKI' is obtained as shown in FIG. 54B, and the tilt is removed.

Finally, in step S5203, the rotation angle is obtained. For example, the Y-axis of the work model WMR at the time of registration shown in FIG. 55A and the Y'-axis of the input image WKI' in the state where the tilt is removed as shown in FIG. 55B are obtained. In this state, as shown in FIG. 55C, the angle between the Y-axis and the Y'-axis when viewed from the vertical direction, that is, the Z-axis, is obtained as the rotation angle. In this way, the tilt angle and rotation angle can be obtained from the three-dimensional posture of the input image.
(Setting the upper limit of the tilt angle)

As an example of setting the upper limit of the tilt angle on the tilt angle/rotation angle setting screen 160 in FIG. 49, for example, in the case of a highly glossy metal work, the range in which three-dimensional measurement cannot be performed increases as the surface tilts. If the three-dimensional search is performed in such a state, the risk of erroneous detection in the three-dimensional search increases. Therefore, by excluding postures tilted more than a certain amount, which have a risk of false detection, from three-dimensional search targets, it is possible to leave only highly reliable search results. For example, by setting the upper limit of the tilt angle to 40 degrees for a workpiece that tends to be erroneously detected when tilted by 40 degrees or more, search results tilted by 40 degrees or more can be excluded.
(Setting the upper limit of the rotation angle range)

On the other hand, as an example of setting the rotation angle range on the tilt angle/rotation angle setting screen 160 in FIG. 49, for example, model A and model B in FIG. Since there are many portions where the three-dimensional measurement cannot be performed due to the influence of multiple reflections, etc., there may be cases where the rotational posture is incorrect by 180 degrees. In addition, in the case where only a certain orientation occurs as the actual stacking of workpieces, by using this posture restriction, it is possible to eliminate the result of the opposite orientation. For example, if the rotation angle range is set to ±30 degrees with respect to the reference angle, only search results within this angle range are detected.
(End effector mounting position setting screen 170)

The method of defining the position and posture using the Euler angles described above can be used not only to define the position and posture of gripping a work model with an end effector model, but also to define the position at which an end effector is attached to the tip of a robot arm. . Here, the procedure for setting the position at which the end effector is attached to the tip of the arm of the robot will be described. The setting of such an end effector mounting position is performed on an end effector mounting position setting screen 170 as shown in FIG. The end effector mounting position setting screen 170 shown in this figure has an image display field 141 and an operation field 142 . An end effector model is displayed in the image display field 141 . The end effector model reads pre-created three-dimensional CAD data. Alternatively, an end effector may be modeled on a basic figure such as a rectangular parallelepiped or a cylinder and displayed.

The operation column 142 is provided with an end effector attachment position setting column 171 that defines the attachment position of the end effector attached to the tip of the arm. The end effector mounting position parameters to be set in the end effector mounting position setting field 171 are the mounting position ( _X , Y, _Z ) and mounting orientation (RX, _RY , RZ) of the end effector model. For example, it is defined using Euler angles with respect to the center of the flange surface FLS at the tip of the arm portion.

Note that on the registration screen for the work model gripping position shown in FIG. Registering relationships. On the other hand, on the end effector mounting position setting screen 170, the mounting position for mounting the end effector model on the tip of the arm is registered. Therefore, it is not necessary to display the work model or the like, and the image display field 141 displays the end effector model and the flange surface FLS.
(Grip position designation screen 180)

On the other hand, on the grip position specification screen 180 for specifying the grip position of the work model, as shown in FIG. 57, the end effector model and the work model are displayed, and the part to grip the work model is specified. A gripping position designation screen 180 shown in this figure has an image display field 141 and an operation field 142. In the image display field 141, a workpiece to be gripped is selected, and the surface to be gripped of this workpiece is directed upward. The image is drawn and displayed in a three-dimensional shape with the posture of the robot. Furthermore, the end effector model is displayed, and the position and posture for gripping the work model are adjusted. The operation field 142 is provided with a gripping position designation field 181 for designating a gripping position. The user adjusts the position parameters from the gripping position specification field 181 so that the relative relationship between the work model and the end effector model is correct. Alternatively, by dragging the end effector model or work model displayed in the image display field 141, the position and posture are adjusted on the screen. The position parameters adjusted on the screen are reflected in the gripping position designation field 181 . The above-mentioned Euler angles can also be used for notation of position parameters in the gripping position specifying field 181 . (Multi-holding position selection screen 190)

Furthermore, the registration of the gripping position is not limited to one position for one surface of one workpiece (for example, the basic direction image), and a plurality of positions can be designated as described above. In addition, it is also possible to confirm gripping positions registered at a plurality of locations in an image. For example, FIG. 58 shows an example of a multiple grip position selection screen 190 . This multiple gripping position selection screen 190 also has an image display field 141 and an operation field 142 . The operation field 142 is provided with a surface selection field 191 for selecting a surface of the workpiece model defining the grip position, and a grip position list display 192 for displaying a list of grip positions set on the surface selected in the surface selection field 191. be done. In this example, the surface C is selected in the surface selection field 191 , and gripping positions 1 to 3 are listed in the gripping position list display 192 as the gripping positions set on this surface C. FIG. When grasping 2 is selected among these, in the image display field 141, a state in which the end effector model grasps the workpiece model is displayed in the grasping position and posture registered in grasping 2. FIG. In this manner, the user can select and confirm a plurality of registered gripping positions. Also, it is possible to modify and update the registered gripping position as necessary. Here, too, the gripping posture can be displayed using the above-described Euler angles. In this way, even though the registration is performed using the Euler angles, the user can display the angles in a visually easy-to-understand manner without a detailed understanding of the concept of the difficult Euler angles, and correct the rotation axis. By displaying the information and limiting the parameters that can be adjusted using the guidance function, it is possible to proceed with the setting work with less confusion.
(Calibration function for end effector mounting position)

As described above, the end effector is attached to the tip of the arm of the robot. On the other hand, the registration (teaching) of the gripping position described above and the gripping determination (simulation) of whether or not the end effector model can grip the work model at the registered position and orientation are performed by the image processing device, not by the actual end effector. It is performed in the virtual three-dimensional space on the side (robot vision side). Transformation from the coordinate position in the vision space, which is the virtual three-dimensional space, to the coordinate position in the robot space, which is the real space, is performed by the conversion unit 8x based on the calibration information obtained by the calibration unit 8w in FIG. It is done.

However, if the mounting state of the end effector model set virtually on the image processing device side differs from the actual mounting state of the end effector EET of the robot RBT, the vision space and the robot space defined by the calibration information may differ. is no longer maintained, and a deviation occurs when an actual robot RBT is used to grip a workpiece.

FIG. 59 shows the positional relationship between the end effector model and the flange surface FLS. In this figure, the origin _{OF of the flange surface FLS at the tip of the arm of the robot and the center O E of} the end effector model EEM are shown as the position of the end effector model EEM. The position of the end effector model EEM is set as a position (X, Y, Z) and a posture (R _X , R _Y , R _Z ) with respect to the flange surface FLS at the tip of the arm portion of the robot on the image processing device side. there is That is, in the example of _FIG . 59, the position (X, Y, Z) and orientation (R _X , R _Y , R _Z ) of the center O _E of the end effector model EEM viewed from the origin OF of the flange surface FLS are set. It is Here, if an error occurs in the actual mounting state of the end effector, a deviation may occur in the position of the end effector model EEM with respect to the origin of the flange surface FLS. As a result, due to the mounting state of the end effector, the image processing apparatus side and the robot side grip the object at different positions.

Therefore, in the present embodiment, an end effector mounting position calibration function is provided to reflect the actual mounting state of the end effector on the image processing apparatus side so as to prevent such deviations and errors from occurring. The function of calibrating the end effector attachment position is performed by the end effector attachment position calibrator 8y shown in FIG. Specifically, using 3D CAD data that virtually represents the 3D shape of the end effector, which is the end effector model used for registration of the gripping position of the workpiece and interference judgment when the end effector is arranged. By performing a three-dimensional search on actual measurement data obtained by imaging and three-dimensionally measuring an actual end effector, and obtaining the position and orientation of the actual mounting position, such errors are detected and the image processing device Calibrate side end effector mounting position settings. In this way, the user can display the difference between the point cloud obtained by three-dimensionally measuring the actual end effector and the attachment state of the end effector model defined in the virtual three-dimensional space on the image processing device side on the image display area of the display unit. You can adjust manually while checking with .
(Procedure for automatic deviation correction)

Here, a calibration function for automatically correcting errors between an actual end effector and an end effector model configured by three-dimensional CAD data or the like will be described with reference to the flow chart of FIG. Note that the calibration unit 8w shown in FIG. It is assumed that the Furthermore, it is assumed that an end effector model is also prepared as three-dimensional CAD data.
(End effector imaging screen 330)

First, in step S6001, the sensor unit prepares for imaging the actual end effector. Specifically, the robot is operated to move the end effector so that the end effector is captured by a three-dimensional camera, which is one form of the sensor unit. Next, in step S6002, three-dimensional measurement is performed on the end effector.

An end effector imaging screen 330 shown in FIG. 61, for example, is used as one mode of the end effector imaging unit for imaging the actual end effector with such a sensor unit. The end effector imaging screen 330 has an image display field 141 and an operation field 142 . In the image display column 141, the end effector EET attached to the flange portion at the tip of the arm portion ARM imaged by the sensor portion 2 is displayed in real time. The operation column 142 is also provided with an end effector position specification column 331 indicating the position and posture of the end effector EET. From this end effector imaging screen 330, the user manually operates the robot so that the end effector EET of the robot is captured by the camera that constitutes the sensor section. At this time, what is imaged is not the claw portion of the end effector EET that grips the work, but the state of attachment to the arm portion. Therefore, it is desirable to take an image so that the area of the end effector EET becomes large. Preferably, the end effector EET is changed from the downward posture in which the workpiece is easily gripped to the sideways posture so that the entire image can be captured by the camera. In this way, the user presses the “imaging” button 332 after positioning the posture of the end effector EET. Thereby, a three-dimensional captured image of the end effector is acquired. In other words, a three-dimensional measurement is performed on the end effector.

Next, in step S6003, the position and orientation A of the flange portion in the robot coordinate system are acquired. Here, the position and orientation A of the robot coordinate system are, for example, the position and orientation of the flange portion FLS to which the imaged end effector is attached in the robot coordinate system. Note that the order of steps S6003 and S6002 may be interchanged.

Further, in step S6004, the position and orientation A of the flange portion in the robot coordinate system are transformed into the position and orientation B in the vision space. Here, the conversion unit 8x in FIG. 17 performs conversion between the real space and the virtual space based on calibration information for performing coordinate conversion between the robot space and the vision space obtained by performing calibration in advance.

Furthermore, in step S6005, a three-dimensional search is performed to detect the position and orientation C of the end effector in the vision space. Here, using the CAD data used for the end effector model as a search model, a three-dimensional search is performed on the three-dimensional measurement data obtained in step S6002. Known algorithms can be used as appropriate for the three-dimensional search method. Thereby, the position and orientation of the end effector on the vision space are detected.

Furthermore, in step S6006, the relative position and orientation of the position and orientation C of the end effector with respect to the position and orientation B of the flange portion FLS in the vision space are calculated. The coordinate position obtained here is the accurate position and orientation of the end effector model with respect to the flange portion FLS, in other words, the position and orientation of the end effector model taking into consideration the positional deviation and the like.

Finally, in step S6007, the obtained position and orientation are reflected in the end effector settings on the vision side. That is, it is reflected in the setting of the position and orientation of the end effector model on the vision side. In this way, the actual mounting state of the end effector can be automatically reflected on the vision side.
(Procedure for manual deviation correction)

The procedure for automatically correcting the deviation between the robot space and the vision space has been described above. However, the present invention is not limited to a configuration in which deviation correction is performed automatically, and can be performed manually. Next, a procedure for manually performing deviation correction will be described with reference to the flowchart in FIG. Here, steps S1501 to S1504 are the same as steps S6001 to S6004 in FIG. 60 relating to the automatic correction described above, and detailed description thereof will be omitted. That is, in step S1501, the robot is operated so that the end effector of the robot is captured by the three-dimensional camera. A is acquired, and in step S1504, the position and orientation A of the flange portion in the robot coordinate system are transformed into the position and orientation B in the vision space.

Next, in step S1505, the position and orientation D of the end effector are calculated from the position and orientation B of the flange portion in the vision space based on the end effector setting. Here, the position and orientation D of the end effector are obtained from the position and orientation B of the flange, which is the tip of the robot in the vision space, and information on the position and orientation of the end effector with respect to the flange set in the end effector setting.

Next, in step S1506, the three-dimensionally measured point cloud and the CAD display of the position and orientation D of the end effector are superimposed and displayed. Here, the end effector model EEM, which is three-dimensional CAD data of the end effector, is displayed at the position and orientation D of the end effector obtained in step S1505, and points obtained by three-dimensional measurement of the actual end effector are displayed. The group PC is superimposed and displayed. 151 and 152 show examples of deviation correction screens displayed in such a superimposed manner.

Then, in step S1507, it is determined whether or not there is an unignorable deviation between the point cloud and the CAD display. If there is a non-negligible deviation, the flow advances to step S1508 to change the end effector settings to correct the deviation, and returns to step S1505 to repeat the above-described processing. This end effector setting includes the position and orientation of the end effector relative to the flange. On the other hand, if there is no deviation, the process ends.
(deviation correction unit)

An example of a deviation correction screen is shown in FIGS. 151 and c as one aspect of the deviation correction unit that performs such deviation correction. In these figures, FIG. 151 shows the case where there is a non-negligible deviation. The deviation correction screen 840 shown in this figure superimposes the point cloud PC on the end effector model EEM in the image display field 141 . As shown in this figure, it can be confirmed that the point group PC, indicated by the white dots, obtained by actually three-dimensionally measuring the end effector is shifted in the lower left direction from the end effector model EEM.

An end effector setting field 841 is provided in the operation field 142, and defines the position ( _X , Y, _Z ) and orientation (RX, _RY , RZ) of the end effector with respect to the flange portion. Here, the longer XYZ coordinate axes shown on the right side of the drawing indicate the origin OF of the flange portion _FLS , which is the tip of the robot. The shorter XYZ coordinate axes shown on the left side represent the origin O _E of the three-dimensional CAD data of the end effector model EEM with respect to the tip of the robot (flange). The user adjusts the position and orientation of the end effector model EEM while checking the end effector model EEM and the point cloud PC displayed in the image display field 141 . For example, in the end effector setting field 841 of the operation field 142, the position and orientation of the end effector model EEM are designated numerically, or the end effector model EEM is dragged and moved on the image display field 141, and the end effector model EEM is displayed. is adjusted so that overlaps with the point cloud PC.

FIG. 152 shows an example of matching the end effector model EEM with the point group PC in this way. In this example, in the end effector setting field 841, the position in the Z direction is set to 90 mm, and the end effector model EEM is offset in the Z direction to match. After the position and orientation of the end effector model are adjusted, pressing the "OK" button 842 updates the end effector settings. Thereby, the position and orientation of the vision side can be adjusted according to the actual mounting state of the end effector.

Furthermore, the manual correction of deviations described above can be combined with the automatic correction. Such an example will be described based on the deviation correction screen 850 in FIG. In the shift correction screen 850 shown in this figure, an end effector setting field 841 for manually adjusting the position and orientation of the end effector model EEM is provided in the operation field 142, and an "automatic correction" button 843 is provided at the bottom. . When this "automatic correction" button 843 is pressed, a three-dimensional search is internally executed to automatically correct the position and orientation of the end effector model EEM so as to match the point group PC. For example, while confirming the state of the end effector model EEM and the point cloud PC superimposed in the image display field 141, it is possible to automatically correct the deviation or finely adjust the automatic correction result manually. In particular, if the end effector is made of highly glossy metal and the point cloud PC cannot be accurately detected due to reflected light, automatic correction may not work well, for example, if the 3D search fails, manually It can be adjusted. In this way, the user can manually correct the deviation between the robot space and the vision space.

Conventionally, even though a mounting member such as a connector is actually assembled to the mounting portion of the flange and the end effector, the vision side forgets to set the offset. , there was a problem that the position and posture were shifted. In such a case, if there is no function for checking or correcting the deviation, problems such as poor gripping or collision may occur when the robot grips the workpiece during actual operation. There are various factors that cause such an error, and conventionally, it took a lot of time and effort to investigate the cause and debug. On the other hand, according to the present embodiment, it is possible to manually or automatically correct such a deviation, and it is possible to easily perform flexible adjustment according to the actual mounting state of the end effector.

In the example of FIG. 59, the position of the end effector model EEM is the center position of the three-dimensional CAD data, but it is not limited to this example and other positions can be defined. For example, in FIG. 59, the center of the rectangular parallelepiped circumscribing the three-dimensional CAD data is used, but the end portion of the circumscribing rectangular parallelepiped or the origin of the input source three-dimensional CAD data may be used as the reference.
(Procedure for registering measured data in the search model)

The procedure for registering three-dimensional CAD data as a search model has been described above. However, as described above, the present invention is not limited to three-dimensional CAD data as a search model, and for example, actual measurement data obtained by actually imaging a workpiece with a sensor unit can be registered as a search model. Here, the procedure for registering such measured data as a search model in step S2701 of FIG. 27 will be described with reference to the flowchart of FIG.

First, in step S6201, the surface of the work to be registered faces upward, the work is placed on a flat surface, and three-dimensional measurement is performed by the sensor unit.

Next, in step S6202, the obtained measured data is registered as a search model.

Finally, in step S6203, it is determined whether or not the number of search models required for the three-dimensional search has been registered. If not, the process returns to step S6201 to repeat the above process. terminates the process. Details of these procedures will be described later with reference to FIGS.
(Procedure 1 for actual operation)

After the necessary setting work is completed as described above, the picking operation is performed for the group of works that are actually randomly stacked. Here, in a state in which the search model is registered in the procedure of FIG. Description will be made based on the flow chart of FIG. Here, the calculation unit 10 in FIG. 6 determines whether or not there is a gripping solution.

First, in step S6301, three-dimensional measurement is started for randomly stacked works. Here, a group of randomly stacked workpieces is imaged by the sensor unit, three-dimensional measurement is performed, and a three-dimensional shape having height information is obtained.

Next, in step S6302, the three-dimensional shape of the obtained work group is subjected to a three-dimensional search using the work model to detect the position and orientation of each work.

Next, in step S6303, the position and orientation at which the end effector should be arranged are calculated for the detected one work based on the position of this work and the gripping orientation of the work registered at the time of setting.

Next, in step S6304, an end effector model is used to determine whether or not the end effector interferes with surrounding objects at the calculated position.

Then, in step S6305, it is determined whether or not the end effector interferes with the workpiece.

On the other hand, if it is determined that there is interference, the process advances to step S6306 to determine whether or not there is another gripping position registered for this workpiece. If another gripping position is registered, the process returns to step S6303 to repeat the above processing for this gripping position.

On the other hand, if another gripping position is not registered, the process advances to step S6307 to determine whether or not there is another workpiece detected. If there is another work, return to step S6303, change the work, and repeat the above process. If there is no other workpiece, the process is terminated as no gripping solution.

In this manner, the computing unit 10 of FIG. 6 determines whether or not there is a gripping solution that can grip the workpiece. Then, when a gripping solution is obtained, an instruction is sent to the robot controller 6 to pick the workpiece at the determined gripping position with the end effector. In accordance with this, the robot controller 6 controls the end effector to pick up the instructed work.

In the above procedure, when a gripping solution is obtained for any workpiece, the process of examining the gripping position is terminated at that point, and the workpiece is gripped at the gripping position corresponding to the obtained gripping solution. ing. However, the method is not limited to this method. For example, it may be configured such that, after obtaining all possible gripping positions as gripping position candidates, it is determined which gripping position candidate is to be selected. For example, the score is calculated by the evaluation index calculator 8q as an evaluation index for each gripping position candidate, and the one with the highest obtained score is selected as the gripping position. In addition, attention is paid to the height information of the work placed, and the one where the work is placed at a high height, in other words, the one positioned higher in the group of randomly piled works is selected as the gripping position. can also Preferably, the calculation unit 10 selects from a plurality of grasping solutions in consideration of both the score and the height information. By doing so, it becomes possible to perform more appropriate picking.
(Procedure 2 during actual operation)

The procedure for actual operation has been described above with the search model registered in the procedure of FIG. When registering the search model, as shown in FIG. 48, it is possible to register the search model in a state in which the orientation of the search model is restricted. Now, with the search model registered according to the procedure in FIG. 48, the procedure for actual operation of picking a group of works that are actually randomly stacked will be described with reference to the flow chart in FIG. First, in step S6401, three-dimensional measurement is started for randomly stacked works. Here, a group of randomly stacked workpieces is imaged by the sensor unit, three-dimensional measurement is performed, and a three-dimensional shape having height information is acquired.

Next, in step S6402, a three-dimensional search is performed on the obtained three-dimensional shape of the work group using the work model to detect the position and orientation of each work.

Next, in step S6403, attitudes outside the ranges of the tilt angle and the rotation angle are excluded.

Next, in step S6404, the position and orientation at which the end effector should be arranged are calculated for the detected one work based on the position of this work and the gripping orientation of the work registered at the time of setting.

Next, in step S6405, an end effector model is used to determine whether or not the end effector interferes with surrounding objects at the calculated position.

Then, in step S6406, it is determined whether or not the end effector interferes with the workpiece.

On the other hand, if it is determined that there is interference, the process advances to step S6407 to determine whether or not there is another gripping position registered for this workpiece. If another gripping position is registered, the process returns to step S6404 to repeat the above processing for this gripping position.

On the other hand, if another gripping position is not registered, the process advances to step S6408 to determine whether there is another workpiece detected. If there is another work, return to step S6404, change the work, and repeat the above process. If there is no other workpiece, the process is terminated as no gripping solution.
(Interference judgment)

Here, a method of determining interference using the end effector model in step S6304 of FIG. 63 and step S6405 of FIG. 64 described above will be described. When gripping a workpiece with the end effector, if the end effector interferes with surrounding obstacles such as other workpieces or storage containers, gripping cannot be performed correctly. Therefore, the position and orientation of the end effector when the end effector model grips the work model at the gripping position candidates are calculated in advance by the interference determination unit 8m in FIG. 31, thereby determining interference with surrounding members. At this time, as the surrounding members, a three-dimensional point cloud of a group of randomly piled workpieces and a storage container obtained by actual measurement by the sensor unit 2 is used. Moreover, it is also possible to register the positions of storage containers and the like in advance and perform interference determination with the end effector model. On the other hand, three-dimensional CAD data of end effectors are generally composed of polygon data. For example, STL data, which is often used as three-dimensional CAD data, is represented by a collection of minute triangles called polygons.

In order to perform interference judgment between such polygon data and 3D point cloud data, conventionally, each 3D point constituting the 3D point cloud data is examined whether it is inside or outside the end effector model, It was determined that there was no interference if it was on the inside and that there was no interference if it was on the outside. However, with this method, it is necessary to perform calculations and comparisons for each point, and the amount of calculation becomes enormous when the data becomes large.
(Procedure for interference judgment using cross-sectional model)

Therefore, in each embodiment of the present invention, a cross-sectional model is created from the polygon data of the end effector model, each point of the three-dimensional point cloud data is projected onto the cross-sectional model, and interference is determined by determining inside and outside. ing. Such interference determination is performed by the interference determination unit 8m in FIG. Here, a procedure for determining interference will be described with reference to the flowchart of FIG. 65 .

First, in step S6501, the polygon data of the end effector are read. Next, in step S6502, a cross-sectional model is created from the polygon data of the end effector. A cross-sectional model is generated by the cross-sectional model generating unit 8s shown in FIG. Here, a method for creating a cross-sectional model by the cross-sectional model generation unit 8s will be described with reference to FIGS. 66 to 67E. First, the basic axis BSL is set for the polygon data of the end effector model EEM shown in FIG. Preferably, the base axis BSL is set along the longitudinal direction of the end effector model EEM. Furthermore, the end effector model is cut along an orthogonal plane orthogonal to this basic axis BSL to create a plurality of cross sections. Here, the position for creating the cross section is, for example, the vertex position of the polygon data. Alternatively, after creating a plurality of cross sections at regular intervals along the basic axis, the obtained cross sections may be arranged according to the shape. For example, cross-sections with the same shape are excluded. It is preferable to set the basic axis BSL so that the total number of required cross sections is small. For example, the cross-sectional model generator may be configured to automatically calculate the direction of the basic axis BSL that reduces the number of cross-sections. Moreover, the designation of the cross-section position for obtaining the cross-section on the set basic axis BSL can also be designated, for example, to automatically extract a cross-section with a large change.

In this way, a cross-sectional model having the shape of each cross section along the basic axis BSL of the end effector model EEM and the cross-sectional position on the basic axis BSL corresponding to each cross section can be created. For example, in the example of FIG. 66, cross sections SS1 to SS5 having shapes shown in FIGS. 67A to 67E are obtained at five cross section positions SP1 to SP5 along the basic axis BSL. Using the cross-sectional model thus obtained, it is determined whether or not the end effector model interferes with the three-dimensional point group.

Specifically, in step S6503, a three-dimensional point to be subjected to interference determination is selected from among the three-dimensional point cloud, and from the position of this point in the direction of the basic axis BSL, one of the plurality of cross sections of the cross-sectional model is selected. , to select which section to judge interference with. For this selection, the cross-section position set for each cross-section is used. For example, let us consider a case where a three-dimensional point TDP shown in FIG. 68A among three-dimensional points is subjected to interference determination with the end effector model EEM. The three-dimensional point TDP is located between the cross-sectional positions SP3 and SP4 in the direction along the basic axis BSL. Therefore, interference with the three-dimensional point TDP is determined using the cross-section SS3 representing the cross-sectional shape therebetween.

Specifically, in step S6504, a projection point PP3 projected onto an orthogonal plane including the cross section SS3 from the three-dimensional point TDP is calculated. Then, in step S6505, interference determination is performed. Here, if the position of the projection point PP3 of the three-dimensional point is outside the cross section SS3 as shown in FIG. 68B, it is determined that there is no interference. On the other hand, if it is inside the cross section SS3 as shown in FIG. 68C, it is determined that there is interference. Finally, in step S6506, it is determined whether or not there is another 3D point, and if an unprocessed 3D point remains, the process returns to step S6503 to repeat the above processing. When the collision determination is completed for all three-dimensional points, the process ends.

In this way, it is possible to determine interference between the measured three-dimensional point group and the end effector model. In addition, although the collision judgment with 3D point cloud data has been described above, the object of the present invention is not limited to 3D points for interference judgment. can be implemented.

Furthermore, in the above example, the CAD data of the end effector is polygon data. can do Furthermore, in the above example, the case where the cross-sectional shape is represented by a two-dimensional plan view has been explained, but the way of holding cross-sectional shape data is not limited to this form. You may have
(Interference judgment with additional model with additional area added)

Further, in the collision judgment of the end effector model, it is also possible to judge interference using an additional model obtained by adding an additional region represented by a three-dimensional basic figure to the end effector model. A procedure for determining interference using such an additional model will be described with reference to the flowchart of FIG. Here, an example will be described in which interference is determined with respect to an additional model added to a three-dimensional CAD model by adding an additional region obtained by combining a rectangular parallelepiped and a cylinder as a basic figure.

First, in step S6901, interference determination is performed for a rectangular parallelepiped area among the basic figures forming the additional area. If it is determined in step S6902 that there is interference according to this result, the processing for determining interference is stopped, and the presence of interference is output to end the processing.

On the other hand, if it is determined that there is no interference, the process advances to step S6903 to determine whether there is interference with another figure, which is a cylinder, among the basic figures forming the additional area. As a result, if it is determined in step S6904 that there is interference, the process of determining interference is stopped, and the process ends by outputting that there is interference.

On the other hand, if it is determined that there is no interference, the process advances to step S6905 to determine interference with the three-dimensional CAD data. As a result, if it is determined that there is interference in step S6906, the process of determining interference is stopped, and the process ends by outputting that there is interference.

On the other hand, if it is determined that there is no interference, the processing is terminated assuming that there is no interference.

In this way, the collision judgment is sequentially performed for each basic figure or for each three-dimensional CAD data, and when it is judged that there is interference, the collision judgment processing is stopped at that point, and it is output that there is interference. Although the example in which the additional area is composed of two basic figures, a rectangular parallelepiped and a cylinder, has been described above, the procedure is the same even if the number of basic figures increases. If it is determined that there is interference, the process is stopped.
(Embodiment 7)
(Gripperability judgment verification function)

The procedure for determining whether or not there is a gripping solution based on the results of interference determination during actual operation has been described above. However, the present invention not only determines whether or not there is a gripping solution, but also has a gripping possibility determination verification function that verifies for what reason a gripping solution could not be obtained for a candidate of a gripping position for which no gripping solution could be obtained. may be provided. For example, a list of gripping solution candidates is displayed, and as a result of interference determination, a gripping position determined as having a gripping solution is displayed as OK, and a gripping position candidate determined as having no gripping solution is displayed as NG. In this state, by selecting the gripping position determined as NG and displaying the reason why it was determined that there is no gripping solution, the user can refer to this information and select any gripping position. It is possible to examine whether it can be selected as a gripping solution, modify the gripping position, or add a new gripping position. Such an example is shown in the block diagram of FIG. 70 as a robot system 7000 according to the seventh embodiment. A robot system 7000 shown in this figure includes an image processing device 700, a display section 3B, an operation section 4, a sensor section 2, a robot controller 6, and a robot RBT. The same members as those in FIG. 6 are denoted by the same reference numerals, and detailed description thereof will be omitted as appropriate.
(Display section 3B)

The display unit 3B includes an image display area 3b and a gripping solution candidate display area 3c. The gripping solution candidate display area 3c includes a workpiece grippability display area 3d and a workpiece grippability factor display area 3e.

The image display area 3b is a member for displaying an end effector model composed of three-dimensional CAD data, which virtually represents the three-dimensional shape of the end effector, three-dimensionally in a virtual three-dimensional space. be.

The gripping solution candidate display area 3c is a member for listing and displaying all gripping positions set for any one of the search results of one or more workpieces searched by the three-dimensional search unit 8k. .

The workpiece grippability display area 3d is a member for displaying the determination result of grippability at the gripping position designated for each workpiece by the three-dimensional pick determination unit 8l.

The workpiece gripping impossible factor display area 3e displays the factors that make it impossible to grip the gripping position specified for each workpiece by the three-dimensional pick determination unit 8l. It is a member for
(Image processing device 700)

The image processing apparatus 700 includes an input image acquisition section 2c, a calculation section 10, a storage section 9, an input/output interface 4b, a display interface 3f, and a robot interface 6b. The calculation unit 10 includes an end effector model registration unit 8u, a work model registration unit 8t, a gripping position specifying unit 8d, a search model registration unit 8g, a three-dimensional search unit 8k, a three-dimensional pick determination unit 8l, an inclination and an angle setting portion 8n.

The gripping position specifying unit 8d includes a work-side gripping point specifying unit 8d1 and an end effector-side gripping setting unit 8d2. The work-side gripping point designation unit 8d1 designates a work model composed of three-dimensional CAD data, which virtually expresses the three-dimensional shape of the work as an end effector model, with the end effector model displayed in the image display area 3b. It is a member for specifying the gripping position when gripping with. The end effector-side gripping setting section 8d2 is a member for designating a gripping position for gripping a workpiece with respect to the end effector model displayed in the image display area 3b.

A search model registration unit 8g performs a three-dimensional search for specifying the posture and position of each work for a plurality of work groups included in the input image for a second work model that virtually represents the three-dimensional shape of the work. It is a member for registering as a search model for performing. It is preferable that the second workpiece model registered as the search model is the same as the workpiece model whose gripping position is specified by the workpiece-side gripping position specifying section 8d1. As a result, the search model for performing the three-dimensional search is shared with the work model for designating the gripping position, thereby saving the user's setting work. Also, during actual operation, by matching the work model for searching for a work that can be gripped with the work model for gripping determination, the searched work model can be gripped at the gripping position set for this work model. Since it is possible to examine whether or not the

The tilt angle setting part 8n is a member for setting the range of the tilt angle that is allowed with respect to the posture of the work.

The three-dimensional pick determination unit 8l uses the end effector to pick up the search result of each workpiece searched by the three-dimensional search unit 8k at the gripping position specified for the workpiece by the workpiece-side gripping position specifying unit 8d1. It is a member for determining whether or not it can be gripped. The three-dimensional pick determination unit 8l includes an interference determination unit 8m and an angle determination unit 8o.

With respect to the search results of each workpiece searched by the three-dimensional search unit 8k, the interference determination unit 8m determines whether the gripping position specified for this workpiece by the workpiece-side gripping position specifying unit 8d1 exists around the workpiece. It is a member for judging the presence or absence of interference with an object. Also, the three-dimensional pick determination unit 8l determines that the workpiece, which has been determined to have interference by the interference determination unit 8m, cannot be gripped. As a result, the cause of the inability to grip the workpiece is indicated, which helps the user reset the gripping positions, such as making it easier for the user to consider what kind of gripping positions should be added.

The angle determination unit 8o is a member for determining whether or not the posture of the workpiece searched by the three-dimensional search unit 8k is within the tilt angle range set by the tilt angle setting unit 8n.

When the angle determination unit 8o determines that the posture of the workpiece searched by the three-dimensional search unit 8k is not within the tilt angle range set by the tilt angle setting unit 8n, the three-dimensional pick determination unit 8l Also, it is configured to determine that the work cannot be gripped. As a result, when the posture of the workpiece is too steep to grip, or when the accuracy of three-dimensional measurement cannot be expected, this eliminates erroneous selection and judgment of the gripping position and improves reliability. can be done.
(Grip solution candidate list display function)

Next, a procedure for displaying the reason for gripping solution NG will be described with reference to the flowchart of FIG. It should be noted that the procedure for setting and the procedure for actual operation are basically the same as those in FIGS.

First, in step S7101, three-dimensional measurement is performed on a target work group. Here, for a group of randomly stacked workpieces, actual measurement data having shape information is acquired using the sensor unit, and this is used as an input image.

Next, in step S7102, a three-dimensional search is performed on the input image to detect the position and orientation of each workpiece in the input image.

Then, in step S7103, one work to be verified is selected from the detected works.

Further, in step S7104, the position and orientation of the end effector model when the end effector attempts to grip this workpiece are calculated from the detected position of the selected workpiece and the gripping orientation registered in advance for this workpiece.

Next, in step S7105, it is determined whether or not the calculated tilt angle in the position and orientation of the end effector model is outside the set range. If it is determined to be out of the set range, the process advances to step S7106 to determine that the gripping is NG, sets the NG factor to the "inclination angle", and jumps to step S7111.

On the other hand, if it is determined in step S7105 that the tilt angle of the end effector model is within the set range, the process advances to step S7107 to determine interference between the end effector model and surrounding objects at the calculated position. Here, the surrounding objects are storage containers, other workpieces, and the like that exist around the end effector model. When the surrounding objects are modeled in advance using three-dimensional CAD data, etc., and when the end effector model is moved to the position and orientation where the work model is gripped, it is determined whether or not the end effector model interferes with these by calculation. judge.

Then, in step S7108, if it is determined that the end effector model interferes with the surrounding object as a result of the interference determination, in step S7109, the gripping NG determination is made, and the NG factor is set to "point group interference". Above, jump to step S7111.

On the other hand, if it is determined in step S7108 that the end effector model does not interfere with the surrounding object as a result of the interference determination, then in step S7110 it is determined that gripping is OK, and the process proceeds to step S7111.

Then, in step S7111, it is determined whether or not another gripping orientation is set for the selected work. On the other hand, if it is determined that there is no gripping posture, in step S7112, for all gripping solution candidates, the determination results of gripping OK/gripping NG and the factors in the case of gripping NG are displayed in the work gripping impossible factor display area 3e. display.

In this way, it is possible to verify whether or not a given workpiece group can be gripped, and to list the factors that prevent it from being gripped.
(Gripping simulation)

Next, the details of the gripping simulation for performing a specific determination of gripping will be described. Here, an example will be described in which it is determined whether or not the work WK10 shown in FIG. 72 can be gripped when bulk picking is performed. For this work WK10, a basic direction image generation unit 8e' generates a basic direction image corresponding to a six-sided view, and each basic direction image is registered as a search model in a search model registration unit 8g. Examples of cardinal direction images are shown in FIGS. 73A-73E. In these figures, FIG. 73A is a height image A of the work in FIG. 72 viewed from the positive direction side of the Z axis, FIG. 73B is a height image B viewed from the negative direction side of the Z axis, and FIG. is the height image C viewed from the positive direction side of the X axis, FIG. 73D is the height image D viewed from the negative direction side of the X axis, and FIG. 73E is the height image D viewed from the positive direction side of the Y axis FIG. 73F shows the height image F viewed from the negative side of the Y-axis.
(Work selection screen 210)

In order to determine whether or not a workpiece can be gripped by the end effector, a three-dimensional search is performed on the input image of a group of randomly piled workpieces, and when the workpiece is detected, the target workpiece is selected. (Step S7103 in FIG. 71). Selection of a work is performed from a work selection screen 210 shown in FIG. 74, for example. The work selection screen 210 shown in this figure has an image display field 141 and an operation field 142 . The image display field 141 displays an input image, which is actually measured data obtained by imaging a group of randomly stacked workpieces. In addition, the search result is displayed as a point cloud superimposed on the position where the workpiece is detected in the input image. By dragging the screen of the image display column 141, the viewpoint can be changed.
(label number, model number)

The operation column 142 includes a target work selection column 211 for selecting a target work, a detection search model display column 212 for indicating a search model used for three-dimensionally searching this target work, and a selected work. A "grip confirmation" button 213 is provided for displaying a list of all grip position candidates. In the example of FIG. 74, 18 workpieces exist as a result of the three-dimensional search of the input image, and the third workpiece among them is selected. In order to distinguish the search results detected here, identification information is set to the search results. Here, as the identification information, the label number of the serial number is set. In the target work selection column 211 in FIG. 74, "3" is displayed as the label number of the target work being selected. In the image display field 141, search results of the selected target work are displayed. Specifically, as a state of the search result, the feature points of the corresponding search model are displayed in a form superimposed on the point group of the input image. If the label number of the target work is changed in this state, the work selected in the image display field 141 is also changed accordingly. Further, in the detected search model display column 212, a search model E (height image in FIG. 73E) is displayed as a search model used for three-dimensionally searching the target work. The search model is also given individual identification information, which is called a model number here. In FIG. 74, "E" of the work model with serial number label number 3 is displayed as the model number in the detected search model display field 212 .
(Gripping solution candidate display screen 220)

Furthermore, gripping positions set for each workpiece, that is, gripping solution candidates can be displayed in a list in the gripping solution candidate display area 3c. In the example of the work selection screen 210 in FIG. 74, when the "confirm gripping" button 213 is pressed, the gripping solution candidate display screen 220 in FIG. 75 is displayed on the display unit. On the gripping solution candidate display screen 220, which is one form of the gripping solution candidate display area 3c, all gripping positions set for the target work selected in the target work selection field 211 in FIG. 74 are gripping solution candidates. They are listed in the candidate display column 221 . In the gripping solution candidate display column 221, for each gripping solution, a workpiece gripping position display column 223 indicating the label number of the gripping position, a workpiece gripping availability display column 224 indicating the determination result of gripping availability of the workpiece, and a A workpiece gripping failure factor display field 225 is provided to indicate the reason. In the example of FIG. 75, for the third target work selected in the target work selection field 211, five candidates for gripping are listed, and the determination result of gripping OK or NG is displayed together with NG. The cause is displayed. Also, corresponding to the gripping position of the workpiece selected in the gripping solution candidate display column 221, the workpiece selected in the image display column 141 and the end effector model gripping the workpiece are displayed in the posture of gripping the gripping position. be. When the selection of the gripping position is changed in the gripping solution candidate display field 221, the work in the image display field 141 is changed accordingly, and the posture of the end effector model that grips this work is also updated.

The example of FIG. 75 shows a state in which the workpiece of FIG. 73E is gripped at the gripping position candidate with label number 3. In FIG. In this case, since the determination result is OK for gripping, nothing is displayed in the work gripping impossible factor display column 225 . On the other hand, when the gripping position candidate with label number 4 is selected in the work gripping impossible factor display field 225, the display in the image display field 141 is also changed as shown in FIG. Since the gripping position candidate with label number 4 is the determination result of gripping NG, the reason why it is determined that the workpiece cannot be gripped is displayed in the work gripping impossible factor display column 225 . Further, in the image display field 141, the corresponding end effector model is displayed, and it is possible to visually confirm what state the gripping posture is actually in. Here, "point group interference" is displayed, and it can be confirmed that the end effector interferes with the plane of the storage container. As a result, the user can consider adding a gripping position at a position or posture that does not interfere with a work having such a posture.

Similarly, when the gripping position candidate with label number 2 is selected in the work gripping impossible factor display field 225, the gripping solution candidate display screen 220 shown in FIG. The reason for this is displayed in the work gripping impossible factor display column 225 as "inclination angle". Furthermore, in the image display column 141, the corresponding end effector model is displayed, and the end effector is in a state of being grasped at a steep angle of inclination, and it can be seen that it is excluded from interference determination targets in the first place.

Also, the display mode of the end effector displayed in the image display field 141 may be changed according to the determination result. In the example of FIG. 75, the end effector model is displayed in white when the grip is OK, and in the examples of FIGS. 76 and 77, the end effector model is displayed in red when the grip is NG. This allows the user to visually distinguish the determination result of gripping easily. Further, when adding a gripping position, the display in the image display field 141 may be updated so that the posture of the end effector model is adjusted and the display mode is changed at the time when the interference does not occur. . By configuring in this way, the user can change the posture of the end effector model, for example, from red to white.

By providing a grippability determination verification function in this way, when it is determined that a workpiece to be gripped cannot be gripped, how to set gripping, for example, adding a new gripping position or adjusting an existing gripping position. It serves as a guideline when considering work such as changing the position settings. For example, in the examples shown in FIGS. 78 and 79, there are two gripping position candidates for the workpiece of the search model C (FIG. 73C), but both of them cannot be gripped due to point group interference. there is In both cases, as indicated by the dashed circle in the image display field 141, it is found that the tip of the nail of the end effector model interferes with another work. From this, it can be seen that an OK gripping solution can be obtained by adding a new gripping position so as to grip a point on the opposite side of the current gripping position. Therefore, if the gripping position specifying unit 8d is set to add a new gripping position (gripping posture C-002) on the opposite side of the current gripping position, as shown in FIG. become. In this way, when the user performs work for adjusting the gripping position and posture, an environment is provided in which it is easy to consider countermeasures, and a robot system that is easy to set is realized.

It is also possible for the user to change the threshold value of the inclination angle of the end effector. For example, when using a deep-bottomed box as a storage container for workpieces, if the tilt of the end effector becomes large, the arm part of the robot other than the end effector will easily collide with the wall of the storage container, so the angle range should be narrowed. do. Conversely, when a box with a shallow bottom is used as the storage container, the arm of the robot is unlikely to collide with the box unless the end effector interferes with the box, so a wide angle range can be set. In this way, by adjusting the setting of the angle range, it is possible to flexibly adjust whether or not the grip is possible depending on the actual situation.

As described above, all gripping solutions for the selected workpiece are displayed in a list, and for those that were NG, the NG factors are also displayed. If there is, it becomes easy to identify the cause of why it is not gripped. In addition, since the cause can be identified, it becomes easy to understand what kind of new gripping posture should be added.
(Embodiment 8)

In the above example, an example in which search results are individually obtained for each search model during a three-dimensional search has been described. That is, three-dimensional search is performed using a plurality of basic direction images showing different surfaces of the same work, and the obtained results are also recognized as separate works. In other words, different surfaces of the same workpiece may be detected as different workpieces as a result of being individually searched. On the other hand, in the conventional three-dimensional search, since the three-dimensional shape representing one workpiece model is used for searching, such a situation does not occur, and different surfaces of the same workpiece are detected as one workpiece. However, in the case of works with many surfaces, the search becomes complicated and the possibility of false detection increases. On the other hand, in the method according to the above-described embodiment, since the search is performed in a simple manner, the search processing can be simplified, which is advantageous in terms of low load, high speed, and the like. On the other hand, as a result of searching for each surface, the search results obtained as described above are regarded as separate works, and there is a problem that different surfaces are individually detected even if the same work is used.

Therefore, by integrating the obtained search results and grouping them for each workpiece, or estimating the search result of another surface from the search result of one surface, it is possible to detect surfaces that could not be detected by three-dimensional search, and detection accuracy is low. Faces can also be made available as gripping solution candidates. Such an example is shown in the block diagram of FIG. 81 as a robot system according to the eighth embodiment. A robot system 8000 shown in this figure includes an image processing device 800, a display section 3B, an operation section 4, a sensor section 2, and a robot RBT. 6, 70, etc. are given the same reference numerals, and detailed description thereof will be omitted as appropriate.
(Image processing device 800)

The image processing apparatus 800 includes an input image acquisition section 2c, a calculation section 10, a storage section 9, an input/output interface 4b, a display interface 3f, and a robot interface 6b. The calculation unit 10 includes a basic direction image generation unit 8e′, a grip position identification unit 8d, a search model registration unit 8g, a three-dimensional search unit 8k, an image estimation unit 8z, a search result integration unit 8p, a three-dimensional A pick determination unit 8l and an inclination angle setting unit 8n are provided.

The basic direction image generation unit 8e' generates a plurality of height images of the work model viewed from any one of the directions of three mutually orthogonal axes in the virtual three-dimensional space as basic direction images. is a member of

The gripping position specifying unit 8d specifies a plurality of gripping positions at which the end effector grips the work model indicated by the basic direction image for one of the basic direction images generated by the basic direction image generating unit 8e′. It is a member. The gripping position specifying unit 8d includes a work-side gripping point specifying unit 8d1 and an end effector-side gripping setting unit 8d2.

The work-side gripping point designation unit 8d1 designates a work model composed of three-dimensional CAD data, which virtually expresses the three-dimensional shape of the work as an end effector model, with the end effector model displayed in the image display area 3b. It is a member for specifying the gripping position when gripping with.

The end effector-side gripping setting section 8d2 is a member for designating a gripping position for gripping a workpiece with respect to the end effector model displayed in the image display area 3b.

The search model registering unit 8g registers the plurality of basic direction images generated by the basic direction image generating unit 8e' with respect to the plurality of workpieces included in the input image acquired by the input image acquiring unit 2c. It is a member for registering each as a search model used when performing a three-dimensional search for specifying the posture and position. The search model registration unit 8g can also register relative positions of a plurality of registered search models as relationship information (details will be described later).

The image estimating unit 8z searches the input image by the three-dimensional search unit 8k and extracts each search model from the input image. Based on the direction in which the basic direction image is viewed and the relative positional relationship with the basic direction images of other search models registered for this work model, for the work model indicated by this search result and a member for estimating the position and orientation of an unsearched basic direction image that is not included in the search results of the three-dimensional search unit as an estimated image. The image estimating unit 8z, for example, is operated by an angle determining unit 8o, which will be described later. , to estimate an estimated image having a relative positional relationship with this search result.

The search result integration unit 8p searches the input image by the three-dimensional search unit 8k and extracts each search model from the input image. It is a member for integrating adjacent search results among a plurality of search results as an integration result for a common work based on the relative positional relationship of which direction the basic direction image is viewed from.

The three-dimensional pick determination unit 8l uses the gripping position specifying unit 8d to determine the workpiece model based on the integrated results integrated by the search result integration unit 8p and the unintegrated search results searched by the three-dimensional search unit 8k. It is a member for determining whether or not the end effector can be gripped at the gripping position specified by the . The three-dimensional pick determination unit 8l includes an interference determination unit 8m and an angle determination unit 8o.

As a result, the position and orientation of the workpiece can be estimated more accurately using the relational information between the surfaces, instead of individually determining whether to hold the search results of the three-dimensional search. Alternatively, it becomes possible to detect faces with low accuracy in search results, and it is possible to consider faces that are normally difficult to search as holding position candidates, increasing the possibility of obtaining a holding solution. becomes possible.
(Procedure for registering a search model including relationship information between faces)

For the image processing apparatus described above, the procedure shown in the flowchart of FIG. 27, for example, can be used as the procedure for setting the registration of the work model and the end effector model, the registration of the gripping position, and the like. Here, in step S2701 of FIG. 27, a procedure for registering a search model including relationship information between surfaces of a workpiece while using three-dimensional CAD data as a search model will be described with reference to the flow chart of FIG.

First, in step S8201, a three-dimensional CAD data model of a workpiece is read.

Next, in step S8202, the center of the circumscribed cuboid of the three-dimensional CAD data model is corrected to the origin of the three-dimensional CAD data.

Further, in step S8203, height images viewed from each of the "top", "bottom", "left", "right", "front", and "back" directions are generated as basic direction images. The basic direction image is generated by the basic direction image generation unit 8e' in FIG. Here, when generating a height image from three-dimensional CAD data, the height image is generated so that the origin of CAD is at the center of the height image.

Next, in step S8204, height images that look the same are deleted from the generated height images.

Further, in step S8205, the relative positions of multiple registered search models are registered as relationship information. Here, the search model registration unit 8g saves the remaining height image and the relationship information between the top, bottom, left, right, front and back surfaces.

Finally, in step S8205, a search model is registered using the generated height image. In this way, a search model including relationship information between surfaces of a work is registered.
(Relationship information of faces)

Here, the relationship information indicating the relative positional relationship between the surfaces of the work viewed from a specific direction will be described. For example, in the case of the workpiece as shown in FIG. 7, the surfaces with different views are the four surfaces shown in FIGS. 9A to 9D. In these figures, model A shown in FIG. 9A is a height image viewed from the positive direction of the X axis. However, it also matches the image viewed from the negative direction of the X axis. Also, while model B shown in FIG. 9B is a height image viewed from the positive direction of the Y-axis, it also matches an image viewed from the negative direction of the Y-axis. On the other hand, model C shown in FIG. 9C only matches images viewed from the positive direction of the Z axis. Also, model D shown in FIG. 9D matches only the image viewed from the negative direction of the Z-axis.

Here, the models A, B, C, and D, which are height images generated in advance, match the images of the three-dimensional CAD data of the workpiece shown in FIG. is information representing the relationship information of the face. If such surface relationship information is available, the posture of the original three-dimensional CAD model can be obtained from the search results of each model. For example, on the left and right sides of model A in FIG. 9A, model B in FIG. 9B exists adjacent to each other. On the upper surface of model A, model C of FIG. 9C is present, and on the lower surface thereof, model D of FIG. 9D is present, both adjacent to each other. As a result, in the searched image obtained as a result of the three-dimensional search, when the detection results of model A and model B are adjacent to each other, it can be seen that they are the results of searching for the same workpiece. Therefore, these search results can be integrated as one search result. Here, a set of integrated search results is called an integrated result.

In addition, the evaluation index can be updated by integrating the search results. That is, even if the search result is not highly accurate, if the integrated result is a highly accurate search result, the evaluation index of the integrated result is increased so that the correct gripping solution can be obtained when selecting the gripping position. can be prioritized.
(Unsearched basic direction image)

Furthermore, even if there are faces that are not searched, such as when the three-dimensional search fails or when the search itself is not performed, it is possible to make an estimation using the integration result. That is, even if there is a surface that is not detected in the three-dimensional search among the surfaces constituting the workpiece when obtaining the integrated result of the workpiece by integrating a plurality of search results, the other surfaces If search results are obtained in , the posture and position of the work itself can be calculated. As a result, when a workpiece model in which gripping positions are registered in advance with respect to a workpiece or a search model for three-dimensional search is registered, information on unsearched surfaces can also be estimated from the orientation of the workpiece. Therefore, by using a surface obtained by estimation (referred to as an unsearched basic direction image), which is not obtained as a result of such a search, as a candidate for the gripping position, it is possible to obtain an image that was not actually searched. The grasping position of the surface can also be examined as a candidate for the grasping solution, and an advantage is obtained that an appropriate grasping solution can be easily obtained. For example, when the model B shown in FIG. 9B is obtained as a result of the three-dimensional search, it can be estimated that the surface of the model A shown in FIG. 9A exists next to it. As a result, even if model A is not actually detected as a search result, the surface of model A can be estimated from the search result of model B. Therefore, the surface of model A can be registered as a gripping solution candidate. As a result, the number of gripping solution candidates is increased, making it easier to pick.
(Calculation of grasping solution using surface relationship information)

Next, the effect of gripping solution calculation using surface relationship information will be described using the workpiece WK10 shown in FIG. Basic direction images shown in FIGS. 73A to 73F are generated from the three-dimensional CAD data of the workpiece WK10 by the basic direction image generating section 8e', and these are registered as search models by the search model registering section 8g. In these figures, FIG. 73A is a height image viewed from the positive direction of the Z-axis, and this is model A. FIG. Similarly, FIG. 73B is a height image viewed from the negative direction of the Z-axis, and this is model B. FIG. FIG. 73C is a height image viewed from the positive direction of the X-axis, which is assumed to be Model C. FIG. Further, FIG. 73D is a height image of model D viewed from the negative direction of the X axis, FIG. 73E is a height image of model E viewed from the positive direction of the Y axis, and FIG. Viewed height images and model F are shown, respectively.

Here, for each of these basic direction images, as shown in FIGS. 83A to 83F, one gripping posture for gripping the work model WM10 with the end effector model EM10 is registered by the gripping position specifying unit 8d. shall be Here, FIG. 83A shows a state in which the gripping posture A-000 is registered with respect to the basic direction image of FIG. 73A. FIG. 83B shows a state in which the gripping posture B-000 is registered with respect to the basic direction image of FIG. 73B. Further, FIG. 83C shows the state in which the gripping posture C-000 is registered with respect to the basic direction image of FIG. 73C, and FIG. FIG. 83E shows the state in which the gripping posture E-000 is registered with respect to the basic direction image of FIG. 73E; FIG. Each shows the state of In this case, calculation of a gripping solution using surface relationship information will be described with reference to FIGS. 84, 85, and 86. FIG.

The workpiece selection screen 210 in FIG. 84 shows a state in which 12 workpieces are detected as a result of performing a three-dimensional search on an input image obtained by picking up a group of randomly stacked workpieces. An input image is displayed in the image display field 141 . The operation field 142 includes a target work selection field 211 for selecting a target work, a detected search model display field 212, and a "gripping confirmation" button for displaying a list of gripping position candidates for the selected work. 213 is provided. In the example of FIG. 84, of the 12 works that have been three-dimensionally searched, the second work is selected in the target work selection column 211, and the selected second target work is displayed in the image display column 141. search results are displayed. Specifically, as a state of the search result, the feature points of the corresponding search model are displayed in a form superimposed on the point group of the input image. Further, in the detected search model display column 212, search models C, F, and A are displayed as search models that have detected the second target work. In this state, three types of search models C, F, and A (FIGS. 73C, 73F, and 73A) are integrated.

Here, assuming that the relationship information of the surfaces of the work is not used, the model F and the model A are in a state in which only part of the side surface is visible on the input image as shown in the image display field 141, so the search As a result, it seems that detection was difficult. Also, even if it could be detected, the evaluation index would be low and the priority would be low. On the other hand, by using the relationship information, not only the search model C but also the search model F and the search model A can be detected or estimated and used as gripping position candidates as shown in FIG. Become.

On the workpiece selection screen 210 in FIG. 84, when the "Confirm grasping" button 213 is pressed, a grasping solution candidate display screen 220 in FIG. 85 is displayed. In this gripping solution candidate display screen 220, the gripping positions set for the second target work selected in the target work selection column 211 of FIG. 84 are listed as gripping solution candidates in the gripping solution candidate display column 221. In the gripping solution candidate display column 221, whether or not the workpiece can be gripped at the gripping positions (FIGS. 83C, 83F, and 83A) set for each of the search models C, F, and A displayed in the detected search model display column 212 is displayed. The determination result is displayed in the work grippability display field 224, and the reason why the work cannot be gripped is displayed in the work grippability factor display field 225, respectively. First, for the search model C, the determination result is grasping NG, and the reason is the interference of the point cloud data. In the image display area, the end effector model EM10 attempting to grip the workpiece in the posture corresponding to the gripping position C-000 is displayed in red, indicating that it cannot be gripped due to interference. Here, assuming that the surface relationship information is not used, only the search model C is three-dimensionally searched, and the gripping solution candidate is only C-000 of this search model. There will be no

On the other hand, by using the relationship information, as shown in FIGS. 85 and 86, search models F and A are also targeted in addition to search model C, and F-000 and A-000 are added as gripping position candidates. It turns out that Among the added gripping position candidates, the gripping position candidate F-000 of the search model F is determined as gripping NG, but the gripping position candidate A-000 of the search model A is gripped OK. A determination result has been obtained, the end effector is displayed in white as shown in FIG. 86, and a grasping solution has been successfully obtained without interfering with the point cloud data. As a result, even if a grasping solution cannot be obtained without using the relationship information, the grasping solution can be obtained by using the relationship information. That is, without changing the settings of the three-dimensional search, for example, adjusting the lighting and camera arrangement, or changing the search algorithm, etc., the search conditions remain the same, in other words, without changing the accuracy of the three-dimensional search. , it is possible to increase the possibility of obtaining a grasping solution. In this way, by using surface relationship information, it is possible to increase the number of gripping solution candidates and obtain results that are likely to be picked.
(Procedure 3 for actual operation)

Here, a procedure for performing a three-dimensional search and a graspability determination at the time of actual operation with the search model registered according to the procedure in FIG. 82 will be described with reference to the flowchart in FIG. 87 .

First, in step S8701, three-dimensional measurement is started for randomly stacked works. In this case, the sensor unit 2 shown in FIG. 81 captures images of a group of randomly stacked workpieces, performs three-dimensional measurement, and acquires a three-dimensional shape having height information as an input image by the input image acquiring unit 2c.

Next, in step S8702, the three-dimensional shape of the obtained work group is subjected to a three-dimensional search using the work model to detect the position and orientation of each work. A three-dimensional search is performed by the three-dimensional search section 8k in FIG.
(Generation of integration result)

Further, in step S8703, the search results of the three-dimensional search and the results indicating the same work are integrated from the relationship information of each surface. Here, the search result integration unit 8p associates the search results obtained by imaging the same work using the relationship information registered in the search model used for the search. That is, based on the relative positional relationship of the basic direction image from which direction the common work model from which the search results are searched is viewed, adjacent ones of the plurality of search results are searched. The results are integrated by the result integration unit 8p. The integration results obtained in this manner are commonly treated as representing the same work.
(Generation of unsearched basic direction image)

Further, in step S8704, surfaces that have not been detected are estimated from the three-dimensional search result and relationship information of each surface. Here, based on the orientation and position of the work indicated by the result of integration, the position and orientation of unsearched basic direction images that are not included in the three-dimensional search results for this work are estimated. Then, the gripping position set by the gripping position specifying unit 8d with respect to the unsearched basic direction image is used as the object of determination of whether or not the grip can be gripped. As a result, by estimating from the orientation and position of the workpiece indicated by the integrated search result, even the surface of the workpiece that was originally not searched as a result of the three-dimensional search can be made a target for the determination of whether or not it can be gripped. It is possible to increase the possibility of obtaining a solution.

Next, in step S8705, the position and orientation at which the end effector should be arranged are calculated for the one detected workpiece based on the position of this workpiece and the gripping orientation of the workpiece registered at the time of setting.

Next, in step S8706, an end effector model is used to determine whether or not the end effector interferes with surrounding objects at the calculated position.

Then, in step S8707, it is determined whether or not the end effector interferes with the workpiece.

On the other hand, if it is determined that there is interference, the process advances to step S8708 to determine whether or not there is another gripping position registered for this workpiece. If another gripping position is registered, the process returns to step S8705 to repeat the above processing for this gripping position.

On the other hand, if another gripping position is not registered, the process advances to step S8709 to determine whether there is another workpiece detected. If there is another work, return to step S8705, change the work, and repeat the above process. If there is no other workpiece, the process is terminated as no gripping solution.

In this manner, the presence or absence of a gripping solution capable of gripping the workpiece is determined using the integration result and the unsearched basic direction image. Then, when a gripping solution is obtained, the robot controller 6 is controlled so that the workpiece is picked at the determined gripping position by the end effector.

In the present specification, the term "image" is not limited to strictly continuous data, and is used in the sense of including a set of discrete data such as a set of point cloud data.
(Embodiment 9)
(Integration of evaluation indicators)

In addition to the above procedure, it is also possible to integrate an evaluation index that defines the order of priority with respect to the integrated results. Such an example is shown in FIG. 88 as a robot system 9000 according to the ninth embodiment. An image processing apparatus 900 shown in this figure includes an evaluation index calculation unit 8q and a grip priority determination unit 8r. Members common to those shown in FIG. 81 and the like are denoted by the same reference numerals, and detailed description thereof will be omitted.

The evaluation index calculation unit 8q is a member for calculating an evaluation index for each grip position candidate, which is determined to be grippable by the three-dimensional pick determination unit 8l.

The gripping priority determination unit 8r is a member for determining the priority of gripping the workpiece with the end effector based on the evaluation index calculated by the evaluation index calculation unit 8q.

This image processing apparatus uses an evaluation index calculator 8q to calculate an evaluation index for each search result. As the evaluation index, as described above, for example, the score ratio of feature points corresponding with an error of a certain distance or less, which is included in the search results, can be used.

Here, the search result integration unit 8p gives the integrated result the highest evaluation index among the evaluation indexes calculated by the evaluation index calculation unit 8q for each search result constituting the integrated result. In general, the evaluation index for 3D search includes search conditions and workpiece posture (for example, when the workpiece tilt angle is large and sufficient reflected light cannot be obtained, or when the reflected light is too strong due to a glossy workpiece), etc. depends on For this reason, the evaluation index of the search result may be calculated to be low, the priority may be low, or the reflected light may not be obtained in the first place, and thus may be excluded from the target of the three-dimensional search. These will either be low-ranked as gripping position candidates or will not be listed as candidates. may be placed in a suitable position. Even in such a case, conventionally, it would have been excluded or had a low priority in the 3D search stage, so there was a low possibility that it would be selected as a grasping solution, and it could not be fully utilized. . On the other hand, according to the present embodiment, even if the result of the three-dimensional search is a search result with a low evaluation index, if other adjacent search results have a high evaluation value, this can be used as a gripping position candidate with a high priority. As a result, even gripping position candidates that have not been used in the past can be used, and the possibility of obtaining a gripping solution can be increased. In other words, it is possible to grip an appropriate workpiece regardless of the accuracy of the three-dimensional search.

In the above methods, the overall result of the integration is evaluated with the highest evaluation index among the search results that make up the integration result, or the evaluation index of each search result is increased from the original evaluation value to a higher evaluation value. The mode of rewriting has been described. This is based on the reliability that the reliability of the three-dimensional search results is high for search results with high evaluation indexes. However, the present invention is not limited to this aspect, and may be handled in another aspect, for example, by averaging the evaluation index of each search result that constitutes the integrated result and treating it as the evaluation index of this integrated result. In either case, a search result originally with a low evaluation index is evaluated with a higher evaluation index than the original evaluation index so that it can become a candidate for a gripping position or the like by using other search results with a high evaluation index. It is the gist of this embodiment to make it so.
(GUI of image processing program)

An example of the GUI of the image processing program will now be described with reference to FIGS. 89 to 149. FIG. FIG. 89 shows a function selection screen 340 for selecting various functions, and available functions are listed in a function list display field 341 in the form of buttons. When the "3D search" button 342 is pressed in this state, the three-dimensional search function is executed.
(3D search screen)

Before executing the three-dimensional search function, it is necessary to make necessary settings. Specifically, registration of a search model, registration of a floor on which a workpiece is placed and a storage container, setting of three-dimensional search conditions, and the like are mentioned. These settings can be set by the user arbitrarily for all items, or can be set for each item while sequentially showing the necessary procedures to the user. The guidance function for such setting will be described based on the three-dimensional search screens shown in FIGS. 90 to 101. FIG.

In these three-dimensional search screens, a flow display section 351 for displaying the flow of procedures to be set is provided above the image display column 141 at the bottom left of the screen. In the flow display section 351, the outline of each procedure is indicated by text, patterns, etc., and the procedure currently being set is highlighted. In the example of FIG. 90, the "model registration" icon 352 is displayed in a bright color, and the other "floor/box setting" icon 353 and "search setting" icon 354 are displayed in a dark color. You can notify the subject who is doing Furthermore, in the operation field 142 provided on the right side of the three-dimensional search screen, explanations of the items currently being set are displayed in text or the like, and settings to be made can be explained to the user in characters. The explanation is not only text information, but may be appropriately combined with patterns and moving images as necessary. Also provided are columns for setting parameters to be set.
(Registration of search model based on measured data)

The three-dimensional search screens of FIGS. 90 to 101 show an example of registering actual measurement data obtained by imaging the actual workpiece with the sensor unit as a search model. This registration procedure can be performed according to the procedure described in the flow chart of FIG. In the three-dimensional search screens of FIGS. 90 to 101, the "model registration" icon 352 is highlighted as the first procedure in the flow display section 351 as described above.

FIG. 90 shows a search model registration method selection screen 350 for selecting a method of registration as a search model. The operation column 142 of the search model registration method selection screen 350 is provided with a model registration method selection column 355 for selecting a model registration method. In this model registration method selection field 355, as a model registration method, it is possible to select with a radio button whether to register by photographing an actual workpiece or to register from CAD data. In the example of the search model registration method selection screen 350 in FIG. 90, the image of the actual work is selected for registration. is displayed in letters and patterns. In this example, "Model the 3D information of the workpiece surface that can be seen from the camera. Register as many surfaces as you want to search. This tool matches the registered surface model with workpieces that take various postures. You can recognize the position and posture of the work by letting it move." The work to be done is indicated to the user with text information and patterns. This allows the user to easily grasp the work to be done on this screen. When the user selects the registration of the actual work on the search model registration method selection screen 350 of FIG.
(Actual measurement data imaging unit)

The procedure for actually registering an actual work is shown in FIGS. 91-93. In this example, the work of model registration is divided into three screens, which are sequentially explained to the user. First, on the actual measurement data imaging screen 360 in FIG. 91, the workpiece is imaged as the first work of model registration. In this example, in the operation field 142, "1. Place the workpiece on a plane so that the surface of the workpiece to be registered can be seen by the camera, and press the "measurement execution" button on the lower right." 2. Arrange the model area so as to surround the workpiece on the left camera screen. is explained. Furthermore, by pressing the "?" button 364, a detailed explanation of the optimum work placement can be displayed on another screen. As a result, it is possible to avoid a situation in which a large amount of information is displayed on the screen, making it difficult to see the screen and confusing the user. In this way, the user places the workpiece at a position where the sensor unit can pick up the image, and positions the workpiece while checking the state of the workpiece in the image display field 141 . Then, when the "execute measurement" button 361 is pressed, the photographed work is displayed as shown in FIG. Here, the image is displayed as a three-dimensional image in which the color is changed according to the height of the obtained workpiece. It should be noted that the height image is not limited to the color display, and may be a height image that expresses the height with the luminance value of the pixel. In the example of FIG. 91, six works of the same shape are arranged and arranged in different postures. In this state, the user sets the model area 362 so as to enclose the image of the work desired to be registered as the search model. After finishing the setting of the model area 362, the “next” button 363 is pressed. As a result, the screen is switched to the search model exclusion area setting screen 370 shown in FIG.
(Background removal setting part)

On the search model exclusion area setting screen 370 in FIG. 92, as the second work of model registration, an exclusion area to be excluded when registering as a search model is set. As an example of the search model exclusion area, there is a function of removing the background such as the floor on which the work is placed when registering the actual work. In the search model exclusion area setting screen 370 of FIG. 92, as the second work for model registration, an operation field 142 explains "1. Set background removal so that only the workpiece surface to be registered remains." be. Further, by pressing the "?" button 371, a detailed explanation of the background removal setting method can be displayed on another screen. Further, a background removal setting field 372 is provided in the middle of the operation field 142, and the height to be removed as the background is designated numerically. In this example, 1.5 mm is specified, and the range from 0 to 1.5 mm in height within the range specified as the model area 362 is excluded from the search model registration target. The model region 362 displayed in the image display column 141 is updated according to the setting in the background removal setting column 372, and the region excluded from the registration target of the search model is displayed in black. While confirming this state, the user can judge whether or not the setting of the exclusion area is appropriate, and can perform resetting or the like as necessary.
(Mask area setting section)

In addition to the height direction, it is also possible to specify a range to be excluded from search model registration targets as a mask region in the planar direction. Below the background removal setting column 372 of the search model exclusion area setting screen 370 in FIG. 92, there is an explanation "2. Place a mask area if something other than the workpiece surface to be registered remains." . Similarly, by pressing the "?" button 373, a detailed explanation of the mask area setting method can be displayed on another screen. When a "mask area" button 374 is pressed, it is possible to set a mask area within the range specified as the model area 362 as an area to be excluded from search model registration targets. The mask area can be set by automatic detection of a free-form curve or a boundary portion, in addition to a prescribed figure such as a rectangle or circle. Also, as described above, the model area 362 displayed in the image display field 141 is updated according to the setting of the mask area.

In the example of FIG. 92, mask area setting is performed after background removal setting, but the order of these may be changed. After completing the setting of the area to be excluded from the registration of the search model in this manner, the "Next" button 375 is pressed. This switches to the rotational symmetry setting screen 380 shown in FIG.
(Rotational symmetry setting part)

In the rotational symmetry setting screen 380 of FIG. 93, the rotational symmetry is set as the third work of model registration. In this example, the operation column 142 explains "1. If the work surface to be registered has rotational symmetry, specify the corresponding one." Similarly, by pressing the "?" button 381, a detailed explanation of how to set rotational symmetry can be displayed on another screen. A rotational symmetry setting field 382 is provided in the middle of the operation field 142, and a rotational symmetry candidate can be selected from options such as circular symmetry, N-fold symmetry, and none. Furthermore, when N-fold symmetry is selected, a numerical input box for the number of symmetry is displayed, and the number of rotational symmetry is set according to the shape of the workpiece surface. For example, if the surface of the work is rectangular and the appearance of the work surface is the same when rotated 180 degrees with respect to the axis viewed directly above, 2 is the appropriate value for the number of rotational symmetry. Alternatively, if the surface of the work is a square and the appearance of the work surface is the same when rotated every 90 degrees with respect to the axis viewed directly above the work surface, 4 is an appropriate value for the number of rotational symmetry. Alternatively, if the work surface is hexagonal and the appearance of the work surface is the same when rotated every 30 degrees with respect to the axis viewed directly above, 6 is an appropriate value for the number of rotational symmetry. On the other hand, circular symmetry is set when the appearance of a spherical or cylindrical workpiece is the same for any rotation angle, such as when the surface of the workpiece looks circular when viewed from directly above. is recommended. When rotational symmetry is set, even if a workpiece is in a posture specified as rotational symmetry when registering a search model, it will be recognized as the same workpiece. For example, in the case of a 2-fold rotational symmetry work whose number of rotational symmetry is set to 2, the posture rotated by 180 degrees and the posture before rotation are recognized as the same work.

After completing the setting of the rotational symmetry in this manner, the “Register” button 383 is pressed. Thereby, the search model is registered according to the set conditions. Note that the second and third tasks among the three tasks required for model registration may be interchanged.
(Multiple registration of search models)

When the search model is registered, a search model registration screen 390 shown in FIG. 94 is displayed. The operation field 142 of the search model registration screen 390 is provided with a model list display field 391 for displaying a list of registered search models. Here, a search model with model number 0 is displayed together with a reduced image as a registered search model. It is also displayed that the rotational symmetry setting is "none" for the search model.

To register additional search models, press the “Add” button 392 . As a result, the search model can be registered in the same procedure as described above. As a method for registering the search model, it is possible to select in the model registration method designation field 393 whether to register by photographing the actual workpiece or to register from the three-dimensional CAD data. Here, "registration with actual work" is selected in the model registration method specification field 393 to additionally register.

FIG. 95 shows a search model registration screen 410 to which search models have been added in this way. In the search model registration screen 410 shown in this figure, of the six works arranged in the image display field 141, the work arranged in the lower left is additionally registered as a new search model. The additionally registered search model is added as model number 1 to the model list display field 391 .
(Display of simple three-dimensional search results)

Furthermore, on the search model registration screens of FIGS. 94 and 95, it is possible to display the result of a simple three-dimensional search using a registered search model. For example, in the example of FIG. 94, as a result of performing a three-dimensional search on the six works displayed in the image display field 141 using the search model with model number 0, the upper left work used for registration was searched. there is A point group matching the search model is displayed on the searched workpiece. In this example, of the surface shape of the search model, the planar point group is displayed in white, and the point group indicating the contour is displayed in purple. Furthermore, the score calculated as the evaluation index of the simple three-dimensional search result can be displayed numerically. In the example of FIG. 94, 96.4 is displayed as the score value. It should be noted that the reason why the score does not reach the maximum value (99.9 in this example) for the original work registered as the search model is that the imaging state at the time of model registration and the input image at the time of execution of the three-dimensional search fluctuate. This is because there is a slight decrease in the score. From the result of the score value, it can be understood that the state matches the state at the time of model registration by approximately 96% or more.
(Search area setting section)

After the search model is registered in this manner, a search area is set as the setting of the three-dimensional search function. Specifically, after completing the registration of the necessary search models on the search model registration screen 410 in FIG. 95, pressing the lower right "Complete" button 411 proceeds to the search area setting screen 420 in FIG. On this search area setting screen 420, a search area for three-dimensional search is specified. In the flow display area 351 at the top of the screen, the highlight display shifts from the "model registration" icon 352 to the "floor/box setting" icon 353, and after the model registration is completed, the floor surface on which the work is placed and the storage container are displayed. The user is shown to be in the process of registering for.

A search area setting field 421 is provided in the operation field 142 of the search area setting screen 420 . The search area setting field 421 is provided with a "designation method selection navigation" button 422 for explaining to the user how to designate a search area for three-dimensional search, and a designation method selection field 423 for selecting a search area designation method. be done. When a "designation method selection navigation" button 422 is pressed, a navigation screen is displayed that guides the user on how to designate a search area for performing a three-dimensional search. On the other hand, in the specification method selection column 423, one of floor specification, box specification, box search, and no specification can be selected from the drop box. If floor specification is selected from among these, a search area setting screen 430 shown in FIG. 97 is displayed, and a floor specification dialog 431 is displayed. is presented to the user. Accordingly, when the user designates an arbitrary point indicating the floor surface on the image display field 141 with a pointing device such as a mouse, the spatial coordinates (X, Y, Z) of the designated point are displayed. Also, a gauge is displayed as an extraction area. This extraction area represents the range to be referenced from the clicked position when calculating the Z-coordinate value of the clicked point. Since the 3D measured point-by-point data contains errors, it is better to use the average value or the median value of the data within a certain range in order to suppress the influence of the error. preferable. In this case, the Z coordinate value is obtained from the average value of the range specified by the extraction area. When the floor is specified in this way, the floor information is calculated, and the calculated floor information is displayed in the search area setting column 421 as in the search area setting screen 440 of FIG. It is displayed in column 441 . Here, in the floor surface information display field 421, the X inclination, the Y inclination, and the Z intercept of the floor surface are displayed as numerical values as floor surface information. Also, the offset amount can be set as needed. For example, considering variations in the thickness of the floor surface of the storage container, a predetermined height from the floor surface is removed from the search area. In the example of FIG. 98, a noise removal column 442 is provided to remove noise from the floor surface. Here, the range from the height of the specified floor surface to the height of 1.5 mm specified in the noise removal column 442 is excluded from the three-dimensional search target area. When the setting of the floor surface is completed as described above, the "Complete" button 443 is pressed. This switches to the search parameter setting screen 450 of FIG.
(Search parameter setting section)

On the search parameter setting screen 450 of FIG. 99, search parameters are set as conditions for executing a three-dimensional search. In the flow display area 351 at the top of the screen, the highlight display shifts from the "floor/box setting" icon 353 to the "search setting" icon 354, and after the search area setting is completed, the current stage shifts to the stage of setting the search conditions. The user is shown that The operation column 142 of the search parameter setting screen 450 includes columns for setting search parameters necessary for executing the three-dimensional search. Specifically, a detection condition setting field 451, a feature extraction condition setting field 452, and a determination condition setting field 453 are provided. Among these, the feature extraction condition setting column 452 sets an edge extraction threshold, which is a threshold for extracting edges when detecting a workpiece from an input image. In the judgment condition setting column 453, upper and lower limits are set for the measured value of the score, or upper and lower limits are set for the measured value of the X coordinate of the position, and a judgment condition is set as to whether or not to select it as the search result. do.

In the detection condition setting column 451, conditions for detecting workpieces are set. In this example, a detection number designation field 454 for designating the upper limit of the number of workpieces to be detected, a tilt angle upper limit designation field 455 for designating the upper limit of the tilt angle of the detected work, and a score for designating the lower limit of the score that is the evaluation index. A lower limit specification field 456 is provided. In addition to being specified numerically, it is also possible to continuously adjust using a slider or the like. Further, when the detail setting button 457 is pressed, the search parameter setting screen 460 shown in FIG. 100 is displayed, and a detection detailed condition setting dialog 461 for setting more detailed detection conditions is displayed.

In the detection detailed condition setting dialog 461, a basic setting field 462, a detection condition detailed setting field 463, and an option setting field 464 are provided. The basic setting field 462 includes a search sensitivity setting field 465 for setting the sensitivity of the three-dimensional search, and a search accuracy setting field 466 for setting the precision of the three-dimensional search, in addition to the detection number designation field 454 . Further, in the detection condition detail setting field 463, in addition to an inclination angle upper limit designation field 455 and a score lower limit designation field 456, a rotation angle range setting field 163 includes a reference angle setting field 163a for setting a reference angle and an angle range setting field 163a. A range setting field 163b for setting is provided.

In addition, the detection conditions can be set in common for all search models, or can be set individually for each search model. In the example of FIG. 100, an individual search model designation field 467 is provided, and by turning on this check box, it is possible to individually set the three-dimensional search detection conditions for each search model. Such an example is shown in the search parameter setting screen 470 of FIG. Here, when the check box of the search model individual designation column 467 is turned ON, a search model selection column 467b is displayed, and a search model for setting detection conditions can be selected. This example shows a state in which search model A is selected. The range setting field 163b and the score lower limit designation field 456 can be specified individually.

On the other hand, in the option setting field 464 shown in FIG. 100 and the like, a label order designation field 468 for defining the order of labels and a judgment label designation field for designating a judgment label are provided for label numbers assigned as serial numbers for identifying search results. A column 469 is provided. In the label order specification field 468, descending order or ascending order of correlation values, descending order of height in the Z direction, ascending order, or the like can be selected. The correlation value here is synonymous with the above-mentioned "score". In the determination label specification field 469, it is possible to specify a target for determination as to whether or not a specific search result is in a different position or orientation than expected. Here, a serial number is used to identify the search result. For example, this setting is made when it is necessary to determine whether the search results are unreliable data, such as whether the search results are at an unexpected position, rotation angle, or tilt angle.

When the search settings for the three-dimensional search are sequentially completed in this way, the "OK" button 471 provided at the lower right of the operation field 142 is pressed to end the condition setting for the three-dimensional search.
(Registration of search model based on three-dimensional CAD data)

In the above example, the procedure for photographing and registering an actual workpiece as a search model has been described (corresponding to FIG. 62, etc.). However, as described with reference to FIG. 28 and the like, the present invention can also register three-dimensional CAD data representing the shape of a pre-made workpiece as a search model. A procedure for registering such three-dimensional CAD data as a work model will be described below with reference to FIGS. 102 to 106. FIG.

First, in the search model registration method selection screen 480 of FIG. 102, the radio button for registering from CAD data (STL format) is selected from the model registration method selection column 355 . In response to this, in the operation field 142, an explanation of the procedure for registering the three-dimensional CAD data as a search model is displayed in characters and patterns. In this example, "Model the three-dimensional information of the workpiece surface that can be seen from each positive and negative direction of the XYZ axes (6 directions in total) in the three-dimensional CAD data. Select only the surface that you want to search from the generated model. This tool recognizes the position and posture of a workpiece by matching registered surface models to workpieces that take various postures." shown in In this way, when registration from CAD data is selected on the search model registration method selection screen 480 of 102 and the "OK" button 481 is pressed, the CAD data reading screen of FIG. 103 is displayed. Specifically, the screen shifts to a search model registration screen 490, and a CAD data reading dialog 491 is displayed.
(CAD data reading unit)

In the CAD data reading dialog 491 shown in FIG. 103, the storage destination of the CAD data to be read is specified, and the CAD data is selected. In addition to the storage unit 9, a reading medium such as a semiconductor memory can be selected as the reading destination. When a file is selected from the file list display column 492 and an "execute" button 493 is pressed, reading of the three-dimensional CAD data is executed.
(Registered model selection part)

When the three-dimensional CAD data is read, a surface to be registered as a search model is selected. Here, the search model registration screen 510 shown in FIG. 104 is displayed, and six views corresponding to the basic direction image of the work model are automatically displayed on the registration model selection screen 511 . In this state, the user can select the surface to be registered. Here, a check box is turned on by default, and surfaces that do not need to be registered as a search model can be excluded from the search model by turning off the check box. When the registration target is selected in this manner, the "next" button 512 is pressed to proceed to the search model registration screen 520 in FIG.
(rotational symmetry specification part)

In the search model registration screen 520 of FIG. 105, a rotational symmetry designation dialog 521 is displayed, and the user can select one of the basic direction images selected for registration that has rotational symmetry, such as up and down, front and back, left and right, etc. A surface having a symmetrical shape can be set even if it is rotated. After completing the setting of rotational symmetry in this way, the "Register" button is pressed to register the search model.

When the search model is registered, a list of the registered search models is displayed in the model list display column 391 as in the search model registration screen 530 of FIG. As described above, a plurality of search models can be registered for one work. For example, in the example of FIG. 106, six search models A, B, C, D, E, and F shown in FIG. Therefore, in the model list display column 391, the search models A to F with the model number 0 are listed. Search models that cannot be displayed on one screen in the model list display field 391 can be scrolled to switch the display.
(Search model editorial department)

Furthermore, it is also possible to edit registered search models. In the example of FIG. 106, pressing the "Edit" button 531 in the operation field 142 displays a model edit screen 620 shown in FIG. In this model edit screen 620, the registered search model is displayed in the image display field 141, and the set items are displayed in the operation field 142. FIG. Here, a model region setting field 621, a background removal setting field 327 as the feature setting field 622, a rotational symmetry setting field 382, and a model number display field as the model registration field 623 are provided. The user can select an item to be changed and change it as appropriate.
(Model area detail setting part)

When a detail setting button 624 provided in the model area setting field 621 is pressed, a model area detail setting screen 630 shown in FIG. 115 is displayed. The model area detail setting screen 630 enables setting of a mask area within the range of the model area 362 displayed in the image display field 141 as an area to be excluded from search model registration targets. The operation field 142 has a model area designation field 621 , a mask area designation field 631 , and a weighting area designation field 632 . A plurality of mask areas can be set in the mask area specification column 631 . Each mask area can be set by automatic detection of a free curve or a boundary portion, in addition to a prescribed figure such as a rectangle or a circle. In addition, weighting can be set in a weighting area designation field 632 separately from each mask area.

When the search results are scored, if there is a feature point that is to be emphasized more than other feature points included in the search model, the score result can be adjusted by surrounding it with a weighting area. For example, if there is a shape in which the presence or absence of some features in the overall shape leads to the identification of the search model, or if there is a shape in which the posture in the direction of rotation is determined, a weighting region is set for that important feature. By doing so, it is possible to improve the detection capability of the search. Also, the degree of weighting for other feature points can be designated by a ratio. For example, when the weighting degree is set to "3", the score value can be calculated with the degree of influence three times higher than that of other feature points.
(Feature detail setting section)

Further, when a detailed setting button 625 provided in the feature setting field 622 is pressed, a detailed feature setting screen 640 shown in FIG. 116 is displayed. On the detailed feature setting screen 640 , feature settings and whether or not there is rotational symmetry are set for the model region 362 displayed in the image display field 141 . The operation field 142 is provided with a detailed feature setting field 641 and a rotation target setting field 382 . The feature detail setting field 641 is provided with a feature extraction interval designation field 642 for designating the feature extraction interval, a background removal setting field 327, and an edge extraction threshold value setting field 643 . The feature extraction interval is a parameter that affects the feature point extraction interval at the time of search model generation, and can be selected from among "precision priority", "standard", and "speed priority". When "precision priority" is selected, the extraction interval of feature points in the search model becomes narrower, and even finer features are captured, thereby improving the detection accuracy. On the other hand, when "speed priority" is selected, the extraction interval of feature points in the search model is widened, and fine features are omitted, resulting in a decrease in detection accuracy. It can be adjusted according to the shape and size of the work to be used, the allowable processing time and detection accuracy.

In this way, the user can refer to the results of the simple three-dimensional search, the scores obtained, etc., and adjust the necessary items so as to obtain more appropriate search results. After the registration of the search model is completed in this way, the subsequent procedures (search area setting, search parameter setting) are performed by imaging the actual workpiece and registering the search model, as explained with reference to FIGS. The procedure is the same as the procedure for performing this procedure, so the explanation is omitted.

After registering the search model in advance as described above, the three-dimensional search is performed. The 3D search is used in actual operation, that is, when picking a group of randomly piled workpieces by imaging them with a sensor and specifying the gripping position of a workpiece that can be gripped. It is also used when performing The three-dimensional pick simulation refers to whether or not each search result searched by the three-dimensional search unit 8k can be gripped by the end effector at a gripping position specified in advance for the workpiece model by the gripping position specifying unit 8d. This is an operation for determining whether or not
(Search result display area)

Next, an example of performing a three-dimensional search on randomly stacked works will be described with reference to FIGS. 107 to 113. FIG. FIG. 107 shows an example of a search result display screen 540 showing the result of executing a three-dimensional search on randomly stacked works. The search result display screen 540 is provided with an image display column 141 and an operation column 142. In the operation column 142, a model list display column 391 displays a list of search models used for the three-dimensional search in the vertical direction. On the other hand, in the image display field 141, a point cloud as a search result and a score as an evaluation index are displayed overlaid on the work obtained as the search result for the randomly piled work obtained as the input image. I am letting
(Search result list section)

A search result list display field 541 is provided above the image display field 141, and a list of search models and scores used for the searched search results is displayed. Here, the number of searched works, presence/absence of work, total volume/volume of work, model number and score of search result of specific label number, position (X, Y, Z), etc. are displayed. The total volume/work volume indicates the value obtained by dividing the total volume of the point cloud at a position higher than the floor or the bottom of the returnable box by the average volume assumed from the registered search model. This data indicates how many workpieces are present in the image. If this numerical value is greater than a certain value, it is determined that there is a work presence (1). If the number of pieces searched is 0 and a gripping solution cannot be obtained, it is possible to confirm whether or not the result of the presence or absence of the workpiece is present or absent, thereby determining whether the picking ended when there were actually no workpieces. It is possible to check whether it ended with remaining.

The display of the search result list display field 541 can be switched. For example, a switch button 542 for switching display contents is provided at the lower right of the search result list display field 541 on the search result display screen 540 shown in FIG. The switch button 542 is composed of left and right arrows, and a screen number 543 is displayed above it. In this example, "1/11" is displayed as the number of screens 543 to indicate that it is the first screen out of 11 screens. The display contents of the display field 541 can be switched. For example, when the switching button 542 is operated to change the number of screens 543 to "2/11", the screen switches to the search result display screen 550 shown in FIG. be. Here, in addition to the number of pieces searched, presence/absence of workpieces, total volume/volume of workpieces, inclination angle, rotation angle, orientation (R _X , R _Y , R _Z ), etc. are displayed. When the number of screens 543 is changed to "3/11", the screen switches to the search result display screen 560 shown in FIG. , the total volume/work volume, and the XYZ inclination etc. are displayed as floor surface information.

Further, when the number of screens 543 is changed to "4/11", the screen switches to the search result display screen 570 shown in FIG. , the model number and score of the search model used to detect this search result are displayed in the horizontal direction. Further, when the number of screens 543 is changed to "5/11", the screen switches to the search result display screen 580 shown in FIG. , the position (X, Y, Z) is displayed. Further, when the number of screens 543 is changed to "6/11", the screen switches to the search result display screen 590 shown in FIG. The tilt angle and the rotation angle are displayed every time. Further, when the number of screens 543 is changed to "7/11", the screen switches to the search result display screen 610 shown in FIG. The attitude (R _X , R _Y , R _Z ) is displayed every time.
(three-dimensional pick)

Next, a procedure for simulating a three-dimensional pick to determine whether or not each search result of the three-dimensional search can be gripped by the end effector will be described with reference to FIGS. 117 to 149. FIG. When the "3D pick" button 343 is pressed on the function selection screen 340 of FIG. 89, the 3D pick initial setting screen 650 of FIG. so that the guidance function is executed. Here, a 3D pick guidance field 655 is displayed in the upper part of the operation field 142, and the operations to be performed sequentially are displayed with icons. In this example, four steps of "initial setting", "holding registration", "conditions/sweepstakes", and "place setting" are displayed.
(3D pick initial setting screen)

On the 3D pick initial setting screen 650 in FIG. 117, initial settings for 3D picking are first performed. In the 3D pick guidance field 655 at the top of the operation field 142, the "initial setting" icon is highlighted to show the user what to do at this stage. The operation field 142 is provided with a calibration data selection field 651 , a detection tool setting field 652 , and a hand registration field 653 . In the calibration data selection field 651, the calibration data calculated in order to coordinate-transform the coordinate system displayed in advance by the image processing device into the coordinate system in which the robot controller operates the end effector is called and registered. A detection tool setting field 652 specifies means for detecting a workpiece. In this example, the three-dimensional search for workpiece detection set to tool number 100 is selected. The search model settings registered through the three-dimensional search settings described above are read. When search models for different workpieces are set in advance for different tool numbers, by switching the selection of this detection tool, it is possible to switch the set targets for three-dimensional picking.
(End effector model setting part)

In the hand registration field 653, an end effector model is registered. If the "create hand model" button 654 is pressed here, an end effector model setting screen 660 shown in FIG. 118 is displayed. On the end effector model setting screen 660 , a list of registered end effector models is displayed in a hand model list display field 661 . Since the example in FIG. 118 is unregistered, no end effector model is displayed in either the image display field 141 or the end effector model list display field 661 of the operation field 142 . Therefore, an "Add" button 662 is pressed to add an end effector model. In the image display field 141 of the end effector model setting screen 660, the flange surface FLS is displayed as the reference surface of the arm tip of the robot on which the end effector model is arranged. Furthermore, a tool coordinate axis TAX indicating the coordinate axis on the end effector model EEM side is also displayed (details will be described later).
(End effector model editorial department)

When the "Add" button 662 is pressed in FIG. 118, an end effector model editing screen 670 shown in FIG. 119 is displayed. On the end effector model edit screen 670, the parts that make up the end effector model are set. Here, registered parts are displayed in the parts list display field 671 . In the example of FIG. 119, since no parts have been registered yet, nothing is displayed in the parts list display field 671 . When the "Add" button 672 is pressed in this state, a parts addition dialog shown in FIG. 120 is displayed, and the part to be added is registered by reading CAD data, or newly created in the form of a rectangular parallelepiped model or a cylinder model. Select by radio button. Note that the additional part is one form of the additional region described above, and is not limited to a rectangular parallelepiped or a cylinder, and can be of any shape such as a prismatic shape, a triangular shape, a spherical shape, or the like. In the example of FIG. 120, CAD data is selected. When the "Add" button 681 is pressed to select the created three-dimensional CAD data, a CAD position/orientation setting screen 690 shown in FIG. Along with the CAD model CDM being displayed, the operation field 142 also displays a position/orientation field 691 for designating the position and orientation of this CAD model CDM. The position and orientation of the CAD model CDM are specified with reference to the flange surface FLS. The user designates the mounting position and angle of the end effector model from the position/orientation column 691 . For example, if 20° is specified as the rotation angle R _X from the state shown in FIG. 121, the CAD model CDM displayed in the image display field 141 is rotated 20° around the X axis as shown in FIG. . Alternatively, if 50 mm is specified in the Z direction from the state of FIG. 121, the CAD model CDM is displayed with an offset of 50 mm in the Z direction as shown in FIG. After adjusting the position and posture of the CAD model CDM in this way, pressing the “OK” button 692 in FIG.

Multiple parts can be added. For example, when the "Add" button 682 is pressed from the end effector model edit screen of FIG. 124, the cylinder model is selected in the parts addition dialog 680 of FIG. The posture setting screen is displayed, and a columnar model is newly displayed in the image display field 141 as an additional area ADA. The part being edited is highlighted. In the example of FIG. 125, the additional area ADA is displayed in red, and the CAD model CDM is displayed in gray or wireframe. This makes it easier for the user to visually grasp the parts that are currently being edited. Similarly, the position and orientation of the additional area ADA to be added is designated from the position and orientation column 691 . In this example, the Z position is specified as −50 mm in the position/orientation column 691, and the additional area ADA is arranged in the direction opposite to the CAD model CDM from the flange surface FLS. Also, as the size of the additional area ADA, the radius and height of the cylindrical model can be specified in the size specification field 693 . After setting the conditions for the additional area ADA in this way, pressing the "OK" button 692 registers the additional area ADA, and as shown in FIG. is added and displayed.
(Tool coordinate specification part)

It is also possible to adjust the position and orientation of the registered parts. For example, if the part to be selected is changed from the additional area ADA to the CAD model CDM from the end effector model edit screen of FIG. , the position (X, Y, Z) and orientation (R _X , R _Y , R _Z ) can be adjusted in the tool coordinate designation field 673 . For example, as shown in FIG. 128, when the Z position is designated as 145 mm, the tool coordinate axis TAX is displayed in the image display field 141 by being moved in the Z direction from the flange surface FLS by the designated distance. Further, in this state, if the rotation R _X is specified to be 20° as shown in FIG. 129, the tool coordinate axis TAX is rotated 20° clockwise about the X axis and displayed.

A plurality of parts are registered in this way, and the end effector model EEM is constructed by combining them. When the setting of each part is completed, pressing the "OK" button 674 on the end effector model editing screen shown in FIG. As indicated by 660 , “hand 0” is displayed in the hand model list display field 661 . Also, in the image display column 141, the parts that were displayed separately are displayed as an integrated end effector model EEM.

In this example, an example in which parts are combined to form an end effector model has been described, but parts are not necessarily limited to end effector models or parts thereof, and cables and the like can also be represented by parts. In this way, a part is a form of an additional model representing a part of the end effector model, an object added to its surface, or a peripheral member.
(Grip reference point HBP)

Furthermore, the gripping reference point HBP of the end effector model EEM is superimposed on the image display field 141 and displayed. The gripping reference point HBP indicates the position at which the end effector model EEM grips the workpiece. For example, in the end effector model EEM, the midpoint between the claws that grip the workpiece is set as the gripping reference point HBP. The gripping reference point HBP may be calculated by the image processing apparatus so as to be determined according to a predetermined rule, or may be configured so that the user can designate any position.

Furthermore, it is preferable that the tool coordinate axes that define the position and orientation of the end effector model EEM are also displayed superimposed on the image display field 141 . The tool coordinate axes are preferably rotated coordinate axes RAX that are varied in accordance with the rotation of the end effector model EEM. Furthermore, it is more preferable to match the origin of the rotated coordinate axis RAX with the gripping reference point HBP. This makes it easier for the user to visually grasp the state of the end effector model EEM when setting the gripping position.
(Grip registration unit)

After the end effector model EEM is registered as the initial setting in this manner, the holding position is registered. On the gripping registration screen 710 in FIG. 131, the highlight display in the 3D pick guidance field 655 changes from the "initial setting" icon to the "gripping registration" icon. The operation field 142 includes a search model switching field 711 for selecting a registered search model, and a gripping registration list display field for displaying a list of registered gripping positions for the search model selected in the search model switching field 712 . 712 are displayed. As described above, in this example, the work model for registering gripping positions is shared with the search model used for the three-dimensional search. Here, it is assumed that six search models A to F with model number 0 registered in FIG. 104 are registered in the search model switching column 711 as registered search models. In the example of FIG. 131, since the grip position has not yet been registered, nothing is displayed in the grip registration list display field 712 . The search model A is selected in the search model switching field 711 and the search model SMA corresponding to A is displayed in the image display field 141 . Furthermore, a reference coordinate axis BAX, which is a coordinate axis indicating the position and orientation of the search model SMA, is displayed in an overlapping manner. From this state, the user registers gripping positions for the search model A. FIG. Specifically, when the "Add" button 713 is pressed, the screen is switched to the grip registration screen shown in FIG. 132, and the grip setting dialog 720 is displayed.
(Grip setting part)

In the gripping setting dialog 720 , the operation field 142 is provided with a basic setting field 721 , a gripping position coordinate designation field 724 , and a movement amount setting field 727 . The basic setting field 721 has a gripping label setting field 722 and a hand selection field 723 . In the gripping label setting field 722, a gripping label, which is identification information for specifying the gripping position, is set. Here, as gripping labels, "A" indicating the search model and the gripping position serial number "000" are set. Also, in the hand selection field 723, the end effector model EEM that grips the grip position set in the grip label setting field 722 is selected. In this example, "hand 0" is selected in the hand selection field 723, and the end effector model EEM is displayed in the image display field 141 accordingly. In this way, the grip position is set in association with the end effector model EEM.

In the grip position coordinate specification field 724, the user can directly specify the coordinate position (X, Y, Z) and orientation (R _X , R _Y , R _Z ) indicating the grip position by numerical values; A “simple setting navigation” button 726 is provided for executing a guidance function for guiding the user to set the gripping position.

Also, in the movement amount setting field 727, the direction in which the end effector model EEM is moved is defined. Here, the approach movement amounts for bringing the end effector model EEM closer to the search model SMA are defined as distances in the X, Y, and Z directions so that the search model SMA is gripped by the end effector model EEM. The example in FIG. 132 shows an example of moving 100 mm in the Z direction.
(Grip position setting guidance function)

When the "simple setting navigation" button 726 is pressed from the screen of FIG. 132, the gripping position setting guidance function is executed. Specifically, a setting method selection dialog 730 is displayed as shown in FIG. Make a selection with a radio button. In this example, setting on the 3D viewer is selected. When the "OK" button 731 is pressed in this state, the registration guidance function on the 3D viewer is executed.
(Registration guidance function on 3D viewer)

In the registration guidance function on the 3D viewer, the gripping position is registered through four steps. Here, the user is prompted to specify the gripping position through the three-dimensional viewer registration screens shown in FIGS. Specifically, when the "OK" button 731 is pressed in FIG. 133, the screen is switched to the XY designation screen 740 in FIG.
(First step: XY designation)

In FIG. 134, an XY designation screen 740 is displayed as one form of the XY designation section. Here, as a first step for registering the gripping position, the gripping position is designated from the plan view while the search model SMA is displayed in the form of a plan view. Specifically, the search model SMA is displayed two-dimensionally in the XY plane in the image display field 141, and the X and Y coordinates to be gripped are specified on the search model SMA. In the operation column 142, as STEP 1, the operation to be performed is explained to the user by text or the like. Here, "Please select the XY position to be gripped on the camera image." is displayed. In the example of FIG. 134, the search model A is displayed enlarged as a height image (search model SMA) in the image display field 141. In this state, the user uses a pointing device such as a mouse to move the part to be gripped. Select P1. The XY coordinates are entered in the XY coordinate designation field 741 in accordance with the grip designation position P1. Further, when the X coordinate and the Y coordinate in the XY coordinate designation field 741 are adjusted, the grip designation position P1 in the image display field 141 is updated accordingly. In this XY designation screen 740, only the X coordinate and the Y coordinate among the positional parameters can be designated, in other words, other positional parameters cannot be designated. In this way, by allowing multiple position parameters to be set at the same time as in the past, the user is not confused, and by limiting the position parameters that can be set, the user can accurately grasp the position parameters that can be set. Then you will be guided to set the position parameters sequentially. After specifying the XY coordinates in this way, the user presses the “Next” button 742 . As a result, the process shifts to the second step and switches to the ZR _Z designation screen 750 of FIG.
(Second process: Z-R _Z designation)

A _ZRZ specification screen 750 in FIG. 135 is one form of a _ZRZ specification section, and is a screen for allowing the user to specify the _Z coordinate and the orientation RZ. In the operation column 142, the work to be performed by the user in the second step is displayed in text for explanation. Here, "STEP 2 Please specify the Z position to grip and the posture angle R _Z of the hand" is displayed. The operation field 142 is also provided with a Z designation field 752 for designating the Z position and an R _Z designation field 753 for designating the R _Z attitude as a ZR _Z designation field 751 . In the image display field 141, the end effector model EEM and the search model SMA are displayed three-dimensionally. That is, the image display field 141 functions as a three-dimensional viewer that displays the end effector model and the search model in three dimensions. When the values in the Z designation field 752 and the R _Z designation field 753 are changed, the three-dimensional display contents of the end effector model EEM and the search model SMA in the image display field 141 are updated accordingly. Further, by dragging the image display field 141, the user can change the viewpoint of the three-dimensional viewer, so that the grip setting state can be confirmed in detail.

In addition, in the image display field 141, which is a three-dimensional viewer, only the Z-axis is displayed as the coordinate axis related to the position parameter currently being set among the coordinate axes. This Z-axis is the post-correction corrected rotational Z-axis AXZ represented by the ZYX Euler angles described above. In this way, by displaying only the coordinate axes that are currently adjustable and making it impossible to adjust other position parameters on this screen, the user is not unnecessarily confused, and an appropriate display is made according to a predetermined order. Appropriate guidance is realized so as to sequentially set only the appropriate position parameters.

Also, in the example of FIG. 135 and the like, the end effector model EEM is brought closer to the work model for the user by displaying the gripping direction axis (here, the correction rotation Z-axis AXZ) extending through the gripping reference point HBP. It is possible to clearly indicate the direction of movement for the purpose. Alternatively, the direction of movement of the end effector model may be indicated by an arrow. At this time, the movement direction of the end effector model may be displayed superimposed on the gripping direction axis, or may be displayed at a different position.

Furthermore, the ZR _Z designation screen 750 is provided with a “fit” button 154 as one form of the relative position setting section. As described above, the "fit" button 154 implements a fit function that automatically moves the end effector model to the gripping position of the work model. When the "Fit" button 154 is pressed, the end effector model EEM is moved as shown in FIG. That is, the gripping reference point HBP of the end effector model EEM is moved toward the specified gripping position P1 of the search model SMA. It is preferable to adjust the initial position when displaying the end effector model EEM in the state of FIG. 135 so that this moving direction coincides with the Z axis. As a result, by moving the end effector model EEM only in the Z direction, it can be brought into contact with the search model SMA and displayed in an intuitively understandable manner. Easy to do.

Also, according to the movement of the end effector model EEM in the image display field 141, the Z position of the Z designation field 752 in the operation field 142 is changed. In this example, the length is changed from 200 mm in FIG. 135 to 50 mm. Here, with the fit function, when the end effector model EEM is moved in the height direction, that is, in the Z-axis direction, toward the specified gripping position P1 specified in FIG. From here, it is moved to a position returned forward by a predetermined offset amount. Further, only the Z-axis currently being set among the coordinate axes is displayed in the image display field 141 with reference to the specified gripping position P1.

In this state, the user further designates the posture R _Z from the R _Z designation field 753 . In the posture of the end effector model EEM shown in FIG. 136, the side surface of the search model SMA inclined in a V shape is gripped. Adjust to grip a smooth surface. At this time, the rotation of the end effector model EEM can be designated more simply by providing an angle selection button 754 for inputting a predetermined angle in addition to directly inputting the rotation angle in the R _Z designation field 753 by the user. In this example, four angle buttons of 0°, 90°, 180°, and -90° are arranged as angle selection buttons 754 . By pressing the 90° button, the user inputs 90° in the R _Z designation field 753, rotates the end effector model EEM 90° in the Z-axis direction as shown in FIG. 137, and correctly grips the search model SMA. You can fix it as much as you can.
(Third process: R _Y designation)

After completing the ZR _Z specification of the second step in this manner, the "next" button 755 is pressed to proceed to the _RY specification of the third step. Specifically, when the "next" button 755 is pressed, the _RY designation screen 760 shown in FIG. 138 is displayed. The _RY designation screen 760 is one form of the _RY designation section, and designates the posture _RY . An _RY designation field 761 is displayed in the operation field 142 of the _RY designation screen 760, and a message "STEP 3 Please designate the posture angle _RY of the hand" is displayed to the user as work to be performed in the third step. Is displayed. Furthermore, in the image display field 141, only the corrected rotation Y-axis AXY, which is the Y-axis currently being set, is displayed among the coordinate axes. From this state, the user designates the orientation R _Y from the R _Y designation screen 760 in accordance with the orientation of the end effector model EEM holding the search model SMA. Alternatively, the end effector model EEM is dragged and rotated on the image display field 141, which is a three-dimensional viewer. For example, in the state of FIG. 138, if 90° is specified as the posture _RY on the _RY specifying screen 760, the screen switches to the _RY specifying screen of FIG. Is displayed. After completing the _RY designation of the third step in this manner, the "next" button 762 is pressed to proceed to the _RY designation of the fourth step.
(Fourth step: R _Y designation)

FIG. 140 shows the R _X designation screen 770 of the fourth step. The R _X designation screen 770 is one form of the R _X designation section, and is provided with an R _X designation field 771 for designating the posture R _X . In the image display field 141, the corrected rotation _X -axis AXX, which is the _X -axis related to the position parameter Rx that can be set on the Rx designation screen 770 within the coordinate axes, is displayed superimposed on the end effector model EEM and the search model SMA. be. In this state, the user designates the posture R _X from the R _X designation field 771 . Alternatively, the end effector model EEM is dragged and rotated in the image display field 141 . For example, in the example of FIG. 140, 180° is entered as the posture R _X in the R _X specification field 771 corresponding to the current gripping position. The postures of the end effector model EEM and the search model SMA change in the image display field 141, which is a three-dimensional viewer, according to the posture _Rx .

After specifying the posture R _X in this way, the user presses the "Complete" button 772 to finish registering the gripping position. When the gripping position is registered, a gripping registration A-000 is added to the gripping registration list display field 712 as shown in FIG. Also, the search model A corresponding to the holding registration A-000 is highlighted in the search model switching field 711 . Further, in the image display field 141, the manner in which the search model SMA is gripped at the position and posture of the end effector model EEM that has been gripped and registered is displayed three-dimensionally. Furthermore, as the coordinate axis, the reference coordinate axis BAX on the side of the search model SMA is superimposed and displayed.

In the same way, other gripping positions are registered in search model A, or gripping positions are added to other search models BF. A gripping position can be added by pressing the “Add” button 713 provided in the gripping registration list display field 712 as described above. The example of FIG. 143 shows an example of registration of holding registration B-000 for search model B. In FIG.
(copy function)

Also, when additionally registering gripping positions, a copy function for copying already registered gripping positions can be used. By pressing a "copy" button 714 provided in the gripping registration list display field 712 on the screen such as FIG. 142, the registered gripping position can be copied. Furthermore, by correcting the copied gripping position as a base, the gripping position can be registered efficiently. When the "Edit" button 715 is pressed from the state shown in FIG. 142, etc., a gripping setting dialog is displayed, and the gripping position can be set as described above. Here, as described above, it is also possible to press the "simple setting navigation" button 726 to execute the gripping position setting guidance function and correct necessary portions.
(Condition verification screen 780)

After registering the gripping positions in the search models A to F as described above, the user presses the "Complete" button 716 on the screen shown in FIG. In the condition verification process, a three-dimensional search is performed on a group of randomly piled works to detect the work, and it is verified whether or not the end effector can grip the work at a specified gripping position. Here, a simulation is performed using an end effector model EEM and a search model. An example of the condition verification screen 780 is shown in FIG. On this condition verification screen 780, the highlight display in the 3D pick guidance column 655 is changed from the "grip registration" icon to the "condition/verification" icon. The operation column 142 is provided with a detection condition setting column 781 for setting conditions for detecting the gripping position based on the three-dimensional search result, and a verification column 785 for verifying whether or not the detected workpiece can be gripped. In the detection condition setting field 781, a detection number designation field 782 designates the upper limit of the number of gripping solutions to be detected, a hand tilt angle upper limit designation field 783 designates the upper limit of the tilt angle of the end effector model, and the end effector model interferes with each other. A margin setting column 784 is provided for setting a margin from the wall surface of the storage container in order to expand the range of the storage container. When the end effector model EEM enters the range of the distance (5 mm in FIG. 144) specified in the margin setting field 784, it is determined that there is interference.

On the other hand, in the verification column 785, a detection number display column 786 indicating the number of search results actually detected as a result of the simulation of the three-dimensional search and a label number of the search model to be verified among the detected search results are displayed. A display label specification field 787 to be specified and a "verify each work" button 788 for executing verification for each work are provided. The search result of the label number designated in the display label designation field 787 is displayed in the image display field 141. FIG.
(Verification for each workpiece)

Also, when the "verify for each work" button 788 is pressed, a verification dialog 790 shown in FIG. 145 is displayed. The verification dialog 790 is one form of the work selection screen 210 described in FIG. A verification dialog 790 in FIG. 145 verifies whether or not each detected workpiece can be gripped. In this example, the verification dialog 790 has a target work selection field 791 , a detected search model display field 792 and a “hold confirmation” button 793 . When the "confirm grasping" button 783 is pressed, a detection result display dialog 810 shown in FIG. 146 is displayed. This detection result display dialog 810 is one form of the gripping solution candidate display screen 220 shown in FIG. 75 and the like. Here, for each surface (search image and estimated image) detected for the workpiece selected in the target workpiece selection field 791 (the second workpiece in FIG. 145), the determination result of whether or not the workpiece can be gripped at the set gripping position is shown. are listed in the gripping solution candidate display field 811 . In the example of FIG. 146, the gripping label C-000 is selected, and accordingly, the end effector model EEM is arranged and displayed at the corresponding gripping position in the image display column 141, and the end effector model EEM is displayed according to the determination result. The effector model EEM is displayed in color. In this example, the end effector model EEM is displayed in red because the determination result is NG. Also, in the gripping solution candidate display field 811, a point group interference error is displayed as the cause of the inability to grip. This provides the user with guidelines for taking necessary measures, such as correcting or adding gripping positions.

147, when the gripping label A-000, which is determined as OK, is selected, the end effector model EEM gripped at the corresponding gripping position is displayed in the image display field 141 in white. is displayed in In this way, it is possible to determine whether or not the object can be grasped, and to take appropriate measures as necessary.

Further, when a detection condition detail setting button 789 provided in a detection condition setting column 781 is pressed on the condition verification screen 780 of FIG. 144, the screen is switched to a detection condition detail setting screen 820 of FIG. The detection condition detail setting screen 820 is provided with a grip position detection condition setting field 821, an end effector-to-box/floor interference determination setting field 822, and an end effector-to-three-dimensional point group interference determination setting field 823. It is

When the condition verification process is finished in this manner, the place setting process is finally performed to define the placement position of the workpiece. An example of the place setting screen 830 is shown in FIG. In the 3D pick guidance column 655 of the place setting screen 830, the highlighting shifts from the "condition/verification" icon to the "place setting" icon. The operation field 142 is provided with a tool coordinate setting field 831 and a place position registration field 832 . From this screen, the user makes settings related to the work place.

INDUSTRIAL APPLICABILITY The image processing apparatus of the present invention can be suitably used for verifying the operation of a robot in bulk picking.

1000, 2000, 3000, 4000, 7000, 8000, 9000... Robot system 100, 200, 300, 400, 700, 800, 900... Image processing device 2... Sensor unit: 2b... Sensor control unit; 2c... Input image acquisition unit 3, 3B... display unit; 3a... six view display area; 3b... image display area; 3c... gripping solution candidate display area; 3d... workpiece grippability display area; 4 operation unit; 4b input/output interface 5 robot body 6 robot controller; 6b robot interface 7 robot operation tool 8c positioning unit; 8d gripping position specifying unit; 8d3: Gripping reference point setting unit; 8d4: Gripping direction setting unit; 8d5: Relative position setting unit; 8d6: Second designating unit; 8d7: Third designating unit; 8d9 First designating section 8e Basic direction image selecting section 8f End effector mounting surface setting section 8g Search model registering section 8g' Model registering section 8h Rotational angle limiting section 8i Search model Selection section; 8j... Image selection section; 8k... Three-dimensional search section; 8l... Three-dimensional pick determination section; 8m... Interference determination section; 8q: Evaluation index calculation unit; 8r: Grasping priority order determination unit; 8s: Cross-sectional model generation unit; 8t: Work model registration unit; 8u: End effector model registration unit; 8x Conversion unit 8y End effector mounting position calibration unit 8z Image estimation unit 9 Storage unit 9b Holding position storage unit 10 Calculation unit 8e' Basic direction image generation unit 130, 130B Search model registration screen 131..."Registration target" check box; 131B...Selection check box 140...Grip registration screen 141...Image display column 142...Operation column 143...Search model selection column 144...Grip posture display column 145...Edit button 146...Add button 147... Delete button 150 -- Gripping posture addition screen 153 -- Gripping posture coordinate information 154 -- "Fit" button 160 -- Tilt angle/rotation angle setting screen 161 -- Posture restriction search model selection field 162 -- Tilt angle upper limit setting field 163 -- Rotation angle range setting Fields: 163a: Reference angle setting field; 163b: Range setting field 170: End effector mounting position setting screen 171: End effector mounting position setting field 180 Grasping position designation screen 181 Grasping position designation field 190 Multiple gripping position selection screen 191 Surface selection field 192 Gripping position list display 210 Work selection screen 211 Target work selection field 212 Detection search model display field 213 Gripping confirmation" button 220 --- Gripping solution candidate display screen 221 --- Gripping solution candidate display column 223 --- Work gripping position display column 224 --- Work gripping possibility display column 225 --- Work gripping impossible factor display column 230 --- XY designation screen 240 --- Z -R _Z designation screen 241 Z coordinate designation field 242 R _Z rotation angle designation field 250 R _Y designation screen 251 R _Y rotation angle designation field 260 R _X designation screen 261 R _X rotation angle designation field 270 Position Parameter designation screen 271: X, Y coordinate designation field 272: "Next" button 273: Z coordinate, R _Z rotation angle designation field 274: R _X , R _Y rotation angle designation field 280: XYZ designation screen 281 ... X, Y, Z coordinate designation field 290 ... Three-dimensional image viewer 310 ... Additional area setting screen 311 ... Basic figure display field 312 ... "Add" button 313 ... "Edit" button 320 ... Basic figure editing screen 321 ... Basic image parameters Adjustment column 327 Background removal setting column 330 End effector imaging screen 331 End effector position specifying column 332 "Imaging" button 340 Function selection screen 341 Function list display column 342 "3D search" button 343 "3D pick button 350 --- search model registration method selection screen 351 --- flow display section 352 --- "model registration" icon 353 --- "floor/box setting" icon 354 --- "search setting" icon 355 --- model registration method selection field 356 --- "OK" Button 360 -- Actual measurement data imaging screen 361 -- "Measure execution" button 362 -- Model area 363 -- "Next" button 364 -- "? Button 370 Search model exclusion area setting screen 371 "?" button 372 Background removal setting field 373 "?" button 382 rotational symmetry setting field 383 registration button 390 search model registration screen 391 model list display field 392 add button 393 model registration method designation field 410 search model registration screen 411 "Complete" button 420 Search area setting screen 421 Search area setting field 422 Designation method selection navigation button 423 Designation method selection field 430 Search area setting screen 431 Floor designation dialog 440 Search area setting screen 441 Floor information display field 442 Noise removal field 443 "Complete" button 450 Search parameter setting screen 451 Detection condition setting field 452 Characteristic extraction condition setting field 453 Judgment condition setting field 454 Detection number designation field 455 Inclination angle upper limit designation field 456 Score lower limit designation field 457 Detailed setting button 460 Search parameter setting screen 461 Detailed detection condition setting dialog 462 Basic setting field 463 Detection condition detailed setting field 464 Option setting field 465 Search sensitivity setting field 466 Search accuracy setting field 467 Search model individual designation field 467b Search model selection field 468 Label order designation field 469 Judgment label designation field 470 Search parameter setting screen 471 OK button 480 Search model registration method selection screen 481 --- "OK" button 490 --- Search model registration screen 491 --- CAD data reading dialog 492 --- File list display field 493 --- "Execute" button 510 --- Search model registration screen 511 --- Registered model selection screen 512 ... "Next" button 520 ... Search model registration screen 521 ... Rotational symmetry designation dialog 522 ... "Register" button 530 ... Search model registration screen 531 ... "Edit" button 540 ... Search result display screen 541 ... Search result list display column 542 Switch button 543 Number of screens 550 Search result display screens 560, 570, 580, 590, 610 Search result display screen 620 Model edit screen 621 Model area setting column 622 Characteristic setting column 623 Model registration column 624 Detail setting button 625 Detail setting button 630 Model area detail setting screen 631 Mask area designation field 632 Weighting area designation field 640 Feature detail setting screen 641 Feature detail setting field 642 Feature extraction interval specification field 643 Edge extraction threshold setting field 650 Three-dimensional pick initial setting screen 651 Calibration data selection field 652 Detection tool setting field 653 Hand registration field 654 "Create hand model" button 655 3D Pick guidance field 660 End effector model setting screen 661 Hand model list display field 662 "Add" button 670 End effector model editing screen 671 Parts list display field 672 "Add" button 673 Tool coordinate designation field 674 "OK" button 680 --- Parts addition dialog 681 --- "Add" button 690 --- CAD position and orientation setting screen 691 --- Position and orientation field 692 --- "OK" button 693 --- Size designation field 710 --- Gripping registration screen 711 --- Search model switching field 712 Grasp registration list display field 713 Add button 714 Copy button 715 Edit button 716 Finish button 720 Grasp setting dialog 721 Basic setting field 722 Grasp label setting field 723 Hand selection field 724 Grasping position coordinate designation field 725 Coordinate attitude designation field 726 "Simple setting navigation" button 727 Movement amount setting field 730 Setting method selection dialog 731 OK button 740 XY designation screen 741 ... XY coordinate designation field 742 ... "Next" button 750 ... Z-R _Z designation screen 751 ... Z-R _Z designation field 752 ... Z designation field 753 ... R _Z designation field 754 ... Angle selection button 755 ... "Next" button 760 --- R _Y designation screen 761 --- R _Y designation field 762 --- "Next" button 770 --- R _X designation screen 771 --- R _X designation field 772 --- "Complete" button 780 --- Condition verification screen 781 --- Detection condition setting field 782 ... Detected number designation field 783 ... Hand tilt angle upper limit designation field 784 ... Margin setting field 785 ... Verification field 786 ... Detected number display field 787 ... Display label designation field 788 ... "Verification for each workpiece" button 789 ... Detection condition detail setting Button 790 Verification dialog box 791 Object work selection column 792 Detection search model display column 793 "Gripping confirmation" button 810 Detection result display dialog box 811 Gripping solution candidate display column 820 Detection condition detail setting screen 821 Gripping Position detection condition setting column 822 --- End effector and box, collision judgment setting column 823 --- End effector and three-dimensional point group interference judgment setting column 830 --- Place setting screen 831 --- Tool coordinate setting column 832 --- Place position Registration fields 840, 850... Misalignment correction screen 841... End effector setting Column 842 --- "OK" button 843 --- "Automatic correction" button TAX --- Tool coordinate axis BAX --- Reference coordinate axis RAX --- Rotated coordinate axis HBP --- Grasping reference points EET, EET2, EET3 --- End effector ARM --- Arms WK, WK6, WK7, WK10 Work WK2 Hollow work WK3 Plate work WK9 Flat work WKC Cubic work CWM Three-dimensional CAD data work model CDM CAD model WM9 Flat work models WM10 and WM11 Work model WM11F... Work model displayed in a plan view WM11P... Work model displayed in a perspective view WMR... Work model at the time of search model registration WKI... Work ADA obtained as an input image Additional area FLS... Flange surface VP... Cross product vector WKI'... Rotated input image IBX... Virtual rectangular parallelepiped MWM... Measured data BDI... Basic direction image SM... Search model SMA... Search model P1... Specified gripping position SCP... Characteristic point on surface OCP... On contour Feature points EEM, EM10... End effector model PRJ... Projectors CME1, CME2, CME3, CME4... Camera PC... Point group BSL... Fundamental axes SP1 to SP5... Section positions SS1 to SS5... Section TDP... Three-dimensional point PP3... Projection point RBT … Robot BX … Storage container STG … Stage AXX … Correction rotation X axis AXY … Correction rotation Y axis AXZ … Correction rotation Z axis

Claims (8)

Image processing for controlling a robot that performs a picking operation by measuring the three-dimensional shape of multiple workpieces stacked in a work space with a sensor unit and gripping them with an end effector provided at the tip of the arm of the robot with a robot controller. a device,
an input image acquisition unit that acquires an input image representing the three-dimensional shape of a plurality of works from a sensor unit that measures the three-dimensional shape of the work placed in the work space;
a search model registration unit for registering a search model used for a three-dimensional search for specifying the position and orientation of the workpiece from the input image;
an end effector model registration unit for registering an end effector model that virtually represents a three-dimensional shape of an end effector controlled by the robot controller;
a display unit that displays a work model that virtually represents the three-dimensional shape of the work in a three-dimensional manner in a virtual three-dimensional space;
a gripping position specifying unit that specifies a gripping position at which the end effector model grips the work model displayed on the display unit;
a three-dimensional search unit that performs a three-dimensional search for specifying the position and orientation of one workpiece from the input image using the search model;
an inclination angle setting unit that sets an allowable inclination angle range for the one workpiece three-dimensionally searched by the three-dimensional search unit;
Based on the position and orientation of the one workpiece specified by the three-dimensional search unit and the gripping position specified by the gripping position specifying unit for the work model corresponding to the one workpiece, the one a three-dimensional pick determination unit that determines whether or not the workpiece can be gripped by the end effector model;
with
The three-dimensional pick determination unit
an interference determination unit that determines whether or not the end effector interferes with a surrounding object when the one workpiece three-dimensionally searched by the three-dimensional search unit is gripped by the end effector model;
an angle determination unit that determines whether the one workpiece three-dimensionally searched by the three-dimensional search unit is within the tilt angle range;
with
The display unit displays the end effector model when gripping the one workpiece on the input image, and the three-dimensional pick determination unit determines that the one workpiece cannot be gripped by the end effector model. In the case where the gripping failure factor is determined, the gripping failure factor of the one work can be displayed , and further, different gripping failure factors can be displayed according to the determination results of the interference determination unit and the angle determination unit. image processing device . Image processing for controlling a robot that performs a picking operation by measuring the three-dimensional shape of multiple workpieces stacked in a work space with a sensor unit and gripping them with an end effector provided at the tip of the arm of the robot with a robot controller. a device,
an input image acquisition unit that acquires an input image representing the three-dimensional shape of a plurality of works from a sensor unit that measures the three-dimensional shape of the work placed in the work space;
a search model registration unit for registering a search model used for a three-dimensional search for specifying the position and orientation of the workpiece from the input image;
an end effector model registration unit for registering an end effector model that virtually represents a three-dimensional shape of an end effector controlled by the robot controller;
a display unit that displays a work model that virtually represents the three-dimensional shape of the work in a three-dimensional manner in a virtual three-dimensional space;
a gripping position specifying unit that specifies a gripping position at which the end effector model grips the work model displayed on the display unit;
a three-dimensional search unit that performs a three-dimensional search for specifying the position and orientation of one workpiece from the input image using the search model;
Based on the position and orientation of the one workpiece specified by the three-dimensional search unit and the gripping position specified by the gripping position specifying unit for the work model corresponding to the one workpiece, the one a three-dimensional pick determination unit that determines whether or not the workpiece can be gripped by the end effector model;
with
The display unit displays the end effector model when gripping the one workpiece on the input image, and the three-dimensional pick determination unit determines that the one workpiece cannot be gripped by the end effector model. In the case where the one workpiece is not grippable, it is configured to be able to display the cause of the inability to grip the one workpiece, and furthermore, the display mode of the end effector model is changed and displayed according to the determination result of the three-dimensional pick determination unit. An image processing device configured to be able to: Image processing for controlling a robot that performs a picking operation by measuring the three-dimensional shape of multiple workpieces stacked in a work space with a sensor unit and gripping them with an end effector provided at the tip of the arm of the robot with a robot controller. a device,
an input image acquisition unit that acquires an input image representing the three-dimensional shape of a plurality of works from a sensor unit that measures the three-dimensional shape of the work placed in the work space;
a search model registration unit for registering a search model used for a three-dimensional search for specifying the position and orientation of the workpiece from the input image;
an end effector model registration unit for registering an end effector model that virtually represents a three-dimensional shape of an end effector controlled by the robot controller;
a display unit that displays a work model that virtually represents the three-dimensional shape of the work in a three-dimensional manner in a virtual three-dimensional space;
a gripping position specifying unit that specifies a gripping position at which the end effector model grips the work model displayed on the display unit;
a three-dimensional search unit that performs a three-dimensional search for specifying the position and orientation of one workpiece from the input image using the search model;
Based on the position and orientation of the one workpiece specified by the three-dimensional search unit and the gripping position specified by the gripping position specifying unit for the work model corresponding to the one workpiece, the one a three-dimensional pick determination unit that determines whether or not the workpiece can be gripped by the end effector model;
with
The three-dimensional search unit searches for the position and position of the one workpiece from the input image based on the feature points on the contour representing the outline of the workpiece and the feature points on the surface representing the surface shape included in the input image. configured to perform a three-dimensional search to identify poses ,
The display unit displays the end effector model when gripping the one workpiece on the input image, and the three-dimensional pick determination unit determines that the one workpiece cannot be gripped by the end effector model. an image processing device capable of displaying a cause of incapability of gripping the one workpiece when Image processing for controlling a robot that performs a picking operation by measuring the three-dimensional shape of multiple workpieces stacked in a work space with a sensor unit and gripping them with an end effector provided at the tip of the arm of the robot with a robot controller. a device,
an input image acquisition unit that acquires an input image representing the three-dimensional shape of a plurality of works from a sensor unit that measures the three-dimensional shape of the work placed in the work space;
a search model registration unit for registering a search model used for a three-dimensional search for specifying the position and orientation of the workpiece from the input image;
an end effector model registration unit for registering an end effector model that virtually represents a three-dimensional shape of an end effector controlled by the robot controller;
a display unit that displays a work model that virtually represents the three-dimensional shape of the work in a three-dimensional manner in a virtual three-dimensional space;
a gripping position specifying unit that specifies a gripping position at which the end effector model grips the work model displayed on the display unit;
a three-dimensional search unit that performs a three-dimensional search for specifying the position and orientation of one workpiece from the input image using the search model;
Based on the position and orientation of the one workpiece specified by the three-dimensional search unit and the gripping position specified by the gripping position specifying unit for the work model corresponding to the one workpiece, the one a three-dimensional pick determination unit that determines whether or not the workpiece can be gripped by the end effector model;
with
The display unit displays the end effector model when gripping the one workpiece on the input image, and the three-dimensional pick determination unit determines that the one workpiece cannot be gripped by the end effector model. is configured so as to be able to display the reason why the one work cannot be gripped,
The display unit
a three-dimensional search button for accepting selection of the three-dimensional search;
a three-dimensional pick button for accepting selection of three-dimensional pick determination by the three-dimensional pick determination unit ;
It is possible to display a function selection screen that is arranged side by side,
When the selection of the three-dimensional search button is accepted from the function selection screen,
an icon for displaying a screen for registering the search model on the display unit;
an icon for displaying on the display unit a screen for setting a search area for performing the three-dimensional search;
an icon for displaying on the display unit a screen for setting conditions for executing the three-dimensional search;
are arranged in this order and displayed on the display unit,
When the selection of the three-dimensional pick button is accepted from the function selection screen,
Convert the position and orientation calculated in the coordinate system of the vision space, which is a virtual three-dimensional space displayed on the display unit, to the position and orientation of the coordinate system in the robot space in which the robot controller operates the end effector. an icon for displaying a screen for selecting calibration information to be displayed on the display unit;
an icon for displaying a screen for registering the grip position on the display unit;
an icon for displaying on the display unit a screen for setting conditions for detecting the gripping position;
are arranged in this order and displayed on the display section. The image processing device according to any one of claims 1 to 4 , further comprising:
an input interface that acquires the input image from the sensor unit;
a display interface that outputs display data for displaying the input image and the end effector model on the display;
Information necessary for controlling the robot connected to the robot controller so that the robot controller picks the one work when the three-dimensional pick determination unit determines that the one work can be gripped. a robot interface that outputs
An image processing device comprising: The image processing device according to claim 1 ,
The image processing apparatus configured such that the interference determination unit receives a setting of an extension amount of the end effector model when performing interference determination, and performs interference determination based on the end effector model and the extension amount. . The image processing device according to claim 3 , further comprising:
An evaluation index calculation unit that calculates an evaluation index for the result of the three-dimensional search,
The image processing apparatus, wherein the evaluation index calculation unit is configured to calculate the evaluation index according to the existence ratio of feature points corresponding to the search model registered by the search model registration unit. 8. The image processing apparatus according to claim 7, further comprising a gripping priority determination unit that determines the priority of workpieces to be gripped by the end effector,
The image processing device, wherein the gripping priority determining unit is configured to determine the priority based on the evaluation index and the position of the workpiece in the Z direction.

Images