WO2022208963A1

WO2022208963A1 - Calibration device for controlling robot

Info

Publication number: WO2022208963A1
Application number: PCT/JP2021/040394
Authority: WO
Inventors: 毅北村; 聡笹谷; 大輔浅井; 信博知原
Original assignee: 株式会社日立製作所
Priority date: 2021-03-29
Filing date: 2021-11-02
Publication date: 2022-10-06
Also published as: JP2022152845A; JP7437343B2

Abstract

This calibration device is provided with: an imaging unit that captures an image from a predetermined position; a marker attached to a robot; a posture collection generating unit that generates a posture collection including a plurality of postures of the robot; a non-blocking-posture extracting unit that extracts from the posture collection a plurality of postures that do not block the marker; and a calibration executing unit that estimates coordinate transformation parameters for transforming the coordinate system of the robot and the coordinate system of the imaging unit on the basis of the extracted plurality of postures.

Description

Calibration device for robot control

The present invention relates to a calibration device for robot control.

In order to solve the labor shortage in recent years, autonomous robots are being used to perform various types of work, such as picking work in the field of logistics and assembly work in the industrial field. There is a growing need for automation. To realize this automation, in order for the robot to perform appropriate autonomous work, an imaging unit is required to recognize the surrounding environment such as the workpiece with high accuracy.

In order for the robot to recognize the surrounding environment based on the image captured by the imaging unit and work autonomously according to this recognition result, calibration is required to determine the relative placement (position, posture) of the imaging unit with respect to the robot. implementation is necessary. For this calibration, for example, a target marker with a known shape (hereinafter also referred to as a "marker") is attached to the robot, the robot is photographed in various postures by an imaging unit, and the photographed image and the posture of the robot are captured. There is a method of estimating the relative position and orientation between the robot and the imaging unit using a plurality of sets of data in which the relationship between .

However, with the above calibration method, there is a concern that the number of man-hours will increase. For example, in order to teach a plurality of robot postures (hereinafter also referred to as “posture set”) required for calibration, it is necessary for an expert to make trial and error according to the surrounding environment where the robot system is installed. be. Therefore, many man-hours are required for calibration. In addition, when a robot is caused to perform irregular work such as picking work, there is a possibility that the arrangement of the imaging unit will be changed according to the articles existing in the surrounding environment. Then, when actually changing the arrangement of the imaging units, it is necessary to teach the set of postures of the robot each time. Therefore, the man-hours required for calibration (hereinafter also referred to as "calibration man-hours") may become enormous. Against this background, in recent years, a mechanism for reducing the number of calibration man-hours or prototyping has attracted more attention.

Regarding calibration for robot control, for example, Japanese Patent Laid-Open No. 2002-200000 describes a solution for "easily performing calibration", which states that "a robot control system, based on an image captured by an imaging unit, A measurement unit that measures the three-dimensional coordinates of an arbitrary object existing within the field of view of the robot, and the robot according to a pre-calculated correspondence relationship between the measured three-dimensional coordinates and the position and orientation of the action unit of the robot. A command generation unit that generates a command for positioning the action part, a calibration execution part that executes calibration for calculating the correspondence, and a reference object associated with the action part of the robot in the calibration. and a setting reception unit that receives the setting of the calibration area, which is the area to be calibrated."

That is, in the technique described in Patent Document 1, a space (calibration area) in which the marker is moved during calibration is defined based on information about the robot captured by the imaging unit, and the marker is automatically arranged in that space. By generating and teaching a set of poses, the number of man-hours required for calibration is reduced. According to the technique described in Patent Document 1, since the marker is moved within the range of the calibration area whose setting is received by the setting receiving unit, the number of man-hours required for teaching the posture set of the robot can be reduced and the calibration can be performed. can be executed.

JP 2019-217571 A

However, the technique described in Patent Literature 1 does not take into account the occlusion of markers (hereinafter also referred to as "occlusion") that may occur when the robot is changed to various postures. Therefore, there are the following problems.
In general, when there is something other than the robot around the robot or around the imaging unit, or when a part other than the marker among the parts attached to the robot is interposed between the marker and the imaging unit, Markers may be obscured by viewing. In addition, it takes an enormous amount of man-hours to define the movement space of the marker so that the marker is not blocked and to teach the set of postures of the robot.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a robot control calibration apparatus that teaches a set of robot postures in consideration of the shielding of markers and that can reduce the man-hours required for teaching the set of robot postures. .

In order to solve the above problems, for example, the configurations described in the claims are adopted.
The present application includes a plurality of means for solving the above problems. One of them is an imaging unit that takes an image from a predetermined position, and a marker that is attached to the robot and displaces according to the movement of the robot. a posture set generation unit for generating a posture set including a plurality of postures of the robot for enabling the imaging unit to capture the marker; An unshielded posture extraction unit that extracts a plurality of postures, and estimates coordinate transformation parameters for transforming the coordinate system of the robot and the coordinate system of the imaging unit based on the plurality of postures extracted by the unshielded posture extraction unit. and a calibration execution unit.

According to the present invention, it is possible to teach a set of robot postures in consideration of the shielding of markers, and to reduce the number of man-hours required for teaching the set of robot postures.
Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

It is a figure which shows the structural example of the calibration apparatus for robot control which concerns on 1st Embodiment. It is a block diagram which shows the structural example of the control apparatus in this embodiment. FIG. 2 is a diagram showing a coordinate system and coordinate transformation of a physical device in this embodiment; FIG. 11 is a functional block diagram of a posture set generator; It is a figure which shows the list of the parameter which a parameter acquisition part acquires. 7 is a flow chart showing a procedure for obtaining parameters by a parameter obtaining unit; 4 is a flow chart showing a procedure for generating and saving a posture set; FIG. 5 is a diagram for explaining a method of generating a teaching space by a teaching space generation unit; FIG. 5 is a diagram illustrating a method of dividing a teaching space by a teaching space generation unit; FIG. 5 is a diagram for explaining a teaching position generation method by a teaching position generation unit; FIG. 10 is a diagram showing how a marker looks when the marker is placed at the teaching position in each small space in the initial posture; FIG. 10 is a diagram showing a teaching posture of a marker generated by a teaching posture generation unit; It is a figure which shows an example of the simulator built by the simulator building part. 9 is a flow chart showing a processing procedure of a non-physical interference attitude extraction unit; FIG. 4 is a top view of the posture of the robot when it actually performs picking work. FIG. 3 is a side view of the posture of the robot when it actually performs picking work. 9 is a flow chart showing a processing procedure of a non-shielding posture extraction unit; FIG. 10 is a schematic diagram showing an operation example of a non-shielding posture extraction unit; 4 is a functional block diagram of a posture set evaluation unit; FIG. 5 is a flow chart showing a processing procedure of a calibration execution unit; FIG. 10 is a diagram showing a configuration example of a calibration device for robot control according to a second embodiment;

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In this specification and the drawings, elements having substantially the same function or configuration are denoted by the same reference numerals, and overlapping descriptions are omitted.

<First Embodiment>
FIG. 1 is a diagram showing a configuration example of a calibration device for robot control according to the first embodiment. In this embodiment, as an example of the work performed by the robot, picking work for moving a work to a predetermined position is assumed. In addition, in the present embodiment, as an example of a robot to be controlled, a robot arm that is an articulated robot (hereinafter also simply referred to as "robot") is assumed. Further, in this embodiment, as an example of the marker attached to the robot, a calibration board is assumed in which a pattern with known dimensions is printed on a jig on a plane. Also, in this embodiment, a monocular camera is assumed as an example of the imaging unit. In addition, in the present embodiment, a wall and a conveyor are assumed as examples of environmental installation objects installed around the robot and around the imaging unit. However, the content assumed in this embodiment is merely an example, and the configuration of the calibration device for robot control is not limited to the example assumed in this embodiment.

As shown in FIG. 1, the calibration device 2 includes a control device 1, a real device 2A, and a calibration control device 2B.
The control device 1 centrally controls the entire calibration device 2 . Further, the control device 1 controls the motion of the robot 30 and the motion of the imaging unit 32 in the physical device 2A. Although the control device 1 and the calibration control device 2B are shown separately in FIG. 1, the functions of the calibration control device 2B can be realized by the computer hardware resources of the control device 1. .

The real device 2A has a robot 30 configured by a robot arm, a marker 31 attached to the robot 30, and an imaging unit 32 that measures the work space of the robot 30 and its surroundings. Also, the physical device 2A has a wall 33A and a conveyor 33B as an example of the environmental installation objects 33 installed around the robot 30 and around the imaging unit 32 .

The calibration control device 2B includes a prior knowledge 21, a generation parameter 22, a posture set generation unit 23 that generates a posture set of the robot 30 using the prior knowledge 21 and the generation parameter 22, and a three-dimensional image using the prior knowledge 21. The robot 30 and the marker 31 are selected from the pose sets generated by the simulator constructing unit 24 for constructing a simulator (hereinafter also simply referred to as "simulator") and the pose set generating unit 23, in which part of the physical device 2A. A non-physical interference posture extraction unit 25 that extracts a plurality of postures (posture set) that do not cause physical interference, and a plurality of postures ( A posture set evaluation unit 27 that evaluates the posture set extracted by the non-shielded posture extraction unit 26 from the viewpoint of calibration accuracy. and a calibration execution unit 29 that executes calibration when the posture set evaluation unit 27 evaluates high. The posture set of the robot 30 is a set of multiple postures (posture data) of the robot 30 .

The prior knowledge 21 includes design information including the shape of the physical device 2A, information capable of specifying the space in which the robot 30 works (hereinafter also referred to as "work space"), an evaluation value of the imaging unit 32, It is a database having parameter information such as threshold values used in each functional unit. The design information of the physical device 2A is information including the shape, specifications and layout of the three-dimensional model of the physical device 2A. The evaluation value of the image pickup unit 32 is an evaluation value obtained by an inspection performed when the image pickup unit 32 is received or shipped, and represents the actual performance of the image pickup unit 32 with respect to product specifications and the degree of image distortion. The generation parameter 22 is a database having parameters that determine the number and density of poses included in the pose set of the robot 30 .

The posture set generation unit 23 receives the prior knowledge 21 and the generation parameters 22 as input, generates a posture set of the robot 30 , and stores the generated posture set in the control device 1 . The posture of the robot 30 included in the posture set generated by the posture set generation unit 23 is a posture that enables the imaging unit 32 to image (capture) the marker 31 . The simulator construction unit 24 uses the prior knowledge 21 to construct a three-dimensional simulator that enables virtual motion of the robot 30 and imaging using the imaging unit 32 . The non-physical interference posture extraction unit 25 reads the posture set generated by the posture set generation unit 23 from the control device 1, and moves the robot to each posture on the three-dimensional simulator constructed by the simulator construction unit 24, thereby A plurality of postures are extracted in which the robot 30 and the marker 31 do not physically interfere with either the robot 30 or the environmental installation object 33 . In addition, the non-physical interference posture extraction unit 25 stores the plurality of extracted postures in the control device 1 as a set of postures without physical interference. The non-shielding posture extraction unit 26 reads the posture set extracted by the non-physical interference posture extraction unit 25 from the control device 1, and moves the robot to each posture on the three-dimensional simulator constructed by the simulator construction unit 24. A plurality of postures in which the marker 31 is not blocked between the imaging unit 32 and the marker 31 are extracted. In addition, the non-shielding posture extraction unit 26 stores the plurality of extracted postures in the control device 1 as a posture set. Note that in the present embodiment, the storage location of the posture set generated by the posture set generation unit 23 and the storage destination of the posture sets extracted by the non-physical interference posture extraction unit 25 and the non-shielding posture extraction unit 26 are set to the control device 1. However, the storage destination of the posture set is not limited to the control device 1, and may be, for example, a storage unit provided in the calibration control device 2B.

The posture set evaluation unit 27 reads the posture set extracted by the non-shielding posture extraction unit 26 from the control device 1, and determines whether or not the estimation accuracy of the calibration using the posture set is equal to or greater than a predetermined value set in advance. or not, and the posture set is evaluated based on this determination result. The parameter update unit 28 updates the generation parameter 22 when the posture set evaluation unit 27 cannot expect calibration with a higher accuracy than a predetermined value. The calibration executing unit 29 controls the robot 30 and the imaging unit 32 based on the posture set (a plurality of postures) determined by the posture set evaluating unit 27 to be expected to perform calibration with higher accuracy than a predetermined level. Coordinate transformation parameters for transforming the coordinate system of 30 and the coordinate system of the imaging unit 32 are estimated.

Hereinafter, the control device 1, the real device 2A, the prior knowledge 21, the pose set generation unit 23, the simulator construction unit 24, the non-physical interference pose extraction unit 25, the non-shielding pose extraction unit 26, the pose set evaluation unit 27, and the parameter update unit 28, the calibration execution unit 29 will be described in detail.

(control device 1)
FIG. 2 is a block diagram showing a configuration example of the control device 1 in this embodiment.
As shown in FIG. 2, the control device 1 includes a CPU 11, a bus 12 for transmitting commands from the CPU 11, a ROM 13, a RAM 14, and a CPU 11 as computer hardware resources for overall control of the calibration device 2. A storage device 15, a network I/F (I/F is an abbreviation for interface, hereinafter the same) 16, an imaging I/F 17 for connecting the imaging unit 32, and a screen display I/F 18 for screen output. , and an input I/F 19 for external input. That is, the control device 1 can be configured by a general computer device.

The storage device 15 stores a program 15A for executing each function, an OS (operating system) 15B, a three-dimensional model 15C, and parameters 15D such as a database. A robot controller 16A is connected to the network I/F 16 . The robot control device 16A is a device that controls and operates the robot 30 .

(Real device 2A)
In this embodiment, the robot 30 is a picking robot that performs picking work. Generally, a picking robot is composed of an articulated robot. However, the robot 30 may be of any type as long as it has a plurality of joints. In addition, the robot 30 is installed on a single-axis slider, and by increasing the degree of freedom of movement of the robot 30 with this single-axis slider, it is possible to perform work in a wide space.
As the marker 31, a calibration board on which a pattern with known dimensions is printed on a planar jig is used. When such a marker 31 is used, the position of the marker 31 in the three-dimensional space can be identified by analyzing the data measured by the imaging section 32 . However, if the three-dimensional position of the marker 31 can be identified from the data measured by the imaging unit 32, for example, a spherical jig or a jig with a special shape can be used as the marker 31.
The marker 31 is attached to the arm tip of the robot 30 which is a robot arm. However, the attachment position of the marker 31 is not limited to the tip of the arm, and may be on the robot arm as long as the position satisfies the condition that the marker 31 is displaced according to the motion of the robot 30 .

Also, in this embodiment, a monocular camera is used as the imaging unit 32 . However, the imaging unit 32 is not limited to a monocular camera, and may be configured by, for example, a ToF (Time of Flight) camera, a stereo camera, or the like. In other words, the data obtained by the imaging unit 32 is data such as images and point clouds that can specify the three-dimensional position of the marker 31 . The imaging unit 32 photographs the work space of the robot 30 and the like from a predetermined position. The mounting position of the imaging unit 32 can be set at any location, such as the ceiling or wall of the building where the robot 30 works.

Here, the coordinate system and coordinate transformation of the physical device 2A will be described.
FIG. 3 is a diagram showing the coordinate system and coordinate conversion of the physical device 2A in this embodiment.
The coordinate system of the physical device 2A includes a base coordinate system C1 whose origin is the base of the arm of the robot 30, an arm coordinate system C2 whose origin is the tip of the arm of the robot 30, and a marker coordinate system whose origin is the center of the marker 31. C3 and a camera coordinate system C4 whose origin is the optical center of the imaging unit 32 . Each coordinate system is a three-dimensional coordinate system.

The coordinate transformation of the physical device 2A includes a base/camera transformation matrix M1 representing coordinate transformation between the base coordinate system C1 and the camera coordinate system C4, and a marker/camera transformation matrix representing coordinate transformation between the marker coordinate system C3 and the camera coordinate system C4. A transformation matrix M2, an arm/base transformation matrix M3 representing coordinate transformation between the arm coordinate system C2 and the base coordinate system C1, and an arm/marker transformation matrix M4 representing coordinate transformation between the arm coordinate system C2 and the marker coordinate system C3. There is Of these, the base coordinate system C<b>1 corresponds to the coordinate system of the robot 30 , and the camera coordinate system C<b>4 corresponds to the coordinate system of the imaging unit 32 . The base/camera transformation matrix M1 corresponds to a coordinate transformation parameter for transforming the coordinate system of the robot 30 and the coordinate system of the imaging unit 32 . As an example of coordinate transformation, when the rotation matrix is Rca and the translation matrix is tca in the base/camera transformation matrix M1, the coordinates C (Xc, Yc, Zc) in the camera coordinate system C4 can be expressed by the following formula: It can be transformed into a point P(Xr, Yr, Zr) in the base coordinate system C1 as shown.

Next, a method of acquiring the coordinate transformation parameters (M1, M2, M3, M4) of the physical device 2A will be described.
The base-camera conversion matrix M1 can be obtained as one of the design values in the physical device 2A. However, when the robot 30 is operated by applying the base/camera transformation matrix M1 obtained as the design value, an error may occur in the position of the robot 30 . One possible cause of such an error is, for example, misalignment of the mounting position of the imaging unit 32 in the physical device 2A. Specifically, even if the mounting position of the imaging unit 32 is designed so that the optical axis (z-axis) of the imaging unit 32 is parallel to the vertical direction, a slight positional deviation may occur during actual mounting. . Calibration is performed to accurately obtain the base/camera transformation matrix M so that such a positional deviation does not cause an error in the position of the robot 30 .

The design value of the base/camera transformation matrix M1 is included in the prior knowledge 21. Therefore, the design value of the base-camera conversion matrix M1 can be obtained from the prior knowledge 21 . Also, by executing the calibration, it is possible to obtain the actual values of the base/camera transformation matrix M1 that reflects the positional deviation and the like. Calibration in this embodiment means estimating the base/camera conversion matrix M1 from a plurality of sets of data of the marker/camera conversion matrix M2 and the arm/base conversion matrix M3 when the robot 30 is moved to a certain posture. Say things. Estimation of the base-to-camera transformation matrix M1 is performed computationally.

The marker-camera conversion matrix M2 can be obtained by analyzing an image obtained by capturing the marker 31 by the imaging unit 32 . The arm-base conversion matrix M3 can be obtained by calculation from the encoder values of the robot 30 . The joint portion of the robot 30 is provided with a motor (not shown) for driving the joint and an encoder for detecting the rotation angle of the motor, and the encoder value means the output value of the encoder. The arm-marker conversion matrix M4 is unknown, or a design value can be obtained from the prior knowledge 21 . In this embodiment, since the design value of the arm-marker transformation matrix M4 is included in the prior knowledge 21, the design value of the arm-marker transformation matrix M4 can be obtained from this prior knowledge 21. FIG. In this embodiment, the origin of each coordinate system, the orientation of the coordinate system, and the coordinate conversion are set as described above, but the present invention is not limited to this example.

(Prior knowledge 21)
The prior knowledge 21 is a database having parameters (1) to (11) below.
(1) A design value of the base-camera transformation matrix M1.
(2) A design value of the arm-marker transformation matrix M4.
(3) A three-dimensional model including shape information of the robot 30;
(4) Specification data that defines the upper limit of the joint angles of the robot 30 and the like.
(5) shape data including the type and size of the marker 31;
(6) Specification data including information such as camera parameters of the imaging unit 32, optical blur, resolution, and angle of view.
(7) A distortion evaluation value of the imaging unit 32 .
(8) Information that defines the work space of the robot 30;
(9) Design information of the real device 2A.
(10) Information that defines the shape and arrangement of the environmental installation object 33 .
(11) Threshold used in each functional unit.
In this embodiment, the function using the above prior knowledge 21 will be described, but the function may be implemented using only part of it.

(Posture set generator 23)
The posture set generator 23 generates a posture set of the robot 30 based on the parameters obtained from the prior knowledge 21 and the parameters obtained from the generated parameters 22 . Also, the posture set generator 23 stores the generated posture set in the control device 1 .

FIG. 4 is a functional block diagram of the posture set generator 23. As shown in FIG.
As shown in FIG. 4, the posture set generation unit 23 includes a parameter acquisition unit 231, a teaching space generation unit 232, a teaching position generation unit 233, a teaching posture generation unit 234, a coordinate system conversion unit 235, and a posture set generation unit 231. A storage unit 236 is provided.

The parameter acquisition unit 231 acquires parameters necessary for generating a set of postures from the prior knowledge 21 and the generation parameters 22 . The teaching space generation unit 232 determines a space in which the marker 31 is placed during calibration, and divides this space into a plurality of small spaces. Also, the teaching space generator 232 assigns an index to each small space. The teaching position generation unit 233 generates positions at which the markers 31 are arranged in each small space, that is, teaching positions, based on the camera coordinate system C4. The teaching posture generation unit 234 generates the posture of the marker 31 at each teaching position generated by the teaching position generation unit 233, that is, the teaching posture. The coordinate system transformation unit 235 transforms the set of poses of the marker 31 with respect to the camera coordinate system C4 to the robot with respect to the base coordinate system C1 based on the design values of the base/camera transformation matrix M1 included in the prior knowledge 21. Transform into a set of 30 poses. The posture set storage unit 236 stores the posture set generated by the teaching posture generation unit 234 and the posture set transformed by the coordinate system transformation unit 235 in the control device 1 . The configuration of each part of the posture set generator 23 will be described in more detail below.

FIG. 5 is a diagram showing a list of parameters acquired by the parameter acquisition unit 231. As shown in FIG.
As shown in FIG. 5, the parameters acquired by the parameter acquisition unit 231 include the design values (rotation matrix Rca, translation matrix tca) of the base-camera conversion matrix M1, the measurement range of the imaging unit 32 (angle of view, resolution, optical blur), distortion information of the imaging unit 32 (distortion evaluation), workspace of the robot 30 (three-dimensional space information), shape of the marker 31 (board, sphere, size), resolution in each axis direction of the posture set (X1, Y1 , Z1).

FIG. 6 is a flowchart showing a parameter acquisition procedure by the parameter acquisition unit 231. As shown in FIG.
First, the parameter acquisition unit 231 acquires the design values of the base-camera conversion matrix M1 from the prior knowledge 21 (step S1). At this time, the parameter acquisition unit 231 acquires the rotation matrix Rca and the translation matrix tca as the design values of the base-camera conversion matrix M1.

Next, the parameter acquisition unit 231 acquires the measurement range of the imaging unit 32 and the distortion information of the imaging unit 32 from the prior knowledge 21 (step S2). At this time, the parameter acquisition unit 231 acquires the angle of view, resolution, and optical blur as the measurement range of the imaging unit 32 and acquires the distortion evaluation value as the distortion information of the imaging unit 32 .

Next, the parameter acquisition unit 231 acquires information defining the workspace of the robot 30 from the prior knowledge 21 (step S3). At this time, the parameter acquisition unit 231 acquires three-dimensional space information indicating the work space as information defining the work space of the robot 30 .
Next, the parameter acquisition unit 231 acquires shape data of the marker 31 (step S4). At this time, the parameter acquisition unit 231 acquires the type (board, sphere) and size data of the marker 31 as the shape data of the marker 31 .
Next, the parameter acquisition unit 231 acquires the resolution (X1, Y1, Z1) in each axial direction of the posture set from the generation parameter 22 (step S5). Specifically, with reference to the camera coordinate system C4, the number of poses (X1, Y1, Z1) on each axis is acquired as a parameter that determines the resolution in each axis direction. In this case, the greater the number of postures to be created on each axis (the greater each value of X1, Y1, and Z1), the higher the resolution in each axis direction. In other words, the parameters X1, Y1, and Z1 correspond to parameters that determine the number and density of poses included in the pose set.

FIG. 7 is a flow chart showing a procedure for generating a posture set by the teaching space generation unit 232, the teaching position generation unit 233, the teaching posture generation unit 234, and the coordinate system conversion unit 235 and storing the posture set by the posture set storage unit 236. is.
As shown in FIG. 7, first, the teaching space generator 232 generates a teaching space in which the markers 31 are arranged (step S6). At this time, the teaching space generated by the teaching space generation unit 232 is a three-dimensional space based on the camera coordinate system C4.

Next, the teaching space generator 232 divides the previously generated teaching space into a plurality of small spaces, and assigns an index to each small space (step S7). At this time, the teaching space generator 232 divides the teaching space into small spaces of X1×Y1×Z1. For example, if X1=3, Y1=3, and Z1=3, the teaching space generator 232 divides the teaching space into three in each axial direction, thereby dividing the teaching space into a total of 27 small spaces. do. Also, the teaching space generator 232 assigns an index (X, Y, Z) to each small space.

Next, the teaching position generating section 233 sets the position where the marker 31 is arranged in each small space as a teaching position (step S8). As a result, the teaching positions of the marker 31 are generated for the number of small spaces.
Next, the teaching posture generation unit 234 initially sets the teaching posture of the marker 31 at each teaching position so that the marker 31 faces the imaging unit 32 (step S9). In the following description, the initially set teaching orientation of the marker 31 will be referred to as the initial orientation.
Next, the teaching posture generation unit 234 generates the marker 31 from the initial posture by ±θ (threshold) degrees from 0 degrees in each axis direction of the camera coordinate system C4 from the initial posture, based on the index assigned to each small space, or randomly. By rotating the posture of , the teaching posture of the marker 31 is generated (step S10). As a result, the teaching postures of the marker 31 are generated for the number of small spaces.

Next, the coordinate system conversion unit 235 converts the coordinates of the taught posture of the marker 31 based on the camera coordinate system C4 generated by the taught posture generation unit 234 in step S10 using the design values of the base/camera transformation matrix M1. are converted into the coordinates of the base coordinate system C1 (step S11). As a result, the teaching posture of the marker 31 based on the camera coordinate system C4 is converted into the teaching posture of the robot 30 based on the base coordinate system C1. Also, the teaching postures of the robot 30 are generated for the number of small spaces.
Next, the posture set storage unit 236 saves the posture set (a plurality of taught postures) of the marker 31 generated by the taught posture generation unit 234 in step S10 and the A set of postures (a plurality of teaching postures) of the robot 30 is saved in the control device 1 (step S12). As a result, the controller 1 stores a set of orientations of the marker 31 based on the camera coordinate system C4 and a set of orientations of the robot 30 based on the base coordinate system C1.

FIG. 8 is a diagram for explaining a teaching space generation method by the teaching space generation unit 232. As shown in FIG. The illustrated generation method is applied to step S6 in FIG. 7 above.
First, the teaching space is defined as a common area between the space in which the robot 30 works and the space in which the imaging unit 32 can accurately shoot (measurable) overlap.
In FIG. 8, the work space of the robot 30 is determined based on the camera coordinate system C4. As a method of determining the work space, for example, when the work performed by the robot 30 is a picking work, the space in which the marker 31 is positioned when the robot 30 performs the work of gripping a work (not shown) is determined in advance. be able to. The reason why the work space of the robot 30 is used as the teaching space T1 is as follows. When the robot 30 is moved, movement errors systematically occur due to manufacturing errors of the robot 30 or the like. However, the appearance of the movement error differs depending on the position in space. Therefore, by setting the posture taken by the robot 30 during calibration to be the same as the posture during work, it is possible to use data with the same movement error as during work for calibration.

The space that the imaging unit 32 can shoot is determined by the angle of view of the imaging unit 32, but the space that the imaging unit 32 can shoot with accuracy is limited to a narrower range than the shootable space. In FIG. 8, as a space in which the imaging unit 32 can shoot with high accuracy, the size of the space in the Z-axis direction in the camera coordinate system C4 is specified so that the value of the optical blur of the imaging unit 32 is equal to or less than the threshold. . In the example of FIG. 8, the spatial dimension in the Z-axis direction is specified within a range in which the optical blur value is 1 px or less. In addition, as a space that can be captured by the imaging unit 32 with high accuracy, the size of the space in the X-axis direction and the Y-axis direction in the camera coordinate system C4 is the image captured by the imaging unit 32 (hereinafter also referred to as “captured image”). ) is subjected to distortion correction, and the luminance difference between the image before distortion correction and the image after distortion correction is specified to be within a range of a threshold value or less. As a result, it is possible to control so that the marker 31 is placed in a space in which the captured image is less likely to be distorted. Therefore, it is possible to reduce errors in calibration, which will be described later.

In the example of FIG. 8, the size of the space in the Z-axis direction in the camera coordinate system C4 is defined by the working space of the robot 30. In the example of FIG. This is because the working space of the robot 30 is narrower than the space in which the optical blur value is 1 px or less when the size of the space in the Z-axis direction is compared. On the other hand, the size of the space in the X-axis direction and the Y-axis direction in the camera coordinate system C4 is defined by the space in which the imaging unit 32 can accurately capture images. This is because, when the sizes of the spaces in the X-axis direction and the Y-axis direction are compared, a space in which the image capturing unit 32 can accurately capture images, that is, a space in which the difference in brightness based on the distortion evaluation is equal to or less than the threshold is the space in which the robot 30 is working. This is because it is narrower than the space.
The teaching space T1 is generated by the above method.

FIG. 9 is a diagram for explaining how the teaching space generation unit 232 divides the teaching space. The illustrated division method is applied to step S7 in FIG. 7 above.
When dividing the teaching space, parameters X1, Y1, and Z1 given as the resolution in each axial direction of the posture set are used. Specifically, with reference to the camera coordinate system C4, the teaching space T1 generated in step S6 is divided into X1 pieces in the x-axis direction, Y1 pieces in the y-axis direction, and Z1 pieces in the z-axis direction. split into In the example of FIG. 9, the teaching space T1 is divided with X1=3, Y1=3, and Z1=3. Therefore, the teaching space T1 is divided into a total of 27 small spaces. After dividing the teaching space T1 into a plurality of small spaces in this way, an index (X, Y, Z) is assigned to each small space. Indices (X, Y, Z) are assigned based on the coordinate axes. For example, the x-axis is assigned indices of X=1, X=2, and X=3 from the right side to the left side of FIG. Indexes of Y=2 and Y=3 are assigned, and indexes of Z=1, Z=2 and Z=3 are assigned to the z-axis from the upper side to the lower side of FIG. In such a case, indices (X, Y, Z)=(1, 1, 3) are given to the small space located in the lower right of FIG.

FIG. 10 is a diagram for explaining a teaching position generation method by the teaching position generation unit 233. As shown in FIG. The illustrated generation method is applied to step S8 in FIG. 7 above.
First, as an example, for a teaching position in a small space with indices (X, Y, Z) = (1, 1, 3), a small A position moved by half the height of the space, that is, the central position of the small space given the index (1, 1, 3) can be generated as the teaching position. This point is the same for small spaces to which other indexes are assigned.

Next, a method of generating a taught posture by the taught posture generation unit 234 will be described. The teaching posture is generated by the processing of steps S9 and S10 in FIG. 7 above.
First, the processing contents of step S<b>9 will be described.
FIG. 11 is a diagram showing how the marker looks when the marker is placed at the teaching position in each small space in the initial posture.
In FIG. 11, among the plurality of small spaces generated by the teaching space generation unit 232, indexes (X, Y, Z)=(1, 1, 1) (1, 2, 1) (1, 3, 1) For each small space (2,1,1) (2,2,1) (2,3,1) (3,1,1) (3,2,1) (3,3,1), teach position It shows how the marker 31 looks when the marker 31 is provisionally placed at the teaching position generated by the generation unit 233 and placed so as to face the imaging unit 32 . Under the initial orientation, the marker 31 is arranged in any small space such that the Z axis of the camera coordinate system C4 is oriented exactly opposite to the Z axis of the marker coordinate system C3. Therefore, the appearance of the marker 31 is the same in all small spaces.

Next, the processing of step S10 will be described.
The teaching posture generation unit 234 generates a teaching posture by rotating the posture of the marker 31 in each axis direction of the marker coordinate system C3 from the initial posture set in step S9.
FIG. 12 is a diagram showing the teaching posture of the marker 31 generated by the teaching posture generation unit 234. As shown in FIG. The indexes (X, Y, Z) of each small space in FIG. 12 correspond to the indexes (X, Y, Z) of each small space in FIG. As an example, the taught orientation of the marker 31 in the small space with index (3,3,1) is generated by rotating the marker 31 about the x-axis of the marker coordinate system C3 with respect to the initial orientation. The magnitude of the rotation angle of each axis is from 0 degrees to ±θ (threshold) degrees, and is determined based on an index, rule-based or randomly. In this embodiment, in the small spaces with indices (1,1,1) (1,3,1) (3,1,1), the same posture as the initial posture is used as the teaching posture, and in the small spaces with other indexes Then, the posture rotated around the axis is used as the teaching posture. In this way, the posture set, which is a plurality of postures generated by the teaching posture generation unit 234, includes postures in which the marker 31 is rotated in various directions, and postures in which the orientation of the marker 31 is the same and only translation is different. made up of combinations. As a result, compared to the case where the orientations of the markers 31 are changed randomly (the orientations of the markers 31 are different for all indices), the accuracy of calibration estimation can be improved.

Through the processing of each unit (231 to 236) of the posture set generation unit 23 described above, the posture set is automatically generated in the teaching space that satisfies both the work space of the robot 30 and the space in which the imaging unit 32 can accurately shoot. It is possible to generate

Note that in this embodiment, the rotation of the base/camera conversion matrix M1 is obtained in the form of the rotation matrix Rca, but a quaternion or the like may be used. Further, the shape of the teaching space T1 generated by the teaching space generating section 232 is not limited to the shape shown in FIG. For example, if the working space of the robot 30 is narrower than the space that can be captured by the imaging unit 32 with high accuracy, the teaching space T1 has a shape such as a rectangular parallelepiped. Further, when the distortion evaluation value of the imaging unit 32 used in the present embodiment cannot be obtained as a parameter for generating the teaching space T1, only the angle of view or only the angle of view without using the distortion evaluation value It may be determined to use a predetermined portion of the space. Further, as a method of determining the teaching space, the user may specify the shape and size of the teaching space in advance. Further, in the present embodiment, the teaching space T1 is divided based on the camera coordinate system C4 as a method of dividing the teaching space, but the teaching space T1 may be divided based on an arbitrary coordinate system. Also, the teaching position generated by the teaching position generation unit 233 may be set to an arbitrary position within the small space, for example, the vertex of the small space. In order to detect the marker 31 with high accuracy, the teaching posture generation unit 234 generates the teaching posture by setting the rotation angle of the marker 31 about each axis within a threshold value. No need to set. Further, in the present embodiment, the posture set storage unit 236 stores the posture set in the control device 1. However, the present invention is not limited to this, and may be stored in a memory (not shown). You can also load sets.

Next, the processing contents of the simulator construction unit 24 will be described in detail.
The simulator construction unit 24 has a function of virtually operating the robot 30 by constructing a simulator based on the prior knowledge 21 and generating an image (captured image) captured by the imaging unit 32 .
FIG. 13 is a diagram showing an example of a simulator constructed by the simulator construction unit 24. As shown in FIG.
In FIG. 13, information such as a three-dimensional model including design information of the real device 2A, specifications, layout, etc. is acquired from the prior knowledge 21, and based on the acquired information, the robot 30, the marker 31, the imaging unit 32, and the environment installed objects are detected. 33 are virtually placed on the simulator.
As for the robot 30, by using the joint information obtained as the length of each link and the specifications as shape information of the robot 30, which is information obtained from the prior knowledge 21, the robot 30 can be virtually operated. be. In addition, by operating the robot 30 virtually, it is possible to apply a trajectory planning algorithm used in an actual machine.

For the image capturing unit 32, by using specification data such as the angle of view, resolution, and camera parameters of the image capturing unit 32, which is information acquired from the prior knowledge 21, each image captured by the image capturing unit 32 on the simulator is It is possible to generate pixel values. Also, the color information of the robot 30 and the environment installation object 33 can be easily changed on the simulator. For example, it is possible to change the RGB values representing the colors of the robot 30 and the environment installation object 33 to (255, 0, 0), etc., and generate an image taken on a simulator.

By the processing of the simulator construction unit 24 described above, it is possible to virtually generate an image captured by the imaging unit 32 when the marker 31 is moved to the teaching position based on the posture set. Further, when a three-dimensional model of the robot 30 is obtained, the trajectory planning and control of the robot 30 can be virtually performed. It is possible to virtually generate images taken by

In the present embodiment, the layout information including the design values of the base-camera conversion matrix M1 is acquired from the prior knowledge 21, and the object is placed on the simulator based on the acquired layout information. do not have. For example, rough values indicating the size and arrangement of objects in the physical device 2A may be estimated from images captured by the imaging unit 32, and the objects may be arranged on the simulator based on the estimation results.

In addition, in this embodiment, when the physical device 2A is virtually placed in the simulator, the shape is reproduced using the three-dimensional model of the physical device 2A. The simulator may hold only the design value information for each coordinate system.

In addition, when the imaging unit 32 is a stereo camera or ToF camera capable of three-dimensional measurement, or when a three-dimensional shape can be obtained by machine learning an image from a monocular camera, the real device 2A acquired by the imaging unit 32 3D shape may be utilized to construct a simulator.

Next, the processing contents of the non-physical interference attitude extraction unit 25 will be described in detail.
FIG. 14 is a flow chart showing the processing procedure of the non-physical interference attitude extraction unit 25. As shown in FIG.
The non-physical interference posture extraction unit 25 extracts postures (non-physical interference postures) in which the units of the physical device 2A do not physically interfere with each other from the posture set generated by the posture set generation unit 23 . Also, the non-physical interference posture extraction unit 25 extracts the posture that does not cause physical interference by utilizing the simulator constructed by the simulator construction unit 24 . Physical interference means that the robot 30 and the marker 31 come into contact with the robot 30, the imaging unit 32, the environmental installation object 33, and the like in the physical device 2A.

First, the non-physical interference posture extraction unit 25 loads the posture set generated by the posture set generation unit 23 onto the memory of the control device 1 (step S13).
Next, the non-physical interference posture extraction unit 25 acquires the number N1 of postures included in the posture set (step S14).
Next, the non-physical interference posture extraction unit 25 sets the variable i to an initial value (i=1) (step S15).
Next, the non-physical interference posture extraction unit 25 instructs the trajectory plan to move the robot 30 to the i-th posture on the simulator (step S16).
Next, the non-physical interference attitude extraction unit 25 determines whether or not the trajectory planning instructed in step S16 has succeeded (step S17). Specifically, when the robot 30 is operated according to the instructed trajectory plan, the non-physical interference posture extraction unit 25 determines the trajectory if the robot 30 and the marker 31 do not physically interfere with any part in the real device 2A. It is determined that the plan has succeeded, and if there is physical interference, it is determined that the trajectory plan has failed.

Next, when the non-physical interference attitude extraction unit 25 determines that the trajectory planning is successful in step S17, it holds the i-th attitude as a success pattern in the memory, and then (step S18). Holds success/failure information for the index of the small space in which the posture of is generated (step S19). When holding the success information for the index of the small space, the success flag is held in association with the index of the small space in which the i-th posture is generated. Also, when holding failure information for the index of the small space, the index of the small space in which the i-th posture is generated is held in association with the failure flag.

On the other hand, if the non-physical interference attitude extraction unit 25 determines that the trajectory planning has failed in step S17, the processes in steps S18 and S19 are skipped. Here, when the trajectory planning fails, it corresponds to the case where it is determined that there is physical interference, and when the trajectory planning succeeds, it corresponds to the case where it is determined that there is no physical interference.

Next, the non-physical interference posture extraction unit 25 determines whether the value of the variable i satisfies i=N1 (step S20). Then, if the value of the variable i is less than N1, the non-physical interference attitude extraction unit 25 increments the value of the variable i by 1 (step S21), and then returns to the process of step S16. Further, when the value of the variable i reaches N1, the non-physical interference posture extraction unit 25 saves the posture set of the success patterns and the success/failure information for each index in the control device 1 (step S22). finish processing.
Through the processing of the non-physical interference posture extraction unit 25 described above, only postures in which the physical devices 2A do not physically interfere with each other can be extracted from the posture set.

Next, an example in which the non-physical interference attitude extraction unit 25 determines success or failure of the trajectory planning will be described.
FIG. 15 is a bird's-eye view of the posture when the robot 30 actually performs the picking work.
As shown in FIG. 15, when the marker 31 is moved to the destination 1, the robot 30 does not physically interfere with any part of the physical device 2A. Therefore, when the trajectory planning is instructed to move the marker 31 to the destination 1, the trajectory planning succeeds and the attitude of the success pattern is obtained. On the other hand, when moving the marker 31 to the destination 2, the robot 30 physically interferes with the environmental installation object 33 (wall 33A), and the trajectory planning fails.
Regarding the method of determining the presence or absence of physical interference, for example, for a trajectory candidate generated by a trajectory plan, when the robot 30 operates according to the trajectory, the three-dimensional position of the environmental installation object 33 and the like and the robot 30 can be determined by whether or not they overlap.

FIG. 16 is a side view of the posture when the robot 30 actually performs the picking work.
As shown in FIG. 16, when moving the marker 31 to the destination 3, the robot 30 does not physically interfere with any part of the physical device 2A. Therefore, when the trajectory planning is instructed to move the marker 31 to the destination 3, the trajectory planning succeeds and the attitude of the success pattern is obtained. On the other hand, when moving the marker 31 to the destination 4, the robot 30 physically interferes with the environment installation object 33 (conveyor 33B), and the trajectory planning fails.

In this embodiment, whether the trajectory planning is successful or not is determined by the presence or absence of physical interference between the robot 30 and the environmental installation object 33. However, the present invention is not limited to this. Alternatively, the user may add restrictions on the joint angles of the robot 30 to make the determination. Specifically, when moving the robot 30 to the destination, if the joint angle of the robot 30 is equal to or less than the upper limit, the trajectory planning is determined to be successful, and if the joint angle exceeds the upper limit, the trajectory planning is determined to be unsuccessful. good too.

Next, the processing contents of the non-shielding posture extraction unit 26 will be described in detail.
FIG. 17 is a flow chart showing the processing procedure of the non-shielding posture extraction unit 26. As shown in FIG.
The non-shielding posture extraction unit 26 selects from the posture set generated by the posture set generation unit 23 and extracted by the non-physical interference posture extraction unit 25 whether the marker 31 is shielded between the marker 31 and the imaging unit 32 . A posture that does not occur (non-shielding posture) is extracted. Moreover, the non-shielding posture extraction unit 26 extracts a posture in which the shielding of the marker 31 does not occur by utilizing the simulator constructed by the simulator construction unit 24 .

First, the non-shielding posture extraction unit 26 loads the posture set generated by the posture set generation unit 23 and extracted by the non-physical interference posture extraction unit 25 onto the memory of the control device 1 (step S23).
Next, the non-shielding posture extraction unit 26 acquires the number N2 of postures included in the posture set (step S24).
Next, the non-shielding posture extraction unit 26 sets the variable i to an initial value (i=1) (step S25).
Next, the non-shielding posture extraction unit 26 instructs the trajectory plan to move the robot 30 to the i-th posture on the simulator (step S26).
Next, the non-shielding posture extraction unit 26 generates a virtual captured image by the imaging unit 32 on the simulator (step S27).
That is, the non-shielding posture extraction unit 26 virtually generates a photographed image obtained by the imaging unit 32 when the robot 30 is moved to the i-th posture.

Next, the non-shielding posture extraction unit 26 estimates the marker-camera conversion matrix M2 by analyzing the generated captured image (step S28). As a method of analyzing the photographed image, for example, when a pattern of a plurality of black circles with known sizes and positions is printed on the marker 31, the known sizes and positions of the black circles and the above-mentioned virtual photographed image are analyzed. The marker-camera transformation matrix M2 can be estimated by associating the sizes and positions of the black circles in . Estimating the marker-camera transformation matrix M2 substantially means estimating the position and orientation of the marker 31, that is, the three-dimensional position of the marker 31 in the three-dimensional camera coordinate system C4.

Next, the non-shielding posture extraction unit 26 determines whether or not the estimation of the marker-camera conversion matrix M2 has succeeded (step S29). Then, when the estimation of the marker-camera conversion matrix M2 is successful, the non-shielding posture extraction unit 26 calculates the reliability of the estimation (step S30), and then determines whether or not the reliability is equal to or greater than the threshold. (step S31). Further, when the calculated reliability of the estimation is equal to or higher than the threshold, the non-shielding posture extraction unit 26 stores the i-th posture as a successful pattern in the memory (step S32), and then generates the i-th posture. Success/failure information for the index of the created small space is retained (step S33). When holding the success information for the index of the small space, the success flag is held in association with the index of the small space in which the i-th posture is generated. Also, when holding failure information for the index of the small space, the index of the small space in which the i-th posture is generated is held in association with the failure flag.

On the other hand, if the non-shielding posture extraction unit 26 determines in step S29 that the estimation of the marker-camera transformation matrix M2 has failed, it skips the processing in steps S30 to S34, and in step S31 the reliability is set to the threshold value. If it is determined that the above is not the case, the processing of steps S32 and S33 is skipped. Here, when the estimation of the marker-camera conversion matrix M2 fails, or when the reliability of the estimation is not equal to or greater than the threshold, it corresponds to the case where it is determined that the marker 31 is shielded. On the other hand, when the marker-camera conversion matrix M2 is successfully estimated, or when the reliability of the estimation is equal to or higher than the threshold, it corresponds to the case where it is determined that the marker 31 is not shielded.

Next, the non-shielding posture extraction unit 26 determines whether the value of the variable i satisfies i=N2 (step S34). Then, when the value of the variable i is less than N2, the non-shielding posture extraction unit 26 increments the value of the variable i by 1 (step S35), and then returns to the process of step S26. Further, when the value of the variable i reaches N2, the unshielded posture extraction unit 26 saves the posture set of the success pattern and the success/failure information for each index in the memory of the control device 1 (step S36 ), ending the series of processes.
Through the processing of the non-shielded posture extraction unit 26 described above, only postures in which the marker 31 is not shielded between the marker 31 and the imaging unit 32 can be extracted from the posture set.

Subsequently, the method of calculating the reliability (contents of processing in step S30) and the evaluation method (contents of processing in step S31) by the non-shielding posture extraction unit 26 will be described.
First, in step S28, there are cases where the marker-camera conversion matrix M2 can be estimated even when part of the marker 31 is hidden by shielding. However, in this case, the accuracy of estimating the marker-camera conversion matrix M2 may be lower than in the case where all the markers 31 appear in the captured image. It is not desirable to use an orientation with low estimation accuracy of the marker-camera transformation matrix M2 for calibration.

Therefore, the non-shielding posture extraction unit 26 uses the fact that the relative posture between the marker 31 and the imaging unit 32 in each teaching posture is known on the simulator. The ratio between the area of the marker 31 at that time and the area of the marker 31 actually appearing on the captured image of the imaging unit 32 on the simulator is calculated as the reliability of the estimation. In addition to this, for example, the ratio between the number of feature points on the marker 31 obtained by analyzing the captured image and the total number of feature points is calculated as the reliability. Then, the non-shielding posture extraction unit 26 determines whether or not the reliability calculated as described above is higher than a threshold determined in advance according to the type of the marker 31 .

FIG. 18 is a schematic diagram showing an operation example of the non-shielding posture extraction unit 26. As shown in FIG.
A portion of the posture set loaded onto the memory of the control device 1 in step S23 is shown within the frame line indicated by the dashed line on the left side of FIG. 18 shows part of the set of poses after occlusion determination, extracted by the non-shielding pose extracting unit 26, inside a frame line indicated by a dashed dotted line on the right side of FIG. In this example, the environmental installation object 33 is arranged so as to cover the upper, lower, and left sides of the image captured by the imaging unit 32 on the simulator.

First, in the posture shown in the captured image I1 of FIG. 18, the robot 30 partially shields the marker 31, and the estimation of the position and posture of the marker 31 may fail. If it were possible to estimate the position and orientation of the marker 31 from a part of the marker 31 appearing in the captured image I1, the unshielded orientation extraction unit 26 would capture the marker 31 without shielding, as described above. A ratio between the area of the marker 31 when it appears in the image and the area of the marker 31 actually appearing on the captured image of the imaging unit 32 on the simulator is calculated as the reliability of estimation. In calculating this reliability, the positions and orientations of the camera coordinate system C4 and the markers 31 in a certain posture of the robot 30 are known by the posture set generation unit 23 . Therefore, by using information on the position and orientation of the marker 31 and information on the angle of view, resolution, camera parameters, etc. of the imaging unit 32, pixels of the marker 31 on the image when the marker 31 appears in the image without obstruction. A set of locations (hereinafter referred to as a "pixel set") can be specified.

Subsequently, the non-shielding attitude extraction unit 26 changes the color information of the robot 30 and the environmental installation object 33 in the simulator to a color not included in the marker 31, for example, a color with an RGB value of (255, 0, 0). , the posture of the robot 30 is changed to virtually generate an image captured by the imaging unit 32 . The position of the marker 31 in the captured image matches the set of pixels, and the color of the blocked pixels is (255, 0, 0). Therefore, the non-shielding posture extraction unit 26 calculates the ratio of pixels that are not of the color (255, 0, 0) in the pixel set as the reliability of the estimation. Therefore, in the case of the photographed image I1, since half of the marker 31 is blocked by the robot 30, the estimation reliability is calculated to be approximately 50%. Moreover, when the threshold value to be compared with the reliability is set to 90% in advance, the reliability in the photographed image I1 is less than the threshold.

On the other hand, in the posture shown in the photographed image I2 of FIG. 18, the marker 31 is not blocked and the entire marker 31 is shown in the image. Therefore, in the case of the photographed image I2, the position and orientation of the marker 31 can be estimated, and the reliability of the estimation is higher than the threshold, so this is a successful pattern.
18, the marker 31 is placed within the angle of view of the imaging unit 32, but the marker 31 is partly blocked by the environmental installation object 33. In the posture shown in the photographed image I3 of FIG. Therefore, estimation of the position and orientation of the marker 31 may fail. Further, since 9 out of 12 black dots on the marker 31 are captured in the captured image I3, if the position and orientation of the marker 31 can be estimated, the reliability is calculated. At that time, the non-shielding posture extraction unit 26 uses the ratio of the number of feature points on the marker 31 obtained by analyzing the captured image and the total number of feature points as the reliability of the estimation, for example, as described above. calculate. When the center of the black circle is used as a feature point, the number of feature points obtained by image analysis is 9 out of 12 feature points in total, and the reliability of estimation is calculated as 75%. Moreover, when the threshold value to be compared with the reliability is set to 90% in advance, the reliability in the photographed image I3 is less than the threshold.

Through the above-described processing of the non-shielded posture extraction unit 26, it is determined whether or not the marker 31 is shielded, and the posture set (postures shown in the captured images I2, I4, I5, . . . ) determined to be unshielded. is generated. As a result, only a posture is generated in which the marker 31 appears in the right and center portions of the captured image.
In addition, the non-shielding posture extraction unit 26 virtually generates captured images obtained by the imaging unit 32 when the robot 30 is moved to each posture (postures corresponding to variables i=1 to N2), The three-dimensional position of the marker 31 is estimated by analyzing the generated captured image, and whether or not the marker 31 is shielded is determined depending on whether the estimation is successful. Specifically, if the estimation is successful, it is determined that the marker 31 is not shielded, and if the estimation is unsuccessful, it is determined that the marker 31 is shielded. This makes it possible to extract a posture in which the marker 31 is not blocked.
Further, when the three-dimensional position of the marker 31 is successfully estimated, the unshielded posture extraction unit 26 obtains the reliability of the estimation, and determines whether the marker 31 is shielded or not depending on whether the reliability is equal to or greater than a threshold. do. Specifically, if the reliability of the estimation is equal to or higher than the threshold, it is determined that the marker 31 is not shielded, and if the reliability of the estimation is less than the threshold, it is determined that the marker 31 is shielded. As a result, among the postures for which the three-dimensional position of the marker 31 has been successfully estimated, only postures for which the reliability of estimation is higher than the threshold can be extracted as postures in which the marker 31 is not shielded.

In the present embodiment, the non-shielding posture extraction unit 26 calculates the reliability of the estimation, but depending on the shape or pattern of the marker 31, the reliability of the estimation may not be calculated. That is, the reliability of estimation may be calculated as necessary. Further, calculation of the reliability is not limited to the area ratio and the feature point ratio described above, and when the position and orientation of the marker 31 are estimated using machine learning or the like, the reliability of the estimation itself may be estimated.

Next, the processing contents of the posture set evaluation unit 27 will be described in detail.
The posture set evaluation unit 27 uses the posture sets generated by the posture set generation unit 23 and extracted by the non-physical interference posture extraction unit 25 and the non-shielding posture extraction unit 26 to perform calibration with a high accuracy equal to or higher than a predetermined value. It is determined whether or not the In addition, the posture set evaluation unit 27 instructs the calibration execution unit 29 to execute the calibration when high-precision calibration of a predetermined value or more can be expected. In addition, when the calibration with a higher accuracy than a predetermined value cannot be expected, the posture set evaluation unit 27 instructs the posture set generation unit 23 to add a posture set and/or changes the value of the generation parameter. command to regenerate the posture set.

FIG. 19 is a functional block diagram of the posture set evaluation unit 27. As shown in FIG.
As shown in FIG. 19 , the posture set evaluation unit 27 includes a posture number evaluation unit 271 , a posture set extraction unit 272 , a calibration evaluation unit 273 and an index evaluation unit 274 .

The posture number evaluation unit 271 evaluates the number of postures included in the posture set generated by the posture set generation unit 23 and extracted by the non-physical interference posture extraction unit 25 and the non-shielding posture extraction unit 26 . Specifically, the posture number evaluation unit 271 determines whether or not the number of postures included in the posture set is within a predetermined number.
When the posture number evaluation unit 271 determines that the number of postures included in the posture set is not within the predetermined number, the posture set extraction unit 272 extracts some of the multiple postures included in the posture set. Extract the subset where . For example, if a total of 100 poses are included in the pose set, 20 poses out of the 100 poses are extracted as a subset.

The calibration evaluation unit 273 generates data such as the marker/camera conversion matrix M2 and the arm/base conversion matrix M3 that are virtually used for calibration on the simulator using the above-described posture set, and estimates them by calibration. Evaluate the estimation accuracy of the marker-camera transformation matrix M2. That is, the calibration evaluation unit 273 evaluates the accuracy of calibration.
The index evaluation unit 274 adds or adds a posture set to the posture set generation unit 23 when the calibration evaluation unit 273 determines that a calibration accuracy higher than a predetermined value cannot be expected due to, for example, an insufficient number of postures. Instruct to generate again.

The processing contents of each unit of the posture set evaluation unit 27 will be described in detail below.
First, when the number of postures included in a posture set is greater than a threshold determined by the prior knowledge 21 or the like, the posture number evaluation unit 271 extracts a portion of the posture set, that is, a subset. The extraction unit 272 is instructed. In addition, the number-of-postures evaluation unit 271 causes the calibration evaluation unit 273 to determine whether or not the subset extracted by the posture set extraction unit 272 can be expected to have high calibration accuracy equal to or higher than a predetermined value. In this way, by extracting a part of poses from the pose set as a subset, it is possible to suppress the length of time required for calibration when the number of poses included in the pose set is large, and to perform the calibration efficiently. Accuracy can be evaluated.

The posture set extraction unit 272 extracts some postures from a posture set, which is a set of multiple postures, based on a predetermined rule. As for the rule, for example, there is a method of extracting so that an index assigned to the small space in which each posture is generated appears on average. As a specific example, when the maximum number of postures to be extracted is set to 21, posture set extraction section 272 extracts some postures based on the following rules.

First, as shown in FIG. 9, among postures (teaching postures) corresponding to a total of 27 small spaces obtained by dividing the teaching space T1, 9 postures of the small spaces given an index of Z=1 are 7 poses in the small space indexed with Z = 2, and 7 poses in the small space indexed with Z = 3. When extracted by the extraction unit 25 and the unoccluded orientation extraction unit 26, two orientations of the small space indexed with Z=1, which has the largest number of extracted orientations, are eliminated. This makes it possible to extract some poses so that the indices appear on average.

The calibration evaluation unit 273 generates a marker-camera conversion matrix M2 and an arm-base conversion matrix M3 that are used when performing virtual calibration on the simulator, and performs optimization processing (described later) to convert the base-camera conversion matrix M1. Find an estimate. Further, the calibration evaluation unit 273 compares the estimated value of the base-camera conversion matrix M1 with the correct value at the time of simulator design, and determines the calibration accuracy required for executing the robot work, that is, the calibration accuracy higher than a predetermined value. It is determined whether or not there is A method for obtaining the estimated value of the base-camera conversion matrix M1 will be described later together with the processing content of the calibration execution unit 29 .

The calibration evaluation unit 273 may obtain an estimated value of the base-camera conversion matrix M1 by adding an error to the generated marker-camera conversion matrix M2 and arm-base conversion matrix M3. The reason is as follows.
When calibration is actually performed by the real device 2A, errors due to noise and the like are added to the observed data. Therefore, by adding an error to the marker/camera conversion matrix M2 and the arm/base conversion matrix M3 as described above, calibration evaluation can be performed under conditions close to those of the actual machine.

In that case, the errors to be added to the marker/camera conversion matrix M2 and the arm/base conversion matrix M3 may adopt, for example, errors following a normal distribution. The standard deviation of the normal distribution at this time may be specified by the prior knowledge 21 or the like, or may be specified by a value having a constant ratio with respect to the values of the marker/camera conversion matrix M2 and the arm/base conversion matrix M3.

For example, if the observed data is the encoder value of the robot 30, the noise on the observed data is noise caused by an error in the robot mechanism. and noise caused by image distortion. The observation data may also be the position and orientation of the marker 31 obtained by analyzing the image captured by the imaging unit 32 .

When the calibration evaluation unit 273 determines that calibration with a higher accuracy than a predetermined value can be expected, it instructs the calibration execution unit 29 to perform calibration. The calibration execution unit 29 receives an instruction from the calibration evaluation unit 273 and executes calibration to estimate the base-camera conversion matrix M1. Further, when the calibration evaluation unit 273 determines that calibration with a precision higher than a predetermined value cannot be expected, the calibration evaluation unit 273 instructs the index evaluation unit 274 to perform index evaluation.

When the index evaluation unit 274 receives an index evaluation execution instruction from the calibration evaluation unit 273 (when calibration with a higher accuracy than a predetermined value cannot be expected), the posture set generation unit 274 generates additional postures. and/or instruct the parameter updating unit 28 to change the value of the generation parameter 22 . By performing such feedback, the posture set generation unit 23 can generate a posture set again, and can also generate a posture set including additional postures or a posture set based on the updated generation parameters 22. can be generated. As a result, when the posture set generation unit 23 generates a posture set again, it is possible to increase the possibility that calibration with a precision higher than the predetermined value can be expected.

The index evaluation unit 274 acquires the posture set evaluated by the calibration evaluation unit 273, and if the number of postures included in the posture set is less than a predetermined threshold value, the index (X , Y, Z), the parameter updating unit 28 is instructed to increment the index that has the lowest frequency of appearance when searching the set of all postures. Also, when the number of postures included in the acquired posture set is greater than or equal to a predetermined threshold value, the index evaluation unit 274 instructs the posture set generation unit 23 to generate additional postures. As an example of the generation method, there is a method of randomly extracting a plurality of postures from the obtained posture set and generating a teaching position and a teaching posture in a small space corresponding to the posture index. is not limited to

Through the processing of the posture set evaluation unit 27 described above, it is possible to feed back the evaluation result of the posture set to the posture set generation unit 23 and the parameter update unit 28 . Further, only when highly accurate calibration equal to or higher than a predetermined value can be expected, the calibration executing section 29 can be caused to execute the calibration. Therefore, it is possible to prevent the accuracy of calibration from deteriorating due to an insufficient number of postures or the like. Further, when calibration with a precision higher than a predetermined value cannot be expected, the posture set generation unit 23 can be caused to generate a posture set again.

In the present embodiment, the posture set evaluation unit 27 has the posture number evaluation unit 271 and the posture set extraction unit 272, but is not limited to this, and has only the calibration evaluation unit 273 and the index evaluation unit 274. may be configured. Also, the function of the calibration evaluation unit 273 and the function of the index evaluation unit 274 may be integrated into one functional unit.
Further, in the present embodiment, as a rule applied to the processing of the posture set extraction unit 272, the rule that the index given to each small space appears on average was exemplified. A part of postures may be extracted at random.

The parameter update unit 28 increments the values of the parameters X1, Y1, and Z1 that determine the resolution of the posture set based on the instruction from the posture set evaluation unit 27. Thereby, the generation parameter 22 is updated. Therefore, the pose set generator 23 uses the updated generation parameters 22 to generate a pose set.

FIG. 20 is a flow chart showing the processing procedure of the calibration executing section 29. As shown in FIG.
The calibration execution unit 29 controls the robot 30 and the imaging unit 32 of the physical device 2A, and calculates the marker/camera conversion matrix M2 and the arm/base conversion matrix M3 when the robot 30 is moved to each posture of the posture set. and get. Further, the calibration executing unit 29 estimates the base/camera conversion matrix M1 based on the acquired coordinate conversion parameters of the marker/camera conversion matrix M2 and the arm/base conversion matrix M3.

First, the calibration executing unit 29 loads, onto the memory of the control device 1, a set of postures that the posture set evaluating unit 27 has determined (evaluated) that calibration with a predetermined value or more can be expected (step S37).
Next, the calibration execution unit 29 acquires the number N3 of postures included in the posture set (step S38).
Next, the calibration execution unit 29 sets the variable i to an initial value (i=1) (step S39).
Next, the calibration executing unit 29 actually moves the robot 30 to the i-th posture in the physical device 2A (step S40).
Next, the calibration execution unit 29 captures an image of the work space of the robot 30 using the imaging unit 32 (step S41).
Next, the calibration executing unit 29 estimates the marker-camera conversion matrix M2 by analyzing the captured image obtained by the imaging unit 32 (step S42).

Next, the calibration execution unit 29 determines whether or not the estimation of the marker-camera conversion matrix M2 has succeeded (step S43). In this step S43, there is a possibility that the estimation of the marker-camera transformation matrix M2 may fail due to the actual lighting environment in the physical device 2A or the specification/arrangement deviation from the design values of the imaging unit 32 of the physical device 2A. be.

Next, when the calibration execution unit 29 succeeds in estimating the marker-camera conversion matrix M2, the arm-base conversion matrix M3 indicating the posture value of the i-th robot 30 and the posture of the i-th robot 30 A marker-camera conversion matrix M2 indicating the position and orientation of the corresponding marker 31 is stored in the memory of the control device 1 (step S44).

Next, the calibration execution unit 29 determines whether the value of the variable i satisfies i=N3 (step S45). Then, when the value of the variable i is less than N3, the calibration execution unit 29 increments the value of the variable i by 1 (step S46), and then returns to the process of step S40.

On the other hand, when the calibration execution unit 29 determines that the estimation of the marker-camera conversion matrix M2 has failed in step S43, it skips the process of step S44.

After that, when the value of the variable i reaches N3, the calibration execution unit 29 solves the optimization problem from the data in the memory, estimates the base-camera conversion matrix M1 (step S47), and then performs a series of Finish processing.
Solving the optimization problem in step S47 corresponds to optimization processing. As a method for estimating the base-camera conversion matrix M1 by this optimization process, a known technique shown in the following references can be used.

(References)
"Simultaneous Robot-World and Hand-Eye Calibration" F. Dornaika, R. Horaud, Aug1998.
A summary of the known technology disclosed in this reference is as follows.
Assuming that the coordinate transformation parameter (transformation matrix) between the robot base and the arm tip is AA, and the coordinate transformation parameter between the arm tip and the marker is BB, the coordinate transformation from the robot base to the marker is AA×BB. On the other hand, if CC is the coordinate transformation parameter of the robot base and the camera, and DD is the coordinate transformation parameter of the camera and the marker, the coordinate transformation from the robot base to the marker is CC×DD. That is, the condition AA*BB=CC*DD is established. Here, the coordinate transformation parameter AA is a known parameter based on robot encoder values, the coordinate transformation parameters BB and CC are unknown parameters, and the coordinate transformation parameter DD is a known parameter through image analysis. Therefore, in the optimization process, coordinate transformation parameters BB and CC that satisfy the condition AA×BB=CC×DD are estimated by numerical calculation from a large number of coordinate transformation parameters AA and DD, which are known parameters.
By the processing of the calibration execution unit 29 described above, the base-camera conversion matrix M1 in the physical device 2A can be estimated.

Further, according to the calibration device 2 according to the present embodiment, the posture set generation unit 23 generates the posture set of the robot 30 for enabling the imaging unit 32 to image (shoot) the marker 31, and the posture set is From among them, the non-shielded posture extraction unit 26 extracts postures in which the marker 31 is not shielded. Therefore, it is possible to automatically generate a set of postures in which the marker 31 is not blocked. Therefore, the posture set of the robot 30 can be taught in consideration of the shielding of the marker 31, and the number of man-hours required for teaching the posture set of the robot 30 can be reduced.

Further, according to the calibration device 2 according to the present embodiment, the non-physical interference posture extraction unit 25 extracts postures in which the robot 30 and the marker 31 do not physically interfere from the posture set generated by the posture set generation unit 23. . Therefore, it is possible to automatically generate a set of postures in which the robot 30 and the marker 31 do not physically interfere. Therefore, the pose set of the robot 30 can be taught in consideration of the physical interference in the physical device 2A, and the number of man-hours required for teaching the pose set of the robot 30 can be reduced.

In the above embodiment, the non-physical interference posture extraction unit 25 extracts the non-physical interference posture from the posture set generated by the posture set generation unit 23, and from the posture set of the extracted non-physical interference postures, Although the non-shielding posture extraction unit 26 is configured to extract the non-shielding posture, the present invention is not limited to this. For example, the non-shielding posture extraction unit 26 extracts the non-shielding posture from the posture set generated by the posture set generation unit 23, and the non-physical interference posture extraction unit 25 extracts the non-shielding posture from the extracted posture set of the non-shielding posture. It may be configured to extract the physical interference posture. Further, the non-physical interference posture extraction unit 25 extracts a non-physical interference posture from the posture set generated by the posture set generation unit 23, and the non-shielding posture extraction unit 26 extracts the posture set generated by the posture set generation unit 23. A configuration in which a non-physical interference posture is extracted from the non-physical interference posture extraction unit 25 and the non-shielding posture extraction unit 26, and the posture set evaluation unit 27 evaluates a common posture among the posture sets extracted respectively by the non-physical interference posture extraction unit 25 and the non-shielding posture extraction unit 26. may be Alternatively, of the non-physical interference posture extraction unit 25 and the non-shielding posture extraction unit 26, only the non-shielding posture extraction unit 26 may be provided. The above points also apply to the second embodiment described later.

<Second embodiment>
FIG. 21 is a diagram showing a configuration example of a robot control calibration device according to the second embodiment.
As shown in FIG. 21, the calibration device 2-1 includes a control device 1, a real device 2C, and a calibration control device 2D.

The real device 2C has a robot 30 consisting of a robot arm, a marker 31 attached to the robot 30, and an imaging unit 32-1 that measures the work space of the robot 30 and its surroundings. The configuration of the physical device 2C is the same as that of the first embodiment except for the imaging unit 32-1. The imaging unit 32-2 is composed of a plurality of (two in the figure)

imaging devices

32A and 32B. Each

imaging device

32A, 32B is configured by, for example, a monocular camera, a ToF (Time of Flight) camera, a stereo camera, or the like.

The calibration control device 2D includes prior knowledge 21-1, generated parameters 22-1, posture set generator 23-1, simulator construction unit 24-1, non-physical interference posture extraction unit 25-1, and non-physical interference posture extraction unit 25-1. A common posture extraction unit 201 is provided in addition to a shielding posture extraction unit 26-1, a posture set evaluation unit 27-1, a parameter update unit 28-1, and a calibration execution unit 29-1. Configurations other than the common posture extraction unit 201 are basically the same as those of the first embodiment.

However, in the present embodiment, the imaging section 32-1 is composed of a plurality of

imaging devices

32A and 32B. For this reason, prior knowledge 21-1, generation parameter 22-1, posture set generation unit 23-1, simulator construction unit 24-1, non-physical interference posture extraction unit 25-1, non-shielding posture extraction unit 26-1, a posture set evaluation unit 27-1, a parameter update unit 28-1, and a calibration execution unit 29-1 perform the same operations as in the first embodiment on the plurality of

imaging devices

32A and 32B, respectively. Similar processing operations are performed. For example, the posture set generation unit 23-1 generates a posture set of the robot 30 that enables the imaging device 32A to photograph the marker 31, and a posture set of the robot 30 that enables the imaging device 32B to photograph the marker 31. are generated separately. Individual processing for each of the

imaging devices

32A and 32B is performed not only by the posture set generation unit 23-1, but also by the simulator construction unit 24-1, the non-physical interference posture extraction unit 25-1, the non-shielding posture extraction unit 26-1, the posture set The evaluation section 27-1, the parameter update section 28-1, and the calibration execution section 29-1 are also performed.

The common posture extraction unit 201 is generated by the posture set generation unit 23-1 for each of the

imaging devices

32A and 32B, and is generated by the non-physical interference posture extraction unit 25-1 and the non-shielding posture extraction unit 26-6 for the

imaging devices

32A and 32B. A plurality of postures that can be used in common by the plurality of

imaging devices

32A and 32B are extracted from the posture set extracted for each 32B. Further, the common orientation extraction unit 201 changes the orientation set of each

imaging device

32A, 32B so that the plurality of extracted orientations remain. A detailed description will be given below.

The common posture extraction unit 201 determines whether or not each posture included in a posture set generated corresponding to a certain imaging device 32A is applicable to another imaging device 32B, and performs similar determination processing. This is performed for all combinations of

imaging devices

32A and 32B. As an example, whether or not each orientation included in a set of orientations generated corresponding to a certain imaging device 32A is applicable to another imaging device 32B is determined on the simulator constructed by the simulator construction unit 24-1. When the robot 30 is controlled by , and the marker 31 is moved to each posture, it is possible to determine whether or not the following conditions (1) to (3) are satisfied.
(1) The marker 31 is included within the teaching space of the imaging device 32B.
(2) The rotation matrices of the marker 31 and the imaging device 32B are within the preset threshold range.
(3) The function of the non-occlusion posture extraction unit 26-1 can determine that occlusion does not occur.

In the determination result described above, an orientation that can be used in common by the plurality of

imaging devices

32A and 32B (hereinafter also referred to as "common orientation") constitutes an orientation set of the

imaging devices

32A and 32B to which the orientation can be applied. Add as one of postures. As a result of this processing, if the upper limit of the number of postures determined in advance is exceeded, postures that can be used only by one of the imaging devices (32A or 32B) are removed from the posture set of the

imaging devices

32A and 32B. adjusts (reduces) the number of postures to be applied for calibration by the motion executing unit 29; As for the priority in removing the orientation, for example, there is a method of preferentially removing the orientation in the same small space as the small space in which the added orientation is located.

According to the calibration device 2-1 according to the present embodiment, the following effects can be obtained in addition to the effects similar to those of the first embodiment.
In this embodiment, the common orientation extraction unit 201 extracts a plurality of orientations that can be used in common by the plurality of

imaging devices

32A and 32B, and the calibration execution unit 29-1 performs calibration using the extracted plurality of orientations. run the Therefore, even when the imaging unit 32 is configured with a plurality of

imaging devices

32A and 32B, it is possible to perform calibration while reducing man-hours required for teaching and calibrating the posture set of the robot 30 .

In this embodiment, the common posture extraction unit 201 is generated by the posture set generation unit 23-1 for each of the

imaging devices

32A and 32B, and the non-physical interference posture extraction unit 25-1 and the non-shielding posture extraction unit 26 -6 extracts a common orientation from among the orientation sets extracted for each of the

imaging devices

32A and 32B, but the present invention is not limited to this. For example, the common posture extraction unit 201 may extract common postures from among the posture sets generated for each of the

imaging devices

32A and 32B by the posture set generation unit 23-1. Further, the common posture extraction unit 201 may extract a common posture from among the posture sets extracted for each of the

imaging devices

32A and 32B by the non-physical interference posture extraction unit 25-1. Further, the common posture extraction unit 201 may extract common postures from the posture set extracted for each of the

imaging devices

32A and 32B by the unshielded posture extraction unit 26-6.

In the present embodiment, the common posture extraction unit 201 controls the robot 30 on the simulator and determines common usable postures. , the determination may be made only from the teaching position and orientation of the marker 31 . Further, by increasing the number of postures during calibration without setting the upper limit of the number of postures included in the posture set of each

imaging device

32A, 32B, an improvement in accuracy can be expected in some cases. In that case, it is not necessary to predetermine the upper limit of the number of postures.

<Modifications, etc.>
In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, in the above-described embodiments, the details of the present invention have been described for easy understanding, but the present invention is not necessarily limited to having all the configurations described in the above-described embodiments. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is also possible to delete a part of the configuration of each embodiment, add another configuration, or replace it with another configuration.

2, 2-1... calibration device, 2A, 2C... real device, 23, 23-1... posture set generator, 25, 25-1... non-physical interference posture extraction unit, 26, 26-1... non-shielding posture Extraction unit 27, 27-1 Posture set evaluation unit 28, 28-1 Parameter update unit 29, 29-1 Calibration execution unit 30 Robot (robot arm) 31 Marker 32

Imaging Section

32A, 32B... Imaging device 33... Environment installation object 201... Common posture extraction unit M1... Base/camera conversion matrix (coordinate conversion parameter)

Claims

an imaging unit that captures an image from a predetermined position;
a marker attached to a robot and displaced according to the motion of the robot;
a posture set generation unit that generates a posture set including a plurality of postures of the robot for enabling the imaging unit to capture the marker;
a non-occluded pose extraction unit that extracts a plurality of poses in which the marker is not shielded from the pose set generated by the pose set generator;
a calibration executing unit for estimating coordinate transformation parameters for transforming the coordinate system of the robot and the coordinate system of the imaging unit based on the plurality of poses extracted by the unshielded posture extracting unit;
A calibration device for robot control.
The imaging unit is configured by a plurality of imaging devices,
The pose set generation unit individually generates, for each of the plurality of imaging devices, a pose set of the robot for enabling the imaging device to capture an image of the marker,
a common posture extracting unit that extracts a plurality of postures that can be used in common by the plurality of imaging devices from among the posture sets of the robot individually generated for each of the imaging devices by the posture set generation unit;
2. The calibration device for robot control according to claim 1, wherein the calibration execution unit applies the plurality of orientations extracted by the common orientation extraction unit to estimate the coordinate transformation parameters.
A method for extracting, from the pose set generated by the pose set generation unit, a plurality of poses in which the robot and the marker do not physically interfere with any part of the real device including the robot and the environment installation object. further comprising a physical interference posture extraction unit,
2. The robot control according to claim 1, wherein the non-shielding posture extracting unit extracts a posture in which the marker is not shielded from a posture set including the plurality of postures extracted by the non-physical interference posture extracting unit. calibration equipment for
further comprising a posture set evaluation unit that determines whether or not calibration with a high accuracy of a predetermined value or higher can be expected using the posture set including the plurality of postures extracted by the unshielded posture extraction unit;
2. The calibration for robot control according to claim 1, wherein the calibration execution unit estimates the coordinate transformation parameters when the posture set evaluation unit determines that calibration with a high accuracy equal to or higher than the predetermined value can be expected. Device.
5. The robot control robot according to claim 4, wherein the posture set evaluation unit instructs the posture set generation unit to generate additional postures when calibration with high precision equal to or higher than the predetermined value cannot be expected. calibration device.
The apparatus further includes a parameter updating unit that changes a value of a parameter used by the posture set generation unit to generate the posture set when the posture set evaluation unit determines that calibration with high accuracy equal to or higher than the predetermined value cannot be expected. The calibration device for robot control according to claim 4.
The non-shielding posture extraction unit virtually generates a photographed image obtained by the imaging unit when the robot is moved to each posture, and analyzes the generated photographed image to determine the cubic position of the marker. The calibration device for robot control according to claim 1, wherein an original position is estimated, and whether or not the marker is shielded is determined based on whether or not the estimation is successful.
8. The non-shielding posture extraction unit according to claim 7, wherein, when the estimation is successful, the reliability of the estimation is obtained, and whether or not the marker is shielded is determined based on whether or not the reliability is equal to or greater than a threshold. calibration device for robot control.