JP7437343B2 - Calibration device for robot control - Google Patents

Description

The present invention relates to a calibration device for controlling a robot.

In order to resolve the labor shortage in recent years, autonomous robots are now using autonomous robots to perform various tasks that previously were performed by humans, such as picking in the logistics field and assembly in the industrial field. The need for automation is increasing. To achieve this automation, an imaging unit is required to accurately recognize the surrounding environment of the workpiece, in order for the robot to perform appropriate autonomous tasks.

In order for the robot to recognize the surrounding environment based on the images taken by the imaging unit and to work autonomously according to the recognition results, calibration is required to determine the relative placement (position, posture) of the imaging unit with respect to the robot. It is necessary to implement For this calibration, for example, a target marker (hereinafter also referred to as a "marker") with a known shape is attached to the robot, the robot is photographed in various postures by an imaging unit, and the photographed images and the robot's posture are There is a method of estimating the relative position and orientation between the robot and the imaging unit using a plurality of sets of data associated with each other.

However, with the above calibration method, there is a concern that the number of steps will increase. For example, in order to teach multiple robot postures (hereinafter also referred to as a "posture set") required during calibration, experts need to perform trial and error depending on the surrounding environment where the robot system is installed. be. Therefore, many man-hours are required for calibration. Furthermore, when a robot performs an amorphous task such as a picking task, the arrangement of the imaging unit may be changed depending on the items existing in the surrounding environment. When actually changing the arrangement of the imaging unit, it is necessary to teach the robot's posture set each time. Therefore, the number of man-hours required for calibration (hereinafter also referred to as "calibration man-hours") may become enormous. Against this background, in recent years, there has been increasing attention to mechanisms and prototypes for reducing calibration man-hours.

Regarding calibration for robot control, for example, Patent Document 1 states that, as a solution for ``easily performing calibration,'' ``a robot control system is based on an image captured by an imaging section, A measuring unit that measures the three-dimensional coordinates of an arbitrary object existing within the field of view of the robot, and a measuring unit that measures the three-dimensional coordinates of an arbitrary object that exists within the field of view of the robot. A command generation unit that generates commands for positioning the action part, a calibration execution part that performs calibration to calculate correspondence, and a reference object that is associated with the action part of the robot during calibration. and a settings reception unit that accepts settings for a calibration area that is a calibration area.''

In other words, the technology described in Patent Document 1 defines a space (calibration area) in which markers are moved during calibration based on information about the robot photographed by an imaging unit, and automatically arranges markers within that space. By generating and teaching a set of poses, the number of steps required for calibration is reduced. According to the technology described in Patent Document 1, since the marker is moved within the range of the calibration area whose settings are accepted by the setting reception unit, it is possible to perform calibration while reducing the man-hours required for teaching the robot's posture set. can be executed.

JP2019-217571A

However, the technique described in Patent Document 1 does not take into consideration marker occlusion (hereinafter also referred to as "occlusion"), which may occur when the robot is changed to various postures. Therefore, the following problems arise.
Generally, if there are objects other than the robot around the robot or around the imaging unit, or if there are parts attached to the robot other than the marker between the marker and the imaging unit, the imaging unit Markers may be obscured when viewed. Furthermore, it takes a huge amount of man-hours to define the movement space of the markers so that the markers are not blocked and to teach the robot a set of postures.

SUMMARY OF THE INVENTION An object of the present invention is to provide a calibration device for controlling a robot that is capable of teaching a set of robot postures in consideration of marker occlusion and reducing the number of man-hours required for teaching the set of robot postures. .

In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes a plurality of means for solving the above problems, but to name one, it includes an imaging section that captures an image from a predetermined position, and a marker that is attached to a robot and is displaced according to the robot's movements. , a posture set generation unit that generates a posture set including a plurality of postures of the robot so that the marker can be imaged by the imaging unit, and a posture set that does not cause occlusion of the marker from among the posture set generated by the posture set generation unit. a non-occluded posture extraction section that extracts a plurality of postures; and a first coordinate transformation parameter for converting the coordinate system of the robot and the coordinate system of the imaging section based on the plurality of postures extracted by the non-occluded posture extraction section. This is a calibration device for robot control, comprising: a calibration execution unit for estimating .
In the robot control calibration device of the present application, the calibration execution unit moves the robot to each of the plurality of postures extracted by the non-obstructed posture extraction unit, and moves the robot's work space to the imaging unit. The calibration execution unit estimates a second coordinate transformation parameter for transforming the coordinate system of the marker and the coordinate system of the imaging unit in the posture by analyzing the photographed image obtained by the photographing, and the calibration execution unit: Furthermore, the first coordinate transformation parameters are estimated by performing numerical calculations using the second coordinate transformation parameters for each of the plurality of orientations.

According to the present invention, it is possible to teach a set of robot postures in consideration of marker occlusion, and to reduce the number of man-hours required for teaching the set of robot postures.
Problems, configurations, and effects other than those described above will be made clear by the following description of the embodiments.

1 is a diagram showing a configuration example of a calibration device for robot control according to a first embodiment; FIG. It is a block diagram showing an example of composition of a control device in this embodiment. FIG. 2 is a diagram showing a coordinate system and coordinate transformation of a real device in this embodiment. FIG. 3 is a functional block diagram of a pose set generation unit. FIG. 3 is a diagram showing a list of parameters acquired by a parameter acquisition unit. 3 is a flowchart showing a parameter acquisition procedure by a parameter acquisition unit. 3 is a flowchart showing a procedure for generating and saving a posture set. FIG. 3 is a diagram illustrating a method of generating a teaching space by a teaching space generation unit. FIG. 3 is a diagram illustrating a method of dividing a teaching space by a teaching space generation unit. FIG. 6 is a diagram illustrating a method of generating a taught position by a taught position generating section. FIG. 7 is a diagram showing how the markers appear when the markers are placed in the initial posture at the teaching positions of each small space. FIG. 7 is a diagram showing a teaching attitude of a marker generated by a teaching attitude generation unit. FIG. 2 is a diagram illustrating an example of a simulator constructed by a simulator construction unit. 7 is a flowchart showing a processing procedure of a non-physical interference posture extraction section. FIG. 2 is an overhead view of the posture of the robot when it actually performs picking work. FIG. 3 is a side view of the posture of the robot when it actually performs picking work. 7 is a flowchart illustrating a processing procedure of a non-shielding posture extraction section. FIG. 6 is a schematic diagram showing an example of the operation of a non-shielding posture extraction section. FIG. 3 is a functional block diagram of a posture set evaluation section. 3 is a flowchart showing a processing procedure of a calibration execution unit. FIG. 7 is a diagram illustrating a configuration example of a calibration device for robot control according to a second embodiment.

Embodiments of the present invention will be described in detail below with reference to the drawings. In this specification and the drawings, elements having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted.

<First embodiment>
FIG. 1 is a diagram showing a configuration example of a calibration device for robot control according to a first embodiment. In this embodiment, as an example of the work performed by the robot, a picking work in which a workpiece is moved to a predetermined position is assumed. Further, in this embodiment, a robot arm (hereinafter also simply referred to as a "robot") that is an articulated robot is assumed as an example of a robot to be controlled. Further, in this embodiment, as an example of a marker attached to a robot, a calibration board is assumed, in which a pattern with known dimensions is printed on a flat jig. Further, in this embodiment, a monocular camera is assumed as an example of the imaging unit. Furthermore, in this embodiment, a wall and a conveyor are assumed as examples of environmental installations installed around the robot or around the imaging unit. However, the content assumed in this embodiment is just an example, and the configuration of the calibration device for robot control is not limited to the example assumed in this embodiment.

As shown in FIG. 1, the calibration device 2 includes a control device 1, a real device 2A, and a calibration control device 2B.
The control device 1 controls the entire calibration device 2 in an integrated manner. Further, the control device 1 controls the operation of the robot 30 and the operation of the imaging unit 32 in the actual device 2A. Although the control device 1 and the calibration control device 2B are shown separately in FIG. 1, the functions of the calibration control device 2B can be realized by the computer hardware resources of the control device 1. .

The actual device 2A includes a robot 30 configured by a robot arm, a marker 31 attached to the robot 30, and an imaging unit 32 that measures the work space and surroundings of the robot 30. Further, the actual device 2A includes a wall 33A and a conveyor 33B as examples of environmental installations 33 installed around the robot 30 and around the imaging unit 32.

The calibration control device 2B includes a prior knowledge 21, a generation parameter 22, a posture set generation unit 23 that generates a posture set of the robot 30 using the prior knowledge 21 and the generation parameter 22, and a three-dimensional From among the pose sets generated by the simulator construction unit 24, which constructs a simulator (hereinafter also simply referred to as “simulator”), and the pose set generation unit 23, the robot 30 and the marker 31 are placed in any part of the real device 2A. The non-physical interference posture extraction unit 25 extracts a plurality of postures (posture sets) that do not physically interfere with each other, and the non-physical interference posture extraction unit 25 extracts a plurality of postures (posture sets) that do not cause occlusion of the marker 31 from among the posture sets extracted by the non-physical interference posture extraction unit 25. The pose set evaluation unit 27 evaluates the pose set extracted by the non-shielded pose extraction unit 26 from the perspective of calibration accuracy, and the pose set evaluation unit 27 has a low evaluation. and a calibration execution unit 29 that executes calibration when the evaluation by the posture set evaluation unit 27 is high. The posture set of the robot 30 is a set of a plurality of postures (posture data) of the robot 30.

The prior knowledge 21 includes design information including the shape of the actual device 2A, information capable of specifying a space in which the robot 30 performs work (hereinafter also referred to as "work space"), and an evaluation value of the imaging unit 32. This is a database that includes parameter information such as threshold values used in each functional unit. The design information of the actual device 2A is information including the shape, specifications, and arrangement of a three-dimensional model of the actual device 2A. The evaluation value of the imaging section 32 is an evaluation value obtained by an inspection performed when the imaging section 32 is received or shipped, and is a value representing the actual performance of the imaging section 32 relative to the product specifications and the degree of image distortion. The generation parameters 22 are a database having parameters that determine the number and density of postures included in the posture set of the robot 30.

The posture set generation unit 23 generates a posture set of the robot 30 using the prior knowledge 21 and the generation parameters 22 as input, and stores the generated posture set in the control device 1 . The postures of the robot 30 included in the posture set generated by the posture set generating section 23 are postures that allow the imaging section 32 to image the marker 31 (capture the image). The simulator construction unit 24 uses the prior knowledge 21 to construct a three-dimensional simulator that allows virtual movement of the robot 30 and imaging using the imaging unit 32. The non-physical interference posture extraction section 25 reads the posture set generated by the posture set generation section 23 from the control device 1, and also moves the robot to each posture on the three-dimensional simulator constructed by the simulator construction section 24. A plurality of postures in which the robot 30 and the marker 31 do not physically interfere with either the robot 30 or the environmental installation 33 are extracted. In addition, the non-physical interference posture extraction unit 25 stores the extracted plurality of postures in the control device 1 as a set of postures that do not cause physical interference. The non-shielding posture extraction section 26 reads the posture set extracted by the non-physical interference posture extraction section 25 from the control device 1, and moves the robot to each posture on the three-dimensional simulator constructed by the simulator construction section 24. A plurality of postures in which the marker 31 is not blocked between the imaging unit 32 and the marker 31 are extracted. Further, the non-shielding posture extraction unit 26 stores the extracted plurality of postures in the control device 1 as a posture set. In the present embodiment, the storage destination of the posture set generated by the posture set generation section 23 and the storage destination of the posture set extracted by the non-physical interference posture extraction section 25 and the non-shielding posture extraction section 26 are determined by the control device 1. However, the storage location of the posture set is not limited to the control device 1, and may be, for example, a storage section included in the calibration control device 2B.

The pose set evaluation unit 27 reads the pose set extracted by the non-shielded pose extraction unit 26 from the control device 1, and determines whether the estimation accuracy of calibration using the pose set is equal to or higher than a predetermined value set in advance. The pose set is evaluated based on the result of this determination. The parameter update unit 28 updates the generation parameters 22 when the pose set evaluation unit 27 cannot expect a calibration with higher accuracy than a predetermined value. The calibration execution section 29 controls the robot 30 and the imaging section 32 based on the posture set (a plurality of postures) for which the posture set evaluation section 27 has determined that a calibration with higher precision than a predetermined level can be expected, and controls the robot 30 and the imaging section 32. 30 and the coordinate system of the imaging unit 32 are estimated.

Hereinafter, the control device 1, the actual device 2A, the prior knowledge 21, the pose set generation unit 23, the simulator construction unit 24, the non-physical interference pose extraction unit 25, the non-occluded pose extraction unit 26, the pose set evaluation unit 27, and the parameter update unit 28, the calibration execution unit 29 will be explained in detail.

(Control device 1)
FIG. 2 is a block diagram showing a configuration example of the control device 1 in this embodiment.
As shown in FIG. 2, the control device 1 includes a CPU 11, a bus 12 for transmitting commands from the CPU 11, a ROM 13, and a RAM 14, as computer hardware resources for controlling the entire calibration device 2 in an integrated manner. A storage device 15, a network I/F (I/F is an abbreviation for interface, the same applies hereinafter) 16, an imaging I/F 17 for connecting the imaging unit 32, and a screen display I/F 18 for performing screen output. , and an input I/F 19 for performing external input. That is, the control device 1 can be configured by a general computer device.

The storage device 15 stores a program 15A for executing each function, an OS (operating system) 15B, a three-dimensional model 15C, and parameters 15D such as a database. A robot control device 16A is connected to the network I/F 16. The robot control device 16A is a device that controls and operates the robot 30.

(Real device 2A)
In this embodiment, the robot 30 is a picking robot that performs picking work. Generally, picking robots are composed of articulated robots. However, the robot 30 may be any type of robot as long as it has multiple joints. Moreover, the robot 30 is installed on a single-axis slider, and by increasing the degree of freedom of movement of the robot 30 by the single-axis slider, it is possible to perform work in a wide space.
As the marker 31, a calibration board is used, which is a flat jig with a pattern of known dimensions printed thereon. When such a marker 31 is used, the position of the marker 31 in the three-dimensional space can be specified by analyzing data measured by the imaging unit 32. However, as long as the three-dimensional position of the marker 31 can be specified from the data measured by the imaging unit 32, for example, a spherical jig or a specially shaped jig can be used as the marker 31.
The marker 31 is attached to the tip of a robot 30 consisting of a robot arm. However, the mounting position of the marker 31 is not limited to the tip of the arm, but may be on the robot arm as long as the position satisfies the condition that the marker 31 is displaced in accordance with the movement of the robot 30.

Furthermore, in this embodiment, a monocular camera is used as the imaging unit 32. However, the imaging unit 32 is not limited to a monocular camera, and may be configured by, for example, a ToF (Time of Flight) camera, a stereo camera, or the like. That is, the data obtained by the imaging unit 32 is data such as an image or a point group that can specify the three-dimensional position of the marker 31. The imaging unit 32 photographs the work space of the robot 30 from a predetermined position. The mounting position of the imaging unit 32 can be set at any arbitrary location, such as the ceiling or wall of the building where the robot 30 works.

Here, the coordinate system and coordinate transformation of the actual device 2A will be explained.
FIG. 3 is a diagram showing the coordinate system and coordinate transformation of the actual device 2A in this embodiment.
The coordinate system of the actual device 2A includes a base coordinate system C1 whose origin is at the base of the arm of the robot 30, an arm coordinate system C2 whose origin is at the tip of the arm of the robot 30, and a marker coordinate system whose origin is at the center of the marker 31. C3, and a camera coordinate system C4 whose origin is the optical center of the imaging unit 32. Each coordinate system is a three-dimensional coordinate system.

The coordinate transformation of the actual device 2A includes a base/camera transformation matrix M1 representing the coordinate transformation between the base coordinate system C1 and the camera coordinate system C4, and a marker/camera transformation matrix representing the coordinate transformation between the marker coordinate system C3 and the camera coordinate system C4. A transformation matrix M2, an arm/base transformation matrix M3 representing coordinate transformation between arm coordinate system C2 and base coordinate system C1, and an arm/marker transformation matrix M4 representing coordinate transformation between arm coordinate system C2 and marker coordinate system C3. There is. Of these, the base coordinate system C1 corresponds to the coordinate system of the robot 30, and the camera coordinate system C4 corresponds to the coordinate system of the imaging unit 32. Further, the base/camera transformation matrix M1 corresponds to a coordinate transformation parameter for transforming the coordinate system of the robot 30 and the coordinate system of the imaging unit 32. To give an example of coordinate transformation, in the base/camera transformation matrix M1, when the rotation matrix is Rca and the translation matrix is tca, the coordinates C (Xc, Yc, Zc) in the camera coordinate system C4 are expressed by the following formula. As shown, it can be converted to a point P (Xr, Yr, Zr) in the base coordinate system C1.

Next, a method for acquiring the coordinate transformation parameters (M1, M2, M3, M4) of the actual device 2A will be explained.
The base/camera conversion matrix M1 can be obtained as one of the design values in the actual device 2A. However, when the robot 30 is operated by applying the base/camera transformation matrix M1 determined as a design value, an error may occur in the position of the robot 30. One possible cause of such an error is, for example, a shift in the mounting position of the imaging unit 32 in the actual device 2A. Specifically, even if the mounting position of the imaging unit 32 is designed so that the optical axis (z-axis) of the imaging unit 32 is parallel to the vertical direction, a slight positional shift may occur during actual installation. . Calibration is performed to accurately determine the base/camera transformation matrix M1 so that errors in the position of the robot 30 do not occur due to such positional deviations.

The design value of the base/camera transformation matrix M1 is included in the prior knowledge 21. Therefore, the design value of the base/camera transformation matrix M1 can be obtained from the prior knowledge 21. Furthermore, by executing calibration, it is possible to obtain the actual value of the base/camera conversion matrix M1 that reflects the above-mentioned positional deviation and the like. Calibration in this embodiment refers to estimating the base/camera transformation matrix M1 from multiple sets of data of the marker/camera transformation matrix M2 and the arm/base transformation matrix M3 when the robot 30 is moved to a certain posture. Say something. Estimation of the base/camera transformation matrix M1 is performed by calculation.

The marker-camera conversion matrix M2 can be obtained by analyzing an image obtained by photographing the marker 31 using the imaging unit 32. The arm/base transformation matrix M3 can be obtained by calculating from the encoder values of the robot 30. The joint portion of the robot 30 is provided with a motor for driving the joint (not shown) and an encoder that detects the rotation angle of this motor, and the encoder value means the output value of the encoder. The arm marker transformation matrix M4 is unknown, or a design value can be obtained from the prior knowledge 21. In this embodiment, since the design value of the arm marker transformation matrix M4 is included in the prior knowledge 21, the design value of the arm marker transformation matrix M4 can be obtained from this prior knowledge 21. Note that in this embodiment, the origin of each coordinate system, the orientation of the coordinate system, and the coordinate transformation are set as described above, but the invention is not limited to this example.

(prior knowledge 21)
The prior knowledge 21 is a database having the following parameters (1) to (11).
(1) Design value of base/camera transformation matrix M1.
(2) Design value of arm marker conversion matrix M4.
(3) A three-dimensional model containing shape information of the robot 30.
(4) Specification data that defines the upper limit value of the joint angle of the robot 30, etc.
(5) Shape data including the type and size of the marker 31.
(6) Specification data including information such as camera parameters, optical blur, resolution, and angle of view of the imaging unit 32.
(7) Distortion evaluation value of the imaging unit 32.
(8) Information defining the work space of the robot 30.
(9) Design information of the actual device 2A.
(10) Information that defines the shape and arrangement of the environmental installations 33.
(11) Threshold value used in each functional unit.
Note that in this embodiment, a function using the above-mentioned prior knowledge 21 will be described, but the function may be implemented using only a part of it.

(Posture set generation unit 23)
The posture set generation unit 23 generates a posture set of the robot 30 based on the parameters acquired from the prior knowledge 21 and the parameters acquired from the generation parameters 22. Further, the posture set generation unit 23 stores the generated posture set in the control device 1.

FIG. 4 is a functional block diagram of the pose set generation unit 23.
As shown in FIG. 4, the posture set generation section 23 includes a parameter acquisition section 231, a taught space generation section 232, a taught position generation section 233, a taught posture generation section 234, a coordinate system conversion section 235, and a posture set. A storage section 236 is provided.

The parameter acquisition unit 231 acquires parameters necessary for generating a posture set from the prior knowledge 21 and the generation parameters 22. The teaching space generation unit 232 determines a space in which the marker 31 is placed during calibration, and divides this space into a plurality of small spaces. Furthermore, the teaching space generation unit 232 assigns an index to each small space. The teaching position generation unit 233 generates a position at which the marker 31 is placed in each small space, that is, a teaching position, based on the camera coordinate system C4. The teaching posture generating section 234 generates the posture of the marker 31 at each teaching position generated by the teaching position generating section 233, that is, a teaching posture. The coordinate system conversion unit 235 transforms the posture set of the marker 31 based on the camera coordinate system C4 into a robot based on the base coordinate system C1 based on the design value of the base/camera transformation matrix M1 included in the prior knowledge 21. It is converted into a set of 30 poses. The posture set storage section 236 stores the posture set generated by the taught posture generation section 234 and the posture set converted by the coordinate system conversion section 235 in the control device 1 . The configuration of each part of the posture set generation section 23 will be described in more detail below.

FIG. 5 is a diagram showing a list of parameters acquired by the parameter acquisition unit 231.
As shown in FIG. 5, the parameters acquired by the parameter acquisition unit 231 include the design values of the base/camera transformation matrix M1 (rotation matrix Rca, translation matrix tca), the measurement range of the imaging unit 32 (angle of view, resolution, optical blur), distortion information of the imaging unit 32 (distortion evaluation), work space of the robot 30 (three-dimensional spatial information), shape of the marker 31 (board, sphere, size), resolution in each axis direction of the posture set (X1, Y1 , Z1) are included.

FIG. 6 is a flowchart showing a parameter acquisition procedure by the parameter acquisition unit 231.
First, the parameter acquisition unit 231 acquires the design value of the base/camera transformation matrix M1 from the prior knowledge 21 (step S1). At this time, the parameter acquisition unit 231 acquires the rotation matrix Rca and the translation matrix tca as design values of the base/camera transformation matrix M1.

Next, the parameter acquisition unit 231 acquires the measurement range of the imaging unit 32 and distortion information of the imaging unit 32 from the prior knowledge 21 (step S2). At this time, the parameter acquisition unit 231 acquires the angle of view, resolution, and optical blur as the measurement range of the imaging unit 32, and also acquires a distortion evaluation value as distortion information of the imaging unit 32.

Next, the parameter acquisition unit 231 acquires information defining the work space of the robot 30 from the prior knowledge 21 (step S3). At this time, the parameter acquisition unit 231 acquires three-dimensional space information indicating the work space of the robot 30 as information defining the work space.
Next, the parameter acquisition unit 231 acquires shape data of the marker 31 (step S4). At this time, the parameter acquisition unit 231 acquires data on the type (board, sphere) and size of the marker 31 as the shape data of the marker 31.
Next, the parameter acquisition unit 231 acquires the resolution (X1, Y1, Z1) of the posture set in each axis direction from the generation parameters 22 (step S5). Specifically, based on the camera coordinate system C4, the number of postures (X1, Y1, Z1) in each axis is acquired as a parameter that determines the resolution in each axis direction. In this case, the greater the number of postures created in each axis (the greater the values of X1, Y1, and Z1), the higher the resolution in each axis direction. That is, the parameters X1, Y1, and Z1 correspond to parameters that determine the number and density of postures included in the posture set.

FIG. 7 is a flowchart showing a procedure for generating a posture set by the teaching space generation section 232, the teaching position generation section 233, the teaching posture generation section 234, and the coordinate system conversion section 235, and storing this posture set by the posture set storage section 236. It is.
As shown in FIG. 7, first, the teaching space generation unit 232 generates a teaching space in which the marker 31 is placed (step S6). At this time, the teaching space generated by the teaching space generation unit 232 is a three-dimensional space based on the camera coordinate system C4.

Next, the teaching space generation unit 232 divides the teaching space generated earlier into a plurality of small spaces, and assigns an index to each small space (step S7). At this time, the teaching space generation unit 232 divides the teaching space into small spaces of X1×Y1×Z1. For example, if X1=3, Y1=3, and Z1=3, the teaching space generation unit 232 divides the teaching space into 27 small spaces in total by dividing the teaching space into three in each axis direction. do. Furthermore, the teaching space generation unit 232 assigns an index (X, Y, Z) to each small space.

Next, the teaching position generation unit 233 sets the position at which the marker 31 is placed in each small space as the teaching position (step S8). As a result, the teaching positions of the markers 31 are generated by the number of small spaces.
Next, the teaching attitude generation unit 234 initializes the teaching attitude of the marker 31 at each teaching position so that the marker 31 directly faces the imaging unit 32 (step S9). In the following description, the initially set taught attitude of the marker 31 will be referred to as an initial attitude.
Next, the teaching attitude generation unit 234 creates markers 31 by ±θ (threshold value) degrees from 0 degrees in each axis direction of the camera coordinate system C4 from the initial attitude, based on a rule based on the index assigned to each small space or randomly. By rotating the orientation of the marker 31, a taught orientation of the marker 31 is generated (step S10). As a result, the teaching postures of the markers 31 are generated as many as the number of small spaces.

Next, the coordinate system transformation unit 235 converts the coordinates of the teaching attitude of the marker 31 based on the camera coordinate system C4, which was generated by the teaching attitude generation unit 234 in step S10, using the design values of the base/camera transformation matrix M1. Then, the coordinates are converted into the coordinates of the base coordinate system C1 (step S11). Thereby, the taught attitude of the marker 31 based on the camera coordinate system C4 is converted to the taught attitude of the robot 30 based on the base coordinate system C1. Further, the teaching postures of the robot 30 are generated for the number of small spaces.
Next, the posture set storage section 236 stores the posture set (a plurality of taught postures) of the marker 31 generated by the taught posture generation section 234 in step S10, and the posture set (a plurality of taught postures) of the marker 31 generated by the coordinate system conversion section 235 in step S11. A set of postures (a plurality of taught postures) of the robot 30 is stored in the control device 1 (step S12). As a result, the control device 1 stores a set of postures of the marker 31 based on the camera coordinate system C4 and a set of postures of the robot 30 based on the base coordinate system C1.

FIG. 8 is a diagram illustrating a method of generating a teaching space by the teaching space generating unit 232. The illustrated generation method is applied to step S6 in FIG. 7 above.
First, the teaching space is a common area where the space where the robot 30 works and the space where the imaging unit 32 can accurately photograph (measure) overlap.
In FIG. 8, the work space of the robot 30 is determined based on the camera coordinate system C4. For example, when the work to be performed by the robot 30 is a picking work, the work space is determined in advance as a space in which the marker 31 is located when the robot 30 performs the work to grasp a workpiece (not shown). be able to. The reason why the work space of the robot 30 is used as the teaching space T1 is as follows. When moving the robot 30, movement errors occur systematically due to manufacturing errors of the robot 30 and the like. However, the appearance of movement errors differs depending on the spatial position. Therefore, by making the posture that the robot 30 takes during calibration similar to the posture during work, it is possible to use data for calibration that has the same movement error as during work.

The space that can be photographed by the imaging unit 32 is determined by the angle of view of the imaging unit 32, but the space that can be accurately photographed by the imaging unit 32 is limited to a range narrower than the space that can be photographed. In FIG. 8, the size of the space in the Z-axis direction in the camera coordinate system C4 is specified as a space in which the imaging unit 32 can accurately capture images so that the value of the optical blur of the imaging unit 32 is equal to or less than a threshold value. . In the example of FIG. 8, the size of the space in the Z-axis direction is specified within a range where the optical blur value is 1 px or less. Further, as a space in which the imaging unit 32 can accurately capture images, the width of the space in the X-axis direction and the Y-axis direction in the camera coordinate system C4 is an image captured by the imaging unit 32 (hereinafter also referred to as a “captured image”). ) is specified so that the brightness difference between the image before distortion correction and the image after distortion correction falls within a range below a threshold value. Thereby, it is possible to control the marker 31 to be placed in a space where distortion of the captured image is unlikely to occur. Therefore, errors in calibration, which will be described later, can be reduced.

In the example of FIG. 8, the size of the space in the Z-axis direction in the camera coordinate system C4 is defined by the work space of the robot 30. This is because the working space of the robot 30 is narrower than a space where the optical blur value is 1 px or less when comparing the size of the space in the Z-axis direction. On the other hand, the width of the space in the X-axis direction and the Y-axis direction in the camera coordinate system C4 is defined by a space in which the imaging unit 32 can accurately capture images. This means that when comparing the width of the spaces in the X-axis direction and the Y-axis direction, the space in which the imaging unit 32 can accurately capture images, and in which the brightness difference according to the distortion evaluation is less than the threshold value, is the space where the robot 30 is working. This is because it is narrower than space.
The teaching space T1 is generated by the above method.

FIG. 9 is a diagram illustrating a method of dividing the teaching space by the teaching space generation unit 232. The illustrated division method is applied to step S7 in FIG. 7 above.
When dividing the teaching space, parameters X1, Y1, and Z1 given as the resolution in each axis direction of the posture set are used. Specifically, the teaching space T1 generated in step S6 above is divided into X1 pieces in the x-axis direction, Y1 pieces in the y-axis direction, and Z1 pieces in the z-axis direction, with the camera coordinate system C4 as a reference. Divide into. In the example of FIG. 9, the teaching space T1 is divided into X1=3, Y1=3, and Z1=3. Therefore, the teaching space T1 is divided into a total of 27 small spaces. After the teaching space T1 is divided into a plurality of small spaces in this way, an index (X, Y, Z) is assigned to each small space. Indexes (X, Y, Z) are assigned based on the coordinate axes. For example, for the x-axis, indexes of X=1, X=2, and X=3 are assigned from the right side to the left side of FIG. Indexes of Y=2 and Y=3 are assigned, and for the z-axis, indexes of Z=1, Z=2, and Z=3 are assigned from the top to the bottom of FIG. In such a case, the index (X, Y, Z)=(1, 1, 3) is given to the small space located at the lower right of FIG.

FIG. 10 is a diagram illustrating a method of generating a taught position by the taught position generating section 233. The illustrated generation method is applied to step S8 in FIG. 7 above.
First, as an example, the teaching position in a small space given the index (X, Y, Z) = (1, 1, 3) is a small distance in the normal direction from the center position of the top surface of this small space. A position moved by half the height of the space, that is, the center position of the small space given the index (1, 1, 3) can be generated as the teaching position. This point also applies to small spaces assigned other indexes.

Next, a method of generating a taught posture by the taught posture generating section 234 will be explained. The teaching posture is generated by the processing in step S9 and step S10 in FIG. 7 above.
First, the processing content of step S9 will be described. The taught attitude generation unit 234 sets the initial value of the taught attitude so that the marker 31 directly faces the imaging unit 32.
FIG. 11 is a diagram showing how the markers appear when they are placed in the initial posture at the teaching position of each small space.
In FIG. 11, among the plurality of small spaces generated by the teaching space generation unit 232, index (X, Y, Z) = (1, 1, 1) (1, 2, 1) (1, 3, 1) For each small space (2, 1, 1) (2, 2, 1) (2, 3, 1) (3, 1, 1) (3, 2, 1) (3, 3, 1), the teaching position The figure shows how the marker 31 would appear if the marker 31 were placed at the teaching position generated by the generation unit 233 and placed so as to directly face the imaging unit 32. Under the initial attitude, the marker 31 is arranged in any small space such that the Z-axis of the camera coordinate system C4 is directly opposite to the Z-axis of the marker coordinate system C3. Therefore, the marker 31 looks the same in all the small spaces.

Next, the process of step S10 will be explained.
The teaching attitude generation unit 234 generates a teaching attitude by rotating the attitude of the marker 31 in each axis direction of the marker coordinate system C3 from the initial attitude set in step S9.
FIG. 12 is a diagram showing the teaching posture of the marker 31 generated by the teaching posture generating section 234. Note that the index (X, Y, Z) of each small space in FIG. 12 corresponds to the index (X, Y, Z) of each small space in FIG. 11 above. As an example, the taught attitude of the marker 31 in the small space of index (3, 3, 1) is generated by rotating the marker 31 around the x-axis of the marker coordinate system C3 with respect to the initial attitude. The magnitude of the rotation angle of each axis is from 0 degrees to ±θ (threshold value) degrees, and is determined rule-based or randomly based on the index. In this embodiment, the same attitude as the initial attitude is used as the teaching attitude in small spaces with indexes (1, 1, 1) (1, 3, 1) (3, 1, 1), and in small spaces with other indices. In this case, the posture rotated around the axis is used as the teaching posture. In this way, the posture set, which is a plurality of postures generated by the teaching posture generation unit 234, includes postures in which the marker 31 is rotated to have various orientations, and postures in which the marker 31 has the same orientation but differs only in translation. It is composed of a combination. This makes it possible to improve the accuracy when estimating calibration compared to a case where the orientations of the markers 31 are all changed at random (a case where the orientations of the markers 31 are different for all indexes).

Through the processing of each part (231 to 236) of the posture set generation unit 23 described above, a set of postures is automatically generated in a teaching space that satisfies both the work space of the robot 30 and the space where the imaging unit 32 can accurately capture images. It is possible to generate

Note that in this embodiment, the rotation of the base/camera transformation matrix M1 is obtained in the form of a rotation matrix Rca, but a quaternion or the like may also be used. Further, the shape of the teaching space T1 generated by the teaching space generation unit 232 is not limited to the shape shown in FIG. 8. For example, when the working space of the robot 30 is narrower than the space in which the imaging unit 32 can accurately take images, the teaching space T1 has a shape such as a rectangular parallelepiped. In addition, when the distortion evaluation value of the imaging unit 32 used in this embodiment cannot be obtained as a parameter for generating the teaching space T1, only the angle of view or only the angle of view without using the distortion evaluation value can be used. It may be determined to use a predetermined part of the space. Further, as a method for determining the teaching space, the user may specify the shape and size of the teaching space in advance. Further, in this embodiment, as a method of dividing the teaching space, the teaching space T1 is divided based on the camera coordinate system C4, but the teaching space T1 may be divided based on an arbitrary coordinate system. Further, the teaching position generated by the teaching position generation unit 233 may be set to any position within the small space, such as the vertex of the small space. Furthermore, in order to detect the marker 31 with high precision, the teaching attitude generation unit 234 generates the teaching attitude by setting the rotation angle of the marker 31 around each axis within a threshold value. Does not need to be set. Further, in this embodiment, the posture set storage unit 236 stores the posture set in the control device 1, but the present invention is not limited to this, and the posture set may be stored in a memory (not shown), and subsequent processing is performed from the memory. You may also read sets.

Next, the processing contents of the simulator construction unit 24 will be explained in detail.
The simulator construction unit 24 has a function of virtually operating the robot 30 by constructing a simulator based on the prior knowledge 21 and generating a captured image (captured image) by the imaging unit 32.
FIG. 13 is a diagram showing an example of a simulator constructed by the simulator construction section 24. As shown in FIG.
In FIG. 13, information such as a three-dimensional model, specifications, and layout including design information of the actual device 2A is acquired from the prior knowledge 21, and based on the acquired information, the robot 30, marker 31, imaging unit 32, and environmental installations are 33 are virtually arranged on the simulator.
Regarding the robot 30, it is possible to virtually operate the robot 30 by using the length of each link as the shape information of the robot 30, which is information acquired from the prior knowledge 21, and the joint information obtained as the specifications. be. Further, by operating the robot 30 virtually, it is possible to apply the trajectory planning algorithm used in the actual robot.

Regarding the imaging unit 32, by using specification data such as the angle of view, resolution, and camera parameters of the imaging unit 32, which is information acquired from the prior knowledge 21, each image captured by the imaging unit 32 on the simulator is It is possible to generate a value for a pixel. Further, the color information of the robot 30 and the environmental installations 33 can be easily changed on the simulator. For example, it is also possible to change the RGB values representing the colors of the robot 30 and the environmental installations 33 to (255, 0, 0), etc., and generate an image taken on the simulator.

Through the processing of the simulator construction unit 24 described above, it is possible to virtually generate an image photographed by the imaging unit 32 when the marker 31 is moved to the teaching position based on the posture set. In addition, when a three-dimensional model of the robot 30 is obtained, it is possible to virtually plan and control the trajectory of the robot 30, and based on the design layout information of the prior knowledge 21, when the robot 30 operates, the imaging unit 32 It is possible to virtually generate an image taken by

Note that in this embodiment, placement information including design values of the base/camera transformation matrix M1, etc. is acquired from the prior knowledge 21, and objects are placed on the simulator based on the acquired placement information, but this is not limited to this. do not have. For example, a rough value indicating the size and arrangement of the object in the actual device 2A may be estimated from an image taken by the imaging unit 32, and the object may be arranged on the simulator based on the estimation result.

Furthermore, in this embodiment, when the real device 2A is virtually placed in the simulator, the shape is reproduced using a three-dimensional model of the real device 2A, but if a three-dimensional model cannot be obtained, The simulator may hold only design value information for each coordinate system.

In addition, in the case where the imaging unit 32 is a stereo camera or ToF camera capable of three-dimensional measurement, or when a three-dimensional shape can be obtained by machine learning of images from a monocular camera, the actual device 2A acquired by the imaging unit 32 The three-dimensional shape may be used to construct a simulator.

Next, the processing contents of the non-physical interference posture extraction unit 25 will be explained in detail.
FIG. 14 is a flowchart showing the processing procedure of the non-physical interference attitude extraction unit 25.
The non-physical interference posture extraction section 25 extracts postures (non-physical interference postures) in which the respective parts of the actual device 2A do not physically interfere with each other from among the posture set generated by the posture set generation section 23. Further, the non-physical interference posture extraction section 25 extracts the postures that do not cause physical interference by utilizing the simulator constructed by the simulator construction section 24. Physical interference refers to the robot 30 or the marker 31 coming into contact with the robot 30, the imaging unit 32, the environmental installation 33, etc. in the actual device 2A.

First, the non-physical interference posture extraction section 25 loads the posture set generated by the posture set generation section 23 onto the memory of the control device 1 (step S13).
Next, the non-physical interference posture extraction unit 25 obtains the number N1 of postures included in the above posture set (step S14).
Next, the non-physical interference posture extraction unit 25 sets the variable i to an initial value (i=1) (step S15).
Next, the non-physical interference attitude extraction unit 25 instructs a trajectory plan to move the robot 30 to the i-th attitude on the simulator (step S16).
Next, the non-physical interference attitude extraction unit 25 determines whether the trajectory planning instructed in step S16 is successful (step S17). Specifically, when the robot 30 is operated according to the instructed trajectory plan, the non-physical interference posture extraction unit 25 determines the trajectory if the robot 30 and the marker 31 do not physically interfere with any part in the actual device 2A. It is determined that the plan was successful, and if there is physical interference, it is determined that the trajectory plan has failed.

Next, if it is determined that the trajectory planning is successful in step S17, the non-physical interference attitude extraction unit 25 stores the i-th attitude in the memory as a successful pattern, and then (step S18), the i-th The success/failure information for the index of the small space in which the pose is generated is held (step S19). When retaining success information for the index of a small space, a success flag is stored in association with the index of the small space in which the i-th posture was generated. Furthermore, when retaining failure information for the index of a small space, a failure flag is associated and retained with the index of the small space in which the i-th posture was generated.

On the other hand, if the non-physical interference attitude extraction unit 25 determines in step S17 that the trajectory planning has failed, it passes the processing in steps S18 and S19. Here, if the trajectory planning fails, this corresponds to the case where it is determined that there is physical interference, and if the trajectory planning is successful, it corresponds to the case where it is determined that there is no physical interference.

Next, the non-physical interference posture extraction unit 25 determines whether the value of the variable i satisfies i=N1 (step S20). Then, if the value of the variable i is less than N1, the non-physical interference posture extraction unit 25 increments the value of the variable i by 1 (step S21), and then returns to the process of step S16. In addition, when the value of the variable i reaches N1, the non-physical interference posture extraction unit 25 stores the posture set of the successful pattern and the success/failure information for each index in the control device 1 (step S22), and then Finish processing.
Through the processing of the non-physical interference posture extraction unit 25 described above, it is possible to extract only postures in which the actual devices 2A do not physically interfere with each other from among the posture set.

Next, an example in which the non-physical interference attitude extraction unit 25 determines the success or failure of trajectory planning will be described.
FIG. 15 is an overhead view of the posture when the robot 30 actually performs picking work.
As shown in FIG. 15, when moving the marker 31 to destination 1, the robot 30 does not physically interfere with any part of the actual device 2A. Therefore, when the trajectory planning is instructed to move the marker 31 to the destination 1, the trajectory planning is successful and the attitude of the successful pattern is obtained. On the other hand, when moving the marker 31 to the destination 2, the robot 30 physically interferes with the environmental installation 33 (wall 33A), so the trajectory planning fails.
Regarding the method of determining the presence or absence of physical interference, for example, when the robot 30 moves according to the trajectory of a trajectory candidate generated by trajectory planning, the three-dimensional position of the robot 30 and the environmental installation 33, etc. The determination can be made based on whether or not they overlap.

FIG. 16 is a side view of the posture when the robot 30 actually performs picking work.
As shown in FIG. 16, when moving the marker 31 to the destination 3, the robot 30 does not physically interfere with any part of the actual device 2A. Therefore, when the trajectory planning is instructed to move the marker 31 to the destination 3, the trajectory planning is successful and the posture is in a successful pattern. On the other hand, when moving the marker 31 to the destination 4, the robot 30 physically interferes with the environmental installation 33 (conveyor 33B), so the trajectory planning fails.

In this embodiment, whether the trajectory planning is successful or unsuccessful is determined based on the presence or absence of physical interference between the robot 30 and the environmental installation 33; however, the present invention is not limited to this, and for example, one of the determination conditions is The determination may be made by the user adding constraints on the joint angles of the robot 30. Specifically, when moving the robot 30 to a destination, if the joint angle of the robot 30 is less than or equal to the upper limit value, the trajectory planning is determined to be successful, and if it exceeds the upper limit value, the trajectory planning is determined to be a failure. Good too.

Next, the processing contents of the non-obstructed posture extraction section 26 will be explained in detail.
FIG. 17 is a flowchart showing the processing procedure of the non-shielding posture extraction unit 26.
The non-obstructed posture extraction section 26 determines whether the marker 31 is occluded between the marker 31 and the imaging section 32 from among the posture set generated by the posture set generation section 23 and extracted by the non-physical interference posture extraction section 25. Extract postures that do not occur (non-occluded postures). Furthermore, the non-obscuring posture extracting section 26 extracts a posture in which the marker 31 is not occluded by utilizing the simulator constructed by the simulator constructing section 24 .

First, the non-obstructed posture extraction section 26 loads the posture set generated by the posture set generation section 23 and extracted by the non-physical interference posture extraction section 25 onto the memory of the control device 1 (step S23).
Next, the non-obstructed posture extraction unit 26 obtains the number N2 of postures included in the above posture set (step S24).
Next, the non-shielding posture extraction unit 26 sets the variable i to an initial value (i=1) (step S25).
Next, the non-shielded posture extraction unit 26 instructs a trajectory plan to move the robot 30 to the i-th posture on the simulator (step S26).
Next, the non-shielding posture extraction unit 26 generates a virtual captured image by the imaging unit 32 on the simulator (step S27).
That is, the non-shielded posture extraction section 26 virtually generates a photographed image obtained by the imaging section 32 when the robot 30 is moved to the i-th posture.

Next, the non-obstructed posture extraction unit 26 estimates the marker-camera conversion matrix M2 by analyzing the generated captured image (step S28). For example, when a pattern of a plurality of black circles with known sizes and positions is printed on the marker 31, the method of analyzing the photographed image is to use the known size and position of the black circles and the above-mentioned virtual photographed image. The marker-camera transformation matrix M2 can be estimated by associating the size and position of the black circles in . Estimating the marker-camera transformation matrix M2 essentially means estimating the position and orientation of the marker 31 in the three-dimensional camera coordinate system C4, that is, the three-dimensional position of the marker 31.

Next, the non-occluded posture extraction unit 26 determines whether the estimation of the marker-camera conversion matrix M2 has been successful (step S29). If the non-occluded pose extraction unit 26 succeeds in estimating the marker-camera transformation matrix M2, the reliability of the estimation is calculated (step S30), and then it determines whether the reliability is greater than or equal to the threshold value. (Step S31). Further, if the reliability of the calculated estimation is greater than or equal to the threshold, the non-occluded posture extraction unit 26 stores the i-th posture as a successful pattern in the memory (step S32), and then generates the i-th posture. The success/failure information for the index of the small space is held (step S33). When retaining success information for the index of a small space, a success flag is stored in association with the index of the small space in which the i-th posture was generated. Furthermore, when retaining failure information for the index of a small space, a failure flag is associated and retained with the index of the small space in which the i-th posture was generated.

On the other hand, if the non-occluded pose extraction unit 26 determines that the estimation of the marker-camera transformation matrix M2 has failed in step S29, it passes the processing in steps S30 to S33 , and in step S31, the reliability is determined to be the threshold value. If it is determined that this is not the case, the processing in steps S32 and S33 is passed. Here, a case where estimation of the marker-camera conversion matrix M2 fails or a case where the reliability of estimation is not equal to or higher than a threshold value corresponds to a case where it is determined that the marker 31 is occluded. On the other hand , if the estimation of the marker-camera conversion matrix M2 is successful or if the reliability of the estimation is equal to or higher than the threshold value, this corresponds to a case where it is determined that the marker 31 is not occluded.

Next, the non-shielding posture extraction unit 26 determines whether the value of the variable i satisfies i=N2 (step S34). Then, if the value of the variable i is less than N2, the non-shielding posture extraction unit 26 increments the value of the variable i by 1 (step S35), and then returns to the process of step S26. Further, when the value of the variable i reaches N2, the non-shielding posture extraction unit 26 saves the posture set of the successful pattern and the success/failure information for each index in the memory of the control device 1 (step S36 ), the series of processing is completed.
Through the processing of the non-occluded posture extraction unit 26 described above, only the postures in which the marker 31 is not occluded between the marker 31 and the imaging unit 32 can be extracted from the posture set.

Next, a reliability calculation method (processing content of step S30) and an evaluation method (processing content of step S31) by the non-shielding posture extraction unit 26 will be explained.
First, in step S28, even if a portion of the marker 31 is hidden by shielding, it may be possible to estimate the marker-camera conversion matrix M2. However, in this case, the accuracy of estimating the marker-camera conversion matrix M2 may be lower than when all of the markers 31 are shown in the photographed image. It is not desirable to use a posture with low estimation accuracy of the marker-camera transformation matrix M2 for calibration.

Therefore, the non-obscured posture extraction section 26 utilizes the fact that the relative posture of the marker 31 and the imaging section 32 in each teaching posture is known on the simulator, and uses the fact that the relative posture of the marker 31 and the imaging section 32 in each teaching posture is known on the simulator. The ratio between the area of the marker 31 at that time and the area of the marker 31 actually appearing on the image taken by the imaging unit 32 on the simulator is calculated as the reliability of the estimation. In addition to this, for example, the ratio between the number of feature points on the marker 31 obtained when the photographed image is analyzed and the total number of feature points is calculated as the reliability. Then, the non-shielding posture extraction unit 26 determines whether the reliability calculated as described above is higher than a threshold value determined in advance based on the type of marker 31.

FIG. 18 is a schematic diagram showing an example of the operation of the non-shielding posture extraction section 26.
A part of the posture set loaded onto the memory of the control device 1 in step S23 is shown within the frame indicated by a dashed line on the left side of FIG. Furthermore, a part of the posture set extracted by the non-occlusion posture extraction unit 26 after occlusion determination is shown within the frame indicated by the dashed-dotted line on the right side of FIG. In this example, the environmental installations 33 are arranged so as to cover the upper, lower, and left sides of the image taken by the imaging unit 32 on the simulator.

First, in the posture shown in the photographed image I1 of FIG. 18, a portion of the marker 31 is blocked by the robot 30, and there is a possibility that estimation of the position and posture of the marker 31 will fail. If it is possible to estimate the position and orientation of the marker 31 from a part of the marker 31 shown in the photographed image I1, the non-occluded orientation extracting unit 26 detects that the marker 31 is photographed without being occluded, for example, as described above. The ratio between the area of the marker 31 when it appears in the image and the area of the marker 31 that actually appears on the image taken by the imaging unit 32 on the simulator is calculated as the reliability of the estimation. In calculating this reliability, the camera coordinate system C4 and the position and orientation of the marker 31 in a certain posture of the robot 30 are known by the posture set generation unit 23. Therefore, by using information on the position and orientation of the marker 31 and information on the angle of view, resolution, camera parameters, etc. of the imaging unit 32, pixels of the marker 31 on the image when the marker 31 appears in the image without being obstructed. A set of positions (hereinafter referred to as a "pixel set") can be specified.

Next, the non-shielding posture extraction unit 26 uses the simulator to change the color information of the robot 30 and the environmental installations 33 to a color that is not included in the marker 31, for example, a color with an RGB value of (255, 0, 0). , the posture of the robot 30 is changed, and an image captured by the imaging unit 32 is virtually generated. The position of the marker 31 in the photographed image matches the pixel set, and the color of the pixel where the shielding occurs is (255, 0, 0). Therefore, the non-occluded posture extraction unit 26 calculates the percentage of pixels that are not of the color (255, 0, 0) among the pixel set as the estimation reliability. Therefore, in the case of the photographed image I1, since about half of the marker 31 is blocked by the robot 30, the estimated reliability is calculated to be about 50%. Furthermore, if the threshold value with which the reliability is compared is set to 90% in advance, the reliability in the captured image I1 will be less than the threshold.

On the other hand, in the posture shown in the photographed image I2 of FIG. 18, the marker 31 is not shielded, and the entire marker 31 is visible on the image. Therefore, in the case of the photographed image I2, the position and orientation of the marker 31 can be estimated, and the reliability of the estimation is also higher than the threshold value, so it is a successful pattern.
Furthermore, in the posture shown in the captured image I3 of FIG. 18, the marker 31 is placed within the field of view of the imaging unit 32, but a portion of the marker 31 is blocked by the environmental installation 33. Therefore, estimation of the position and orientation of the marker 31 may fail. Further, since nine of the total number of twelve black circles on the marker 31 are shown in the photographed image I3, if the position and orientation of the marker 31 can be estimated, the degree of reliability is calculated. At this time, the non-occluded posture extraction unit 26 uses the ratio of the number of feature points on the marker 31 obtained when analyzing the photographed image to the total number of feature points as the estimation reliability, for example, as described above. calculate. When the center of the black circle is set as a feature point, the number of feature points obtained by image analysis is 9 out of a total of 12 feature points, and the reliability of estimation is calculated to be 75%. Furthermore, if the threshold value with which the reliability is compared is set to 90% in advance, the reliability in the captured image I3 will be less than the threshold.

Through the processing of the non-shielding posture extraction unit 26 described above, the presence or absence of shielding of the marker 31 is determined, and a set of postures for which it is determined that the marker 31 is not shielded (the postures shown in the photographed images I2, I4, I5, etc.) is generated. As a result, only the posture in which the marker 31 appears on the right and center portions of the photographed image is generated.
In addition, the non-obstructed posture extraction section 26 virtually generates captured images obtained by the imaging section 32 when the robot 30 is moved to each posture (postures corresponding to variables i=1 to N2), and The three-dimensional position of the marker 31 is estimated by analyzing the generated captured image, and whether or not the marker 31 is occluded is determined based on whether the estimation is successful. Specifically, if the estimation is successful, it is determined that the marker 31 is not occluded, and if the estimation is unsuccessful, it is determined that the marker 31 is occluded. Thereby, a posture in which the marker 31 is not occluded can be extracted.
Furthermore, when the three-dimensional position of the marker 31 is successfully estimated, the non-occluded posture extraction unit 26 determines the reliability of the estimation, and determines whether or not the marker 31 is obstructed based on whether the reliability is equal to or higher than a threshold value. do. Specifically, if the reliability of the estimation is greater than or equal to the threshold, it is determined that the marker 31 is not occluded, and if the reliability of the estimation is less than the threshold, it is determined that the marker 31 is occluded. Thereby, among the postures for which the three-dimensional position of the marker 31 has been successfully estimated, only those postures for which the reliability of estimation is higher than the threshold value can be extracted as postures in which the marker 31 is not occluded.

Note that in the present embodiment, the non-obscured posture extraction unit 26 calculates the reliability of the estimation, but depending on the shape or pattern of the marker 31, the reliability of the estimation may not be calculated. That is, the reliability of the estimation may be calculated as necessary. Further, calculation of the reliability is not limited to the above-mentioned area ratio and feature point ratio. For example, when estimating the position and orientation of the marker 31 using machine learning or the like, the reliability of the estimation itself may be estimated.

Next, the processing contents of the posture set evaluation section 27 will be explained in detail.
The posture set evaluation section 27 uses the posture set generated by the posture set generation section 23 and extracted by the non-physical interference posture extraction section 25 and the non-shielding posture extraction section 26 to calibrate with high accuracy above a predetermined value. Determine whether or not you can expect a In addition, if the pose set evaluation unit 27 can expect a highly accurate calibration equal to or higher than a predetermined value, the pose set evaluation unit 27 instructs the calibration execution unit 29 to execute the calibration. In addition, if the pose set evaluation unit 27 cannot expect a highly accurate calibration higher than a predetermined value, the pose set generation unit 23 instructs the pose set generation unit 23 to add a pose set and/or changes the value of the generation parameter. Then, an instruction is given to generate the pose set again.

FIG. 19 is a functional block diagram of the posture set evaluation section 27.
As shown in FIG. 19, the posture set evaluation section 27 includes a posture number evaluation section 271, a posture set extraction section 272, a calibration evaluation section 273, and an index evaluation section 274.

The pose number evaluation unit 271 evaluates the number of poses included in the pose set generated by the pose set generation unit 23 and extracted by the non-physical interference pose extraction unit 25 and the non-occlusion pose extraction unit 26. Specifically, the posture number evaluation unit 271 determines whether the number of postures included in the posture set is within a predetermined number set in advance.
When the posture number evaluation section 271 determines that the number of postures included in the posture set is not within a predetermined number, the posture set extraction section 272 extracts some of the postures from among the plurality of postures included in the posture set. Extract the subset that is . For example, if the posture set includes a total of 100 postures, 20 postures out of the 100 postures are extracted as a subset.

The calibration evaluation unit 273 uses the above posture set to virtually generate data such as a marker/camera transformation matrix M2 and an arm/base transformation matrix M3 used for calibration on the simulator, and performs estimation in the calibration. The estimation accuracy of the marker/camera transformation matrix M2 is evaluated. That is, the calibration evaluation unit 273 evaluates the accuracy of calibration.
When the calibration evaluation section 273 determines that a calibration accuracy higher than a predetermined value cannot be expected due to, for example, a shortage of the number of postures, the index evaluation section 274 adds or adds a posture set to the posture set generation section 23. Instruct to generate again.

The processing contents of each part of the posture set evaluation section 27 will be explained in detail below.
First, when the number of postures included in the posture set is greater than a threshold determined by the prior knowledge 21 or the like, the posture number evaluation unit 271 sets the posture set so as to extract some postures, that is, a subset, from the posture set. The extraction unit 272 is instructed. Further, the posture number evaluation section 271 causes the calibration evaluation section 273 to determine whether or not high calibration accuracy of a predetermined value or higher can be expected in the subset extracted by the posture set extraction section 272. In this way, by extracting some of the poses from the pose set as a subset, when the number of poses included in the pose set is large, the time required for calibration can be suppressed and the calibration can be performed efficiently. Accuracy can be evaluated.

The posture set extraction unit 272 extracts some postures from a posture set, which is a set of a plurality of postures, based on predetermined rules. For example, there is a method of extracting rules such that the indexes assigned to the small spaces in which each pose is generated appear on average. To give a specific example, when the upper limit number of postures to be extracted is set to 21, the posture set extraction unit 272 extracts some postures based on the following rules.

First, among the postures (teaching postures) corresponding to a total of 27 small spaces obtained by dividing the teaching space T1 as shown in FIG. , 7 small space poses with an index of Z = 2, and 7 small space poses with an index of Z = 3, that is, a total of 23 poses are the non-physical interference poses of the previous stage. When extracted by the extracting unit 25 and the non-occluded pose extracting unit 26, the pose of the small space with the index of Z=1, which has the largest number of extracted poses, is reduced by two. This makes it possible to extract some postures so that the index appears on average.

The calibration evaluation unit 273 generates a marker/camera transformation matrix M2 and an arm/base transformation matrix M3 to be used when performing virtual calibration on the simulator, and then changes the base/camera transformation matrix M1 through optimization processing (described later). Find an estimate. Furthermore, the calibration evaluation unit 273 compares the estimated value of the base/camera transformation matrix M1 with the correct value at the time of designing the simulator, and determines the calibration accuracy necessary for executing the robot work, that is, the calibration accuracy higher than a predetermined value. Determine whether or not it appears. The method for obtaining the estimated value of the base/camera transformation matrix M1 will be explained later along with the processing content of the calibration execution unit 29.

The calibration evaluation unit 273 may calculate the estimated value of the base/camera transformation matrix M1 by adding an error to the generated marker/camera transformation matrix M2 and arm/base transformation matrix M3. The reason is as follows.
When actually performing calibration with the actual device 2A, errors due to noise etc. are added to the observed data. Therefore, as described above, by adding errors to the marker/camera transformation matrix M2 and the arm/base transformation matrix M3, calibration evaluation can be performed under conditions close to those of the actual machine.

In that case, the error added to the marker/camera transformation matrix M2 and the arm/base transformation matrix M3 may be, for example, an error that follows a normal distribution. The standard deviation of the normal distribution at this time may be specified by the prior knowledge 21 or the like, or may be specified by a value having a constant ratio to the values of the marker/camera transformation matrix M2 and the arm/base transformation matrix M3.

Note that the noise in the observation data is, for example, noise caused by errors in the robot mechanism when the observation data is an encoder value of the robot 30, and signal noise when the observation data is an image taken by the imaging unit 32. and noise caused by image distortion. Further, the observation data may be the position and orientation of the marker 31 obtained by analyzing an image captured by the imaging unit 32.

If the calibration evaluation section 273 determines that a highly accurate calibration higher than a predetermined value can be expected, the calibration evaluation section 273 instructs the calibration execution section 29 to execute the calibration. The calibration execution unit 29 estimates the base/camera transformation matrix M1 by executing calibration in response to instructions from the calibration evaluation unit 273. Further, if the calibration evaluation section 273 determines that a highly accurate calibration higher than a predetermined value cannot be expected, the calibration evaluation section 273 instructs the index evaluation section 274 to perform index evaluation.

When the index evaluation section 274 receives an instruction to perform index evaluation from the calibration evaluation section 273 (when a highly accurate calibration higher than a predetermined value cannot be expected), the index evaluation section 274 instructs the posture set generation section 23 to generate an additional posture. and/or instructs the parameter update unit 28 to change the value of the generation parameter 22. By performing such feedback, the posture set generation unit 23 is able to generate a posture set again, and also generates a posture set including additional postures or a posture set based on the updated generation parameters 22. It becomes possible to generate . Thereby, when the posture set generation unit 23 generates the posture set again, it is possible to increase the possibility that highly accurate calibration exceeding a predetermined value can be expected.

The index evaluation unit 274 acquires the pose set evaluated by the calibration evaluation unit 273, and if the number of poses included in the pose set is less than a predetermined threshold, the index evaluation unit 274 obtains the index (X , Y, Z), the parameter updating unit 28 is instructed to increment the index that appears least frequently when searching the entire pose set. Alternatively, if the number of postures included in the acquired posture set is less than a predetermined threshold , the index evaluation section 274 instructs the posture set generation section 23 to generate additional postures. An example of a generation method is to randomly extract multiple postures from the acquired posture set and generate a taught position and taught posture in a small space corresponding to the index of that posture. Not limited to.

Through the processing of the posture set evaluation section 27 described above, the evaluation results of the posture set can be fed back to the posture set generation section 23 and the parameter update section 28. Further, the calibration execution unit 29 can be caused to execute the calibration only when a highly accurate calibration higher than a predetermined value can be expected. Therefore, it is possible to suppress deterioration in calibration accuracy due to insufficient number of postures or the like. Furthermore, if a highly accurate calibration higher than a predetermined value cannot be expected, the posture set generation unit 23 can be caused to generate the posture set again.

Note that in the present embodiment, the posture set evaluation section 27 has a posture number evaluation section 271 and a posture set extraction section 272, but is not limited thereto, and has only a calibration evaluation section 273 and an index evaluation section 274. It can also be a configuration. Further, the functions of the calibration evaluation section 273 and the index evaluation section 274 may be integrated into one functional section.
In addition, in the present embodiment, as a rule applied to the processing of the posture set extracting unit 272, a rule has been exemplified in which the index assigned to each small space appears on average, but the rule is not limited to this, and for example, Some postures may be randomly extracted.

The parameter update unit 28 increments the values of parameters X1, Y1, and Z1 that determine the resolution of the pose set based on instructions from the pose set evaluation unit 27. As a result, the generation parameters 22 are updated. Therefore, the posture set generation unit 23 generates a posture set using the updated generation parameters 22.

FIG. 20 is a flowchart showing the processing procedure of the calibration execution unit 29.
The calibration execution unit 29 controls the robot 30 and the imaging unit 32 included in the actual device 2A, and calculates the marker/camera conversion matrix M2 and the arm/base conversion matrix M3 when the robot 30 is moved to each posture of the posture set. and get. Furthermore, the calibration execution unit 29 estimates the base/camera transformation matrix M1 based on the coordinate transformation parameters of the acquired marker/camera transformation matrix M2 and arm/base transformation matrix M3.

First, the calibration execution section 29 loads onto the memory of the control device 1 the posture set for which the posture set evaluation section 27 has determined (evaluated) that a highly accurate calibration of a predetermined value or higher can be expected (step S37).
Next, the calibration execution unit 29 obtains the number N3 of postures included in the above posture set (step S38).
Next, the calibration execution unit 29 sets the variable i to an initial value (i=1) (step S39).
Next, the calibration execution unit 29 actually moves the robot 30 to the i-th posture in the actual device 2A (step S40).
Next, the calibration execution unit 29 photographs the work space of the robot 30 using the imaging unit 32 (step S41).
Next, the calibration execution unit 29 estimates the marker-camera conversion matrix M2 by analyzing the captured image obtained by the imaging unit 32 (step S42).

Next, the calibration execution unit 29 determines whether the estimation of the marker-camera conversion matrix M2 has been successful (step S43). In this step S43, there is a possibility that the estimation of the marker-camera conversion matrix M2 will fail due to the actual lighting environment in the actual device 2A or the specification/positioning deviation from the design value of the imaging unit 32 included in the actual device 2A. be.

Next, if the estimation of the marker/camera transformation matrix M2 is successful, the calibration execution unit 29 calculates the arm/base transformation matrix M3 indicating the orientation value of the i-th robot 30 and the orientation of the i-th robot 30. A marker-camera conversion matrix M2 indicating the position and orientation of the corresponding marker 31 is stored in the memory of the control device 1 (step S44).

Next, the calibration execution unit 29 determines whether the value of the variable i satisfies i=N3 (step S45). Then, when the value of the variable i is less than N3, the calibration execution unit 29 increments the value of the variable i by 1 (step S46), and then returns to the process of step S40.

On the other hand, if the calibration execution unit 29 determines in step S43 that estimation of the marker-camera conversion matrix M2 has failed, it passes the process in step S44.

Thereafter, when the value of the variable i reaches N3, the calibration execution unit 29 solves the optimization problem from the data on the memory, estimates the base/camera transformation matrix M1 (step S47), and then performs a series of Finish processing.
Solving the optimization problem in step S47 corresponds to optimization processing. As a method for estimating the base/camera transformation matrix M1 through this optimization process, the known techniques shown in the following references can be used.

(References)
"Simultaneous Robot-World and Hand-Eye Calibration" F. Dornaika, R. Horaud, Aug1998.
An overview of the known technology disclosed in this reference is as follows.
Assuming that the coordinate transformation parameter (transformation matrix) between the robot base and the arm tip is AA, and the coordinate transformation parameter between the arm tip and marker is BB, the coordinate transformation from the robot base to the marker is AA×BB. On the other hand, if CC is the coordinate transformation parameter between the robot base and the camera, and DD is the coordinate transformation parameter between the camera and the marker, then the coordinate transformation from the robot base to the marker is CC×DD. In other words, the condition AA×BB=CC×DD is satisfied. Here, the coordinate transformation parameter AA is a known parameter based on the robot encoder value, the coordinate transformation parameters BB and CC are unknown parameters, and the coordinate transformation parameter DD is a known parameter based on image analysis. Therefore, in the optimization process, coordinate transformation parameters BB and CC that satisfy the condition AA×BB=CC×DD are estimated by numerical calculation from a large number of coordinate transformation parameters AA and DD, each of which is a known parameter.
Through the processing of the calibration execution unit 29 described above, it is possible to estimate the base/camera transformation matrix M1 in the actual device 2A.

Further, according to the calibration device 2 according to the present embodiment, the posture set generation section 23 generates a posture set of the robot 30 to enable the imaging section 32 to image (photograph) the marker 31, and The non-shielding posture extracting unit 26 extracts postures in which the marker 31 is not occluded. Therefore, a posture set in which the marker 31 is not occluded can be automatically generated. Therefore, it is possible to teach the set of postures of the robot 30 in consideration of the shielding of the marker 31, and to reduce the number of man-hours required for teaching the set of postures of the robot 30.

Further, according to the calibration device 2 according to the present embodiment, the non-physical interference posture extraction section 25 extracts postures in which the robot 30 and the marker 31 do not physically interfere from the posture set generated by the posture set generation section 23. . Therefore, it is possible to automatically generate a posture set in which the robot 30 and the marker 31 do not physically interfere with each other. Therefore, it is possible to teach the set of postures of the robot 30 in consideration of physical interference in the actual device 2A, and to reduce the number of man-hours required for teaching the set of postures of the robot 30.

In the above embodiment, the non-physical interference posture extraction section 25 extracts the non-physical interference postures from the posture set generated by the posture set generation section 23, and the non-physical interference posture extraction section 25 extracts the non-physical interference postures from the posture set of the extracted non-physical interference postures. Although the non-shielding posture extraction unit 26 is configured to extract non-shielding postures, the present invention is not limited to this. For example, the non-shielding pose extraction unit 26 extracts non-shielding poses from the pose set generated by the pose set generation unit 23, and the non-physical interference pose extraction unit 25 extracts non-shielding poses from the extracted pose set of non-shielding poses. The configuration may be such that a physical interference posture is extracted. Further, the non-physical interference pose extraction unit 25 extracts non-physical interference poses from the pose set generated by the pose set generation unit 23, and the non-obstructed pose extraction unit 26 extracts non-physical interference poses from the pose set generated by the pose set generation unit 23. A configuration in which a non-physical interference pose is extracted from among the poses, and a pose set evaluation unit 27 evaluates common poses among the pose sets respectively extracted by the non-physical interference pose extraction unit 25 and the non-obstructed pose extraction unit 26. It may be. Furthermore, of the non-physical interference posture extraction section 25 and the non-shielding posture extraction section 26, only the non-shielding posture extraction section 26 may be included. The above points also apply to the second embodiment described later.

<Second embodiment>
FIG. 21 is a diagram illustrating a configuration example of a calibration device for robot control according to the second embodiment.
As shown in FIG. 21, the calibration device 2-1 includes a control device 1, a real device 2C, and a calibration control device 2D.

The actual device 2C includes a robot 30 consisting of a robot arm, a marker 31 attached to the robot 30, and an imaging unit 32-1 that measures the work space and surroundings of the robot 30. The configuration of the actual device 2C other than the imaging section 32-1 is the same as that of the first embodiment. The imaging unit 32-1 includes a plurality of (two in the illustrated example) imaging devices 32A and 32B. Each of the imaging devices 32A and 32B is configured by, for example, a monocular camera, a ToF (Time of Flight) camera, a stereo camera, or the like.

The calibration control device 2D includes a priori knowledge 21-1, a generation parameter 22-1, a posture set generation section 23-1, a simulator construction section 24-1, a non-physical interference posture extraction section 25-1, and a non-physical interference posture extraction section 25-1. In addition to the shielding posture extraction section 26-1, the posture set evaluation section 27-1, the parameter updating section 28-1, and the calibration execution section 29-1, a common posture extraction section 201 is provided. The configuration other than the common posture extraction unit 201 is basically the same as in the first embodiment.

However, in this embodiment, the imaging section 32-1 is configured by a plurality of imaging devices 32A and 32B. Therefore, the prior knowledge 21-1, the generation parameters 22-1, the pose set generator 23-1, the simulator constructor 24-1, the non-physical interference pose extractor 25-1, and the non-occluded pose extractor 26-1, the pose set evaluation unit 27-1, the parameter update unit 28-1, and the calibration execution unit 29-1 perform the same operations as in the first embodiment for the plurality of imaging devices 32A and 32B, respectively. Performs similar processing operations. For example, the posture set generation unit 23-1 generates a posture set of the robot 30 to enable the imaging device 32A to photograph the marker 31, and a posture set of the robot 30 to enable the imaging device 32B to photograph the marker 31. are generated individually. Individual processing for each of the imaging devices 32A and 32B is performed not only by the pose set generation unit 23-1 but also by the simulator construction unit 24-1, the non-physical interference pose extraction unit 25-1, the non-obstructed pose extraction unit 26-1, and the pose set generation unit 23-1. The evaluation unit 27-1, the parameter update unit 28-1, and the calibration execution unit 29-1 also perform this process.

The common posture extraction section 201 is generated by the posture set generation section 23-1 for each of the imaging devices 32A, 32B, and the non-physical interference posture extraction section 25-1 and the non-occluded posture extraction section 26-6 generate the common posture extraction section 201 for each of the imaging devices 32A, 32B. A plurality of postures that can be commonly used by the plurality of imaging devices 32A and 32B are extracted from the posture set extracted for each 32B. Further, the common posture extraction unit 201 changes the posture set of each imaging device 32A, 32B so that the plurality of extracted postures remains. This will be explained in detail below.

The common posture extraction unit 201 determines whether each posture included in the posture set generated corresponding to a certain imaging device 32A is applicable to another imaging device 32B, and performs similar determination processing. This is performed for all combinations of imaging devices 32A and 32B. For example, whether or not each posture included in a posture set generated corresponding to a certain imaging device 32A is applicable to another imaging device 32B can be determined based on the simulator constructed by the simulator construction unit 24-1. This can be determined based on whether the following conditions (1) to (3) are satisfied when the robot 30 is controlled and the marker 31 is moved to each posture.
(1) The marker 31 is included in the teaching space of the imaging device 32B.
(2) The rotation matrices of the marker 31 and the imaging device 32B are within a preset threshold.
(3) The function of the non-occlusion posture extraction unit 26-1 can determine that occlusion does not occur.

In the above determination result, the postures that can be commonly used by the plurality of imaging devices 32A, 32B (hereinafter also referred to as "common postures") constitute a posture set of the imaging devices 32A, 32B to which the postures can be applied. Add it as one of the postures. As a result of this process, if the predetermined upper limit of the number of poses is exceeded, the poses that can only be used by either of the imaging devices (32A or 32B) are removed from the pose set of the imaging devices 32A and 32B, so that the calibration can be corrected. The number of postures to be applied to the calibration by the simulation execution unit 29 is adjusted (reduced). Regarding the priority when removing postures, for example, there is a method of preferentially removing postures located in a small space similar to the small space in which the added posture is located.

According to the calibration device 2-1 according to the present embodiment, in addition to the same effects as in the first embodiment, the following effects can be obtained.
In this embodiment, the common posture extracting unit 201 extracts a plurality of postures that can be commonly used by the plurality of imaging devices 32A and 32B, and the calibration execution unit 29-1 performs calibration by applying the extracted plurality of postures. Execute the option. Therefore, even when the imaging unit 32 is configured with a plurality of imaging devices 32A and 32B, it is possible to perform calibration while reducing the number of steps required for teaching and calibrating the posture set of the robot 30.

In this embodiment, the common posture extraction section 201 is generated by the posture set generation section 23-1 for each imaging device 32A, 32B, and is generated by the non-physical interference posture extraction section 25-1 and the non-occlusion posture extraction section 26. -6 is configured to extract a common posture from among the posture sets extracted for each of the imaging devices 32A and 32B, but the configuration is not limited thereto. For example, the common posture extracting unit 201 may extract a common posture from among the posture sets generated for each of the imaging devices 32A and 32B by the posture set generating unit 23-1. Further, the common posture extraction section 201 may extract a common posture from among the posture sets extracted for each of the imaging devices 32A and 32B by the non-physical interference posture extraction section 25-1. Further, the common posture extraction section 201 may extract a common posture from among the posture sets extracted for each of the imaging devices 32A and 32B by the non-obscured posture extraction section 26-6.

Further, in the present embodiment, the common posture extraction unit 201 controls the robot 30 on the simulator and determines commonly usable postures. The determination may be made based only on the taught position and orientation of the marker 31. Further, by increasing the number of postures during calibration without setting an upper limit on the number of postures included in the posture set of each imaging device 32A, 32B, improvement in accuracy may be expected. In that case, there is no need to set an upper limit on the number of postures in advance.

<Modified examples, etc.>
Note that the present invention is not limited to the embodiments described above, and includes various modifications. For example, in the embodiments described above, the content of the present invention is explained in detail to make it easier to understand, but the present invention is not necessarily limited to having all the configurations described in the embodiments described above. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment. Furthermore, it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to delete some of the configurations of each embodiment, add other configurations, or replace them with other configurations.

2, 2-1... Calibration device, 2A, 2C... Actual device, 23, 23-1... Posture set generation unit, 25, 25-1... Non-physical interference attitude extraction unit, 26, 26-1... Non-shielding attitude Extraction unit, 27, 27-1... Posture set evaluation unit, 28, 28-1... Parameter update unit, 29, 29-1... Calibration execution unit, 30... Robot (robot arm), 31... Marker, 32... Imaging 32A, 32B...Imaging device, 33...Environmental installation object, 201...Common attitude extraction unit, M1...Base/camera transformation matrix (coordinate transformation parameter)

Claims (8)

an imaging unit that captures an image from a predetermined position;
a marker attached to a robot and displaced according to the movement of the robot;
a posture set generation unit that generates a posture set including a plurality of postures of the robot to enable the imaging unit to image the marker;
a non-obscuring posture extracting unit that extracts a plurality of postures in which the marker is not occluded from the posture set generated by the posture set generating unit;
a calibration execution unit that estimates a first coordinate conversion parameter for converting the coordinate system of the robot and the coordinate system of the imaging unit based on the plurality of postures extracted by the non-obstructed posture extraction unit;
Equipped with
The calibration execution unit is configured to move the robot to each of the plurality of postures extracted by the non-obstructed posture extraction unit, photograph the working space of the robot using the imaging unit, and perform image capture by photographing the robot. Estimating a second coordinate transformation parameter for transforming the coordinate system of the marker and the coordinate system of the imaging unit in the posture by analyzing the obtained captured image,
The calibration execution unit further estimates the first coordinate transformation parameter by performing numerical calculation using the second coordinate transformation parameter in each of the plurality of orientations.
Calibration device for robot control. The imaging unit is configured by a plurality of imaging devices,
The posture set generation unit individually generates a posture set of the robot for each of the plurality of imaging devices to enable the imaging device to image the marker,
further comprising a common posture extraction section that extracts a plurality of postures that can be commonly used by the plurality of imaging devices from among the posture sets of the robot that are individually generated for each of the imaging devices by the posture set generation section;
The calibration device for robot control according to claim 1, wherein the calibration execution section estimates the first coordinate transformation parameter by applying a plurality of postures extracted by the common posture extraction section. A plurality of postures in which the robot and the marker do not physically interfere with any part of the actual device including the robot and environmental objects are extracted from the posture set generated by the posture set generation unit. Furthermore, it is equipped with a physical interference pose extraction section,
The robot control according to claim 1, wherein the non-obstructed posture extraction section extracts a posture in which the marker is not occluded from a posture set including the plurality of postures extracted by the non-physical interference posture extraction section. Calibration device for. further comprising a posture set evaluation unit that determines whether or not a highly accurate calibration of a predetermined value or higher can be expected using the posture set including the plurality of postures extracted by the non-obscured posture extraction unit;
2. The robot control system according to claim 1 , wherein the calibration execution unit estimates the first coordinate transformation parameter when the posture set evaluation unit determines that highly accurate calibration greater than or equal to the predetermined value is expected. calibration equipment. 5. The robot control system according to claim 4, wherein the posture set evaluation section instructs the posture set generation section to additionally generate postures when a highly accurate calibration higher than the predetermined value cannot be expected. Calibration device. The apparatus further includes a parameter updating section that changes the value of a parameter used by the posture set generation section to generate the posture set when the posture set evaluation section determines that highly accurate calibration higher than the predetermined value is not expected. The calibration device for robot control according to claim 4. The non-obstructed posture extraction section virtually generates a photographed image obtained by the imaging section when the robot is moved to each posture, and analyzes the generated photographed image to obtain a three-dimensional image of the marker. The calibration device for robot control according to claim 1, wherein the original position is estimated, and whether or not the marker is occluded is determined based on whether or not the estimation is successful. 8. The non-occluded posture extracting unit determines the reliability of the estimation when the estimation is successful, and determines whether or not the marker is occluded based on whether the reliability is equal to or higher than a threshold value. Calibration device for robot control.

Images