WO2019239848A1

WO2019239848A1 - Robot control system

Info

Publication number: WO2019239848A1
Application number: PCT/JP2019/020623
Authority: WO
Inventors: 中塚　均; 豊男飯田
Original assignee: オムロン株式会社
Priority date: 2018-06-15
Filing date: 2019-05-24
Publication date: 2019-12-19
Also published as: JP7070127B2; JP2019217571A

Abstract

This robot control system includes: a measurement unit that, on the basis of an image captured by an imaging unit, measures three-dimensional coordinates of a given object present in the field of view of the imaging unit; a command generating unit that, in accordance with a previously calculated correspondence between the measured three-dimensional coordinates and the position and attitude of an action part of the robot, generates a command for positioning the action part of the robot; a calibration execution unit that executes a calibration for calculating the correspondence; and a setting receiving unit that, during the calibration, receives a setting for a calibration region, which is a region wherein a reference object associated with the action part of the robot is to be disposed.

Description

Robot control system

The present technology relates to a robot control system for controlling a robot using three-dimensional coordinates calculated based on an image captured by an imaging unit.

Conventionally, a system in which a sensor capable of three-dimensional measurement and a robot are combined is known. In such a system, there are many setting items, and it may be difficult to intuitively grasp the set contents. Therefore, for example, Japanese Patent Application Laid-Open No. 2011-112402 (Patent Document 1) is configured to improve the convenience of the three-dimensional visual sensor by easily displaying the three-dimensional shape of the set effective area together with the surroundings. Is disclosed.

JP 2011-112402 A

The configuration disclosed in Patent Document 1 is a coordinate system for controlling a robot, although calibration between a three-dimensional coordinate system (measurement coordinate system) for stereo measurement and a camera coordinate system is considered. Calibration for is not considered.

Actually, it is necessary to calibrate the relationship between the three-dimensional coordinates measured from the captured image and the position and posture of the robot in advance. There is a need to facilitate the setting for executing such calibration.

A robot control system according to an embodiment of the present technology is present in a field of view of an imaging unit arranged to include an action unit of the robot in the field of view and an image captured by the imaging unit. A measuring unit that measures the three-dimensional coordinates of an arbitrary object to be positioned, and positions the robot operating unit according to a pre-calculated correspondence between the measured three-dimensional coordinates and the position and orientation of the robot operating unit. A region for placing a reference object associated with a robot action unit in a calibration, a command generation unit for generating a command for a calibration, a calibration execution unit for executing a calibration for calculating a correspondence relationship A setting receiving unit that receives the setting of the calibration area.

According to this disclosure, since calibration can be executed in the range of the calibration area set via the setting reception unit, calibration can be made more efficient.

In the above disclosure, the setting reception unit may indicate the range of the calibration area that is set with the imaging unit as a reference.

According to this disclosure, since the calibration area is set based on the imaging unit, the user can easily understand the range of the calibration area.

In the above disclosure, the setting reception unit may display the field of view range of the imaging unit.

According to this disclosure, the range in which the calibration area is set can be grasped at a glance with respect to the visual field range of the imaging unit.

In the above disclosure, the calibration area may be set as a rectangular parallelepiped based on the optical axis of the imaging unit.

According to this disclosure, the position where the reference object is arranged can be easily determined, and the operation range can be determined so as not to be out of the measurement range even when moved in the optical axis direction.

In the above disclosure, the size of the cross section of the rectangular parallelepiped set as the calibration area may be determined according to the field of view of the imaging unit and the distance from the imaging unit to the end face of the calibration area.

According to this disclosure, it is possible to set an optimal calibration area according to the distance from the imaging unit.

In the above disclosure, the setting reception unit may receive a range setting on the optical axis of the imaging unit as the calibration area setting.

According to this disclosure, only the area actually used in the optical axis direction of the imaging unit can be set as the calibration area.

In the above disclosure, the calibration execution unit sequentially gives commands to the robot, and obtains when the reference control unit sequentially arranges the reference objects in the calibration area, and when the reference objects are sequentially arranged in the calibration area. And a calculation unit that calculates a correspondence relationship based on a set of the three-dimensional coordinates of the reference object and the position and orientation of the action unit of the robot.

According to this disclosure, since the user performs the calibration only by setting the calibration area, the robot control system can be operated even by a user who has no specialized knowledge.

In the above disclosure, the calibration execution unit includes a position determination unit that determines a plurality of positions where the reference object is to be arranged in the calibration area, and a relationship between one of the determined positions and the current position of the reference object Based on the combination of the position display unit indicating the reference object and the three-dimensional coordinates of the reference object and the position and orientation of the action part of the robot, which are obtained when the reference object is sequentially arranged at a plurality of determined positions. And a calculation unit for calculating the relationship.

According to this disclosure, the user can set the calibration area and perform the calibration simply by sequentially arranging the reference objects at predetermined positions in accordance with the notification from the position display unit. Even if it exists, the robot control system can be operated.

According to the present technology, it is possible to facilitate the setting required when previously calibrating the relationship between the three-dimensional coordinates measured from the captured image and the position and orientation of the robot.

It is a schematic diagram which shows the example of application of the robot control system according to this Embodiment. It is a schematic diagram which shows the whole structure of the robot control system according to this Embodiment. It is a schematic diagram which shows the structural example of the 3D camera contained in the robot control system according to this Embodiment. It is a schematic diagram which shows the structural example of the image measuring device contained in the robot control system according to this Embodiment. It is a schematic diagram which shows the structural example of the control apparatus contained in the robot control system according to this Embodiment. It is a schematic diagram which shows the structural example of the robot control apparatus contained in the robot control system according to this Embodiment. It is a figure for demonstrating one side of the camera robot calibration in the robot control system according to this Embodiment. It is a schematic diagram which shows an example of the user interface screen which the robot control system according to this Embodiment provides. It is a sequence diagram which shows an example of the automatic processing procedure of the camera robot calibration in the robot control system according to the present embodiment. It is a schematic diagram which shows an example of the user interface screen provided in the automatic process of the camera and robot calibration in the robot control system according to the present embodiment. It is a sequence diagram which shows an example of the manual processing procedure of the camera robot calibration in the robot control system according to the present embodiment. It is a schematic diagram which shows an example of the user interface screen provided in the manual process of the camera robot calibration in the robot control system according to the present embodiment.

Embodiments of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the drawings are denoted by the same reference numerals and description thereof will not be repeated.

<A. Application example>
First, an example of a scene to which the present invention is applied will be described.

FIG. 1 is a schematic diagram showing an application example of the robot control system 1 according to the present embodiment. Referring to FIG. 1, robot control system 1 according to the present embodiment includes imaging unit 50 arranged to include action unit 8 of robot 2 in the field of view as a configuration for controlling robot 2. The robot control system 1 further includes a measurement unit 52 that measures the three-dimensional coordinates of an arbitrary object existing in the field of view of the imaging unit 50 based on the image captured by the imaging unit 50. The three-dimensional coordinates measured by the measurement unit 52 are output to the command generation unit 54 and the calibration execution unit 58. The three-dimensional coordinates measured by the measurement unit 52 can include a position (a position on each axis) and a posture (a rotation angle about each axis) on a three-dimensional coordinate of a specific object.

The command generation unit 54 generates a command for positioning the action unit 8 of the robot 2. Specifically, the command generation unit 54 compares the designated target position with the three-dimensional coordinates from the measurement unit 52, and calculates the correspondence calculated in advance between the position and orientation of the action unit 8 of the robot 2. A command is calculated according to the relationship (typically, calibration parameter 56).

The calibration parameter 56 is determined by the calibration execution unit 58. More specifically, the calibration execution unit 58 sets the calibration parameter 56 based on the three-dimensional coordinates from the measurement unit 52 and the setting of the spatial region (calibration region) used for calibration from the setting reception unit 60. Perform calibration to calculate. This calibration area is set by the user by the setting receiving unit 60. That is, the setting receiving unit 60 receives a setting of a calibration region that is a region in which a reference object (for example, the reference plate 20 is shown) associated with the action unit 8 of the robot 2 is to be arranged. As the calibration area, an arbitrary area of the effective visual field of the imaging unit 50 can be set.

As described above, in the present embodiment, since an arbitrary calibration area based on the imaging unit 50 can be set, the calibration for calculating the calibration parameter 56 can be executed efficiently.

<B. Overall Configuration of Robot Control System 1>
First, the overall configuration of robot control system 1 according to the present embodiment will be described.

FIG. 2 is a schematic diagram showing the overall configuration of the robot control system 1 according to the present embodiment. Referring to FIG. 2, the robot control system 1 includes a robot existing in the field of view of the imaging unit and an arbitrary image based on an image captured by the imaging unit arranged to include the action unit of the robot in the field of view. The three-dimensional coordinates of the object are measured, and the robot is controlled based on the measured three-dimensional coordinates.

More specifically, the robot control system 1 includes a robot 2, a 3D camera 10, an image measurement device 100, a control device 200, and a robot control device 300.

The robot 2 is a mechanism that performs an operation at an arbitrary position in accordance with a command from the robot control device 300. In FIG. 2, a multi-joint robot is illustrated as a typical example of the robot 2, but it may be a SCARA robot or a parallel robot.

The robot 2 shown in FIG. 2 has one or a plurality of arms 4, and a hand piece 6 is attached to the tips of the one or the plurality of arms 4 (corresponding to the action part of the robot 2). FIG. 2 schematically illustrates a state at the time of executing calibration as described later, and a reference plate 20 is mounted on the handpiece 6 as an example of a reference object.

The reference plate 20 is a reference object for determining the correspondence between the measured three-dimensional coordinates and position information indicating the position of the action part of the robot 2 in the calibration. One or more markers 22 are drawn on the surface of the reference plate 20.

The 3D camera 10 is disposed so as to include the action part (the tip of the arm 4 and the handpiece 6) of the robot 2 in the field of view, and an image measurement device that captures an image captured in the field of view at every predetermined period or every predetermined event. Output to 100.

The robot control system 1 according to the present embodiment employs a configuration that can optically measure the three-dimensional coordinates of each part including the action part of the robot 2.

As an example, three-dimensional measurement may be realized using a technique called structured illumination. In the structured illumination method, the subject is irradiated with measurement light, and the distance to the subject is measured based on an image obtained by imaging the subject with the measurement light projected. As a structured illumination method, a spatial encoding method, a phase shift method, a light cutting method, or the like can be used. When such structured illumination is employed, the 3D camera 10 includes a light projecting unit that emits measurement light and an imaging unit that captures an image of the subject in a state where the measurement light is projected.

As another example, three-dimensional measurement may be realized using a multi-viewpoint camera. In this case, the 3D camera 10 includes a plurality of cameras arranged so that the viewpoints are different from each other. Typically, the 3D camera 10 is a stereo camera including a pair of cameras.

The image measuring device 100 measures a three-dimensional image of an arbitrary object existing in the field of view of the 3D camera 10 based on the image captured by the 3D camera 10.

For example, when structured illumination is employed, the image measurement apparatus 100 analyzes the position and displacement of the light and shade pattern included in the image output from the 3D camera 10 to obtain a three-dimensional image of the subject in the field of view. Coordinates (or a coordinate group indicating a three-dimensional shape) are calculated. When a multi-viewpoint camera is employed as the 3D camera 10, the image measurement device 100 uses the three-dimensional coordinates (or the subject) in the field of view based on the parallax of each target position calculated by matching between images. , A coordinate group indicating a three-dimensional shape) is calculated.

The image measuring device 100 can also search for the three-dimensional coordinates of a specific object in the calculated three-dimensional coordinate group. For example, the image measuring apparatus 100 can search for the marker 22 drawn on the reference plate 20 by pattern matching and output the three-dimensional coordinates of each marker 22. Calibration is performed based on the three-dimensional coordinates of these markers 22.

The control device 200 is typically composed of a PLC (programmable controller) or the like, and executes calibration or issues a command to the robot control device 300 based on the three-dimensional coordinates measured by the image measurement device 100. Or give.

The robot control device 300 controls the robot 2 according to a command from the control device 200. More specifically, the robot control device 300 drives a servo motor that drives each axis of the robot 2.

<C. Configuration example of each device constituting robot control system 1>
Next, a configuration example of each device constituting robot control system 1 according to the present embodiment will be described.

(C1: 3D camera 10)
FIG. 3 is a schematic diagram showing a configuration example of the 3D camera 10 included in the robot control system 1 according to the present embodiment. With reference to FIG. 3, the 3D camera 10 includes a processing unit 11, a light projecting unit 12, an imaging unit 13, a display unit 14, and a storage unit 15.

The processing unit 11 manages the entire process in the 3D camera 10. The processing unit 11 typically includes a processor, a storage that stores an instruction code executed by the processor, and a memory that expands the instruction code. In this case, in the processing unit 11, the processor realizes various processes by expanding and executing the instruction code on the memory. All or part of the processing unit 11 may be implemented using a dedicated hardware circuit (for example, ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array)).

The display unit 14 notifies various information acquired or calculated by the 3D camera 10 to the outside.

The storage unit 15 stores an image captured by the image capturing unit 13, a preset calibration parameter, and the like.

The communication interface (I / F) unit 16 is in charge of data exchange between the 3D camera 10 and the image measurement apparatus 100.

(C2: Image measuring device 100)
FIG. 4 is a schematic diagram illustrating a configuration example of the image measurement device 100 included in the robot control system 1 according to the present embodiment. The image measuring apparatus 100 is typically realized using a general-purpose computer. Referring to FIG. 4, the image measurement apparatus 100 includes a processor 102, a main memory 104, a storage 106, an input unit 108, a display unit 110, an optical drive 112, and a communication interface (I / F) unit 114. Including. These components are connected via a processor bus 116.

The processor 102 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and reads a program (for example, an OS (Operating System) 1060 and a three-dimensional measurement program 1062) stored in the storage 106, By developing and executing in the main memory 104, various processes as described later are realized.

The main memory 104 includes a volatile storage device such as DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory). The storage 106 includes, for example, a nonvolatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

In the storage 106, in addition to the OS 1060 for realizing basic functions, a three-dimensional measurement program 1062 for providing functions as the image measurement apparatus 100, model data used for object recognition such as the markers 22 and workpieces, and the like. 1064 and a setting reception program 1066 are stored.

The setting reception program 1066 executes processing for receiving a setting of a calibration area, which is an area in which a reference object associated with the action unit of the robot 2 is to be arranged, in camera / robot calibration as described later.

The input unit 108 is composed of a keyboard, a mouse, and the like, and accepts user operations. The display unit 110 includes a display, various indicators, a printer, and the like, and outputs a processing result from the processor 102.

The communication interface unit 114 is in charge of data exchange between the 3D camera 10 and the image measurement device 100 and is in charge of data exchange between the image measurement device 100 and the control device 200.

The image measuring apparatus 100 includes an optical drive 112, and from a recording medium 113 (for example, an optical recording medium such as a DVD (Digital Versatile Disc)) that temporarily stores a computer-readable program. Is read and installed in the storage 106 or the like.

The three-dimensional measurement program 1062 and the setting reception program 1066 executed by the image measurement apparatus 100 may be installed via the computer-readable recording medium 113, but may be installed by downloading from a server apparatus on the network. You may make it do. Further, the functions provided by the three-dimensional measurement program 1062 and the setting reception program 1066 according to the present embodiment may be realized by using a part of modules provided by the OS.

FIG. 4 shows a configuration example in which functions necessary for the image measurement apparatus 100 are provided when the processor 102 executes a program. However, some or all of these provided functions are provided with dedicated hardware. A hardware circuit (for example, ASIC or FPGA) may be used.

(C3: Control device 200)
FIG. 5 is a schematic diagram showing a configuration example of control device 200 included in robot control system 1 according to the present embodiment. The control device 200 is typically realized using a PLC (programmable controller). Referring to FIG. 5, the control device 200 includes a processor 202, a main memory 204, a storage 206, a communication interface (I / F) unit 208,

field network controllers

210 and 212, and a USB (Universal Serial Bus). A controller 214, a memory card interface 216, and a local bus controller 220 are included. These components are connected via a processor bus 230.

The processor 202 corresponds to a calculation processing unit that executes control calculations and the like, and includes a CPU, a GPU, and the like. Specifically, the processor 202 reads out a program stored in the storage 206, develops it in the main memory 204, and executes it, thereby realizing control according to the control target and various processes as will be described later. .

The main memory 204 is configured by a volatile storage device such as DRAM or SRAM. The storage 206 corresponds to a storage unit, and includes, for example, a nonvolatile storage device such as an HDD or an SSD. The storage 206 stores a calibration parameter 2060, a command generation program 2062, a calibration execution program 2064, and the like in addition to a system program for realizing basic functions.

The calibration parameter 2060 corresponds to a correspondence relationship calculated in advance between the measured three-dimensional coordinates and the position and orientation of the action part of the robot 2.

The command generation program 2062 is for positioning the action part of the robot 2 in accordance with a previously calculated correspondence (calibration parameter 2060) between the measured three-dimensional coordinates and the position and orientation of the action part of the robot 2. A process for generating a command is executed.

The calibration execution program 2064 is a calibration (to be described later) for calculating a pre-calculated correspondence relationship (calibration parameter 2060) between the measured three-dimensional coordinates and the position and orientation of the action part of the robot 2. Execute camera / robot calibration).

The communication interface unit 208 is in charge of data exchange between the image measuring device 100 and the control device 200. The

field network controllers

210 and 212 exchange data with an arbitrary device such as the robot controller 300 (see FIG. 2) via the field network. Although two

field network controllers

210 and 212 are illustrated in FIG. 5, a single field network controller may be employed. The USB controller 214 exchanges data with an arbitrary external device via a USB connection.

The memory card interface 216 receives a memory card 218 which is an example of a removable recording medium. The memory card interface 216 can write data to the memory card 218 and read various data (such as logs and trace data) from the memory card 218.

The local bus controller 220 exchanges data with an arbitrary local IO unit via the local bus 222.

FIG. 5 illustrates a configuration example in which a necessary function is provided by the processor 202 executing a program. However, part or all of the provided function is transferred to a dedicated hardware circuit (for example, an ASIC). Alternatively, it may be implemented using an FPGA or the like. Or you may implement | achieve the principal part of the control apparatus 200 using the hardware (For example, the industrial personal computer based on a general purpose personal computer) according to a general purpose architecture. In this case, a plurality of OSs having different applications may be executed in parallel using a virtualization technique, and necessary applications may be executed on each OS.

(C4: Robot controller)
FIG. 6 is a schematic diagram showing a configuration example of the robot control apparatus 300 included in the robot control system 1 according to the present embodiment. The robot control device 300 calculates an operation pattern of each axis of the robot 2 in accordance with a command from the control device 200 (typically, a three-dimensional coordinate of a target position, a posture (orientation), a moving speed, etc.).

Referring to FIG. 6, the robot control apparatus 300 includes a processor 302, a main memory 304, a storage 306, a communication interface (I / F) unit 308, and a drive controller 310. These components are connected via a processor bus 312. The robot controller 300 further includes one or more servo drivers 320-1, 320-2, 320-3,..., 320-n (hereinafter also referred to as “servo driver 320”) connected to the drive controller 310. Included).

The processor 302 corresponds to a calculation processing unit that executes control calculations and the like, and includes a CPU, a GPU, and the like. Specifically, the processor 302 implements various processes by reading out a program stored in the storage 306, developing it in the main memory 304, and executing it.

The main memory 304 is configured by a volatile storage device such as DRAM or SRAM. The storage 306 corresponds to a storage unit, and includes, for example, a nonvolatile storage device such as an HDD or an SSD. The storage 306 stores system programs related to robot control.

The communication interface unit 308 is in charge of data exchange between the control device 200 and the robot control device 300.

The drive controller 310 drives motors 330-1, 330-2, 330-3,..., 330-n (hereinafter also collectively referred to as “motors 330”) that operate the respective axes of the robot 2. Each of the servo drivers 320 is controlled. The servo driver 320 drives the connected motor 330 with torque, acceleration, and rotation speed in a specified direction in accordance with a command from the drive controller 310.

FIG. 6 shows a configuration example in which a necessary function is provided by the processor 302 executing a program. However, part or all of the provided function is transferred to a dedicated hardware circuit (for example, an ASIC). Alternatively, it may be implemented using an FPGA or the like. Or you may implement | achieve the principal part of the robot control apparatus 300 using the hardware (for example, industrial personal computer based on a general purpose personal computer) according to a general purpose architecture. In this case, a plurality of OSs having different applications may be executed in parallel using a virtualization technique, and necessary applications may be executed on each OS.

<D. Calibration>
Next, calibration in robot control system 1 according to the present embodiment will be described.

(D1: Overview)
In the robot control system 1, it is necessary to execute a plurality of types of calibration.

One of the multiple types of calibration is calibration of a coordinate system (hereinafter also referred to as “camera coordinate system”) defined based on the field of view of the 3D camera 10. In this calibration of the camera coordinate system, reference objects on which a known pattern is drawn are sequentially arranged at a plurality of known positions, and positions and orientations corresponding to the positions and orientations measured by the image measuring apparatus 100 are acquired. . Then, a correspondence relationship between the position and orientation of the reference object and the position and orientation on the three-dimensional coordinate system (typically, a calibration parameter that defines a conversion equation) is determined. By using this calibration parameter, the three-dimensional coordinates of an arbitrary subject arranged in the camera coordinate system can be accurately measured.

Here, the position on the 3D coordinate system means the coordinate specified by the value of each axis on the 3D coordinate system, and the attitude on the 3D coordinate system is based on each axis on the 3D coordinate system. Defined by the direction of rotation. That is, the position on the three-dimensional coordinate system is a three-dimensional value and is information indicating which coordinate exists on the three-dimensional coordinate system. The posture on the three-dimensional coordinate system is a three-dimensional value. It is information indicating which direction the three-dimensional coordinate system is facing.

Another one of the plurality of types of calibration is calibration of a coordinate system (hereinafter also referred to as “robot reference coordinate system”) defined based on the base position of the robot 2. The calibration of the robot reference coordinate system is such that when the three-dimensional coordinates of the target position and target posture are given to the robot control device 300 as a command from the control device 200, the tip of the robot 2 moves to the designated target position. It is a process that guarantees. An arithmetic expression for driving the motor 330 of the robot 2 in the robot control apparatus 300 is corrected by the calibration of the robot reference coordinate system.

The robot control system 1 according to the present embodiment can execute calibration (also referred to as “camera robot calibration”) for determining the correspondence between the camera coordinate system and the robot reference coordinate system. Yes.

In the following description, camera / robot calibration will be mainly described.

FIG. 7 is a diagram for explaining one aspect of camera / robot calibration in robot control system 1 according to the present embodiment. With reference to FIG. 7, in the robot control system 1, three-dimensional coordinates are measured based on an image captured by the 3D camera 10, and the robot 2 is controlled based on the measured value. Therefore, it is efficient to perform the camera / robot calibration in the robot control system 1 within the field of view of the 3D camera 10. Furthermore, even within the field of view of the 3D camera 10, depending on the target application, not all of the field of view may be used, but only a part of the spatial region may be used. In such a case, camera / robot calibration may be executed for a space area that is actually used (that is, a space area where the action part of the robot 2 may exist).

In the robot control system 1 according to the present embodiment, a user can arbitrarily set a space area (hereinafter also referred to as “calibration area”) used for camera / robot calibration. Camera / robot calibration is executed in accordance with a calibration area arbitrarily set by the user.

(D2: calibration area setting)
FIG. 8 is a schematic diagram showing an example of a user interface screen provided by robot control system 1 according to the present embodiment. In the present embodiment, as an example, the setting of the calibration area 418 is received by the setting reception program 1066 of the image measurement apparatus 100, and the user interface screen 400 shown in FIG. It is displayed on the display unit 110 (FIG. 4) of the image measuring device 100. However, it is not limited to the image measuring device 100, and may be displayed on a display device (not shown) connected to the control device 200. As long as settings are given to the image measuring apparatus 100 and / or the control apparatus 200, any display and input forms may be used.

In this embodiment, it is assumed that the 3D camera 10 is optically designed so that the cross section of the effective field of view is rectangular. Therefore, in the user interface screen 400 shown in FIG. 8, the effective field of view of the 3D camera 10 is two orthogonal directions corresponding to the rectangle of the cross section (referred to as “X-axis direction” and “Y-axis direction” for convenience). It is prescribed by. That is, the user interface screen 400 includes an X section setting object 401 and a Y section setting object 402.

The X section setting object 401 and the Y section setting object 402 have effective visual field displays 403 and 404, respectively. The effective visual field displays 403 and 404 indicate a range in which the 3D camera 10 can measure the three-dimensional coordinates. That is, the visual field range 416 of the 3D camera 10 that is the imaging unit is also displayed.

The user sets an area actually used by the application, that is, a calibration area 418 on the user interface screen 400.

More specifically, a measurement bottom surface setting bar 410 and a measurement top surface setting bar 411 are provided so as to extend over both the X cross section setting object 401 and the Y cross section setting object 402. The user operates the measurement bottom surface setting bar 410 and the measurement top surface setting bar 411 to set the top surface and the bottom surface of the calibration area 418. The uppermost surface and the lowermost surface set an effective range (effective range on the Z axis) along the optical axis AX of the 3D camera 10. That is, as a setting of the calibration area 418, a range setting on the optical axis AX of the 3D camera 10 that is the imaging unit can be set.

Typically, the lowermost surface set by the measurement lowermost surface setting bar 410 is set in consideration of the floor surface on which the robot 2 is arranged. The top surface set by the measurement top surface setting bar 411 is set in consideration of the range in which the robot 2 grips and transports the workpiece.

The X section setting object 401 is provided with X axis direction

width setting bars

412 and 413. The user operates the X-axis direction

width setting bars

412 and 413 to set the width of the calibration area 418 in the X-axis cross section. That is, the X-axis direction

width setting bars

412 and 413 are for setting an effective range (effective range on the X-axis) along the direction orthogonal to the optical axis AX of the 3D camera 10.

Similarly, Y-axis direction

width setting bars

414 and 415 are provided for the Y cross-section setting object 402. The user operates the Y-axis direction

width setting bars

414 and 415 to set the width of the calibration area 418 in the Y-axis cross section. That is, the Y-axis direction

width setting bars

414 and 415 are for setting an effective range (effective range on the Y axis) along a direction orthogonal to the optical axis AX of the 3D camera 10.

As shown in FIG. 8, the user can set an arbitrary calibration area 418 on the user interface screen 400. That is, the robot control system 1 has a setting reception function for receiving the setting of the calibration area 418 that is an area in which the reference plate 20 associated with the action part of the robot 2 is to be arranged in the camera / robot calibration. .

Based on the set calibration area 418, camera / robot calibration is executed. That is, the range in the height direction of the calibration area 418 is defined by the measurement bottom surface setting bar 410 and the measurement top surface setting bar 411. The cross section of the calibration area 418 is defined by X-axis direction

width setting bars

412 and 413 and Y-axis direction

width setting bars

414 and 415. As a result, the calibration area 418 is set as a rectangular parallelepiped (including a cube) based on the optical axis AX of the 3D camera 10 that is the imaging unit. At this time, the cross-sectional width of the calibration area 418 is determined in accordance with the visual field range 416 from the 3D camera 10. That is, the size of the cross section of the rectangular parallelepiped set as the calibration area 418 is determined in accordance with the visual field range 416 of the 3D camera 10 and the distance from the 3D camera 10 to the end face of the calibration area 418.

Note that the calibration area 418 does not necessarily have to be set as a rectangular parallelepiped, and a shape whose cross section becomes smaller toward the 3D camera 10 side according to the shape of the effective field of view of the 3D camera 10 (for example, a vertex of a quadrangular pyramid is cut off). May be set. However, when an actual application is assumed, setting as a rectangular parallelepiped is preferable from the viewpoint of measurement stability and the like.

In accordance with the calibration area 418 set by the above operation, camera / robot calibration as described later is executed. The calibration area 418 is arbitrarily set according to the target application. For example, it may be set according to the shape of the target workpiece. Specifically, in an application such as pick and place that grips a placed work and transports it to a specified position, a calibration area 418 is set based on the shape of the container containing the work. Also good.

As described above, in the present embodiment, the user sets the calibration area 418 (that is, an area where measurement of three-dimensional coordinates is necessary) on the user interface screen 400. Typically, each setting bar displayed in the user interface screen 400 is set by a mouse operation or the like.

Then, the image measurement device 100 and / or the control device 200 determines the relative movement position of the reference plate 20 (FIG. 2) so as not to be excessive or insufficient based on the set calibration region 418. To do. Thereafter, the user places the reference plate 20 at the center of the visual field range 416 of the 3D camera 10 and instructs execution of camera / robot calibration. Thereby, a command is given to the robot 2 and camera / robot calibration is executed.

(D3: Overview of camera / robot calibration)
Here, an outline of camera / robot calibration will be described. The camera / robot calibration typically includes a process of calculating matrix coefficients (parameters) for mutual conversion between the position and orientation on the camera coordinate system and the position and orientation on the robot reference coordinate system. Including.

Here, a hand piece 6 is attached to the tip of the robot 2, and a coordinate system that defines the position of the hand piece 6 (hereinafter also referred to as "robot tip coordinate system") is introduced. Furthermore, a coordinate system (hereinafter also referred to as “marker coordinate system”) that defines the position of the marker 22 on the reference plate 20 mounted on the handpiece 6 is introduced.

Here, let A be a matrix indicating the relationship between the camera coordinate system of the 3D camera 10 and the marker coordinate system. This matrix A can be estimated by recognizing the marker 22 by the 3D camera 10. Also, let B be a matrix indicating the relationship between the robot tip coordinate system and the robot reference coordinate system. This matrix B corresponds to a command from the control device 200. Then, a matrix X and a matrix Z that satisfy the following relationship are estimated.

AX = ZB
By using the estimated matrix X and matrix Z, a calculation formula for converting the position and orientation in the camera coordinate system into the position and orientation on the robot reference coordinate system is obtained.

That is, in the camera / robot calibration, the measurement values by the image measurement device 100 and the commands from the control device 200 at each position are known, and the matrix X and the matrix Z are estimated based on a set of these values. The

Such matrix estimation requires a three-dimensional coordinate data set (for example, 10 to 20 points) based on the camera coordinate system and the robot reference coordinate system, respectively. Therefore, the reference plate 20 having one or more markers 22 drawn on the surface is attached to the tip of the robot 2 and the following operations a) and b) are repeated 10 to 20 times.

a) The tip position of the robot 2 is arranged at a predetermined position so that the marker 22 is included in the field of view of the 3D camera 10, and the position and posture of the marker 22 on the robot reference coordinate system at that time (tip of the robot 2) B) Acquire the position and orientation of the marker 22 on the camera coordinate system by performing imaging with the 3D camera 10 in the state of a). B) By repeating the operations a) and b) The matrix as described above is estimated by using the obtained set of positions and orientations (10 to 20 data sets). By using the estimated line example, it is possible to calculate the position and orientation of an arbitrary subject in the robot reference coordinate system based on the positional relationship between the robot 2 and the 3D camera 10 in the camera / robot calibration.

<E. Camera / Robot Calibration (Automatic)>
Next, a description will be given of a processing example when the camera / robot calibration is automatically executed in the robot control system 1 according to the present embodiment.

FIG. 9 is a sequence diagram showing an example of an automatic processing procedure of camera / robot calibration in the robot control system 1 according to the present embodiment. As shown in FIG. 9, the camera / robot calibration automatic processing is mainly executed by the image measuring device 100 and the control device 200.

First, when the execution of camera / robot calibration is instructed, the image measuring apparatus 100 displays the user interface screen 400 for accepting the setting of the calibration area on the display unit 110 (sequence SQ100). The user operates input unit 108 such as a mouse to set a calibration area on user interface screen 400 (sequence SQ102). Image measuring apparatus 100 accepts the calibration area set by the user in sequence SQ102 (sequence SQ104).

Subsequently, the image measuring device 100 determines whether or not the reference plate 20 exists within the field of view of the 3D camera 10 based on the image captured by the 3D camera 10 (sequence SQ106).

When the reference plate 20 does not exist in the field of view of the 3D camera 10 (NO in sequence SQ106), the image measurement device 100 operates the robot 2 with a teaching pendant or the like to the user, and the reference plate 20 Is placed within the field of view of the 3D camera 10 (sequence SQ108). Then, the process of sequence SQ106 is repeated.

When reference plate 20 is present within the field of view of 3D camera 10 (YES in sequence SQ106), image measurement apparatus 100 receives a user operation (sequence SQ110) and receives the calibration area received in sequence SQ104. Together with the above information, a camera / robot calibration start command is transmitted to the control device 200 (sequence SQ112). The information of the calibration area includes positions on the camera coordinate system indicating the positions of the vertices of the calibration area (eight positions if a rectangular parallelepiped).

Control device 200 first acquires a position on the robot reference coordinate system corresponding to the calibration area defined in the camera coordinate system (sequences SQ200 to SQ206).

Specifically, control device 200 selects one position among positions on the camera coordinate system indicating each vertex of the calibration area acquired from image measurement apparatus 100 (sequence SQ200). The image measuring device 100 acquires the position on the camera coordinate system of the marker 22 on the reference plate 20 measured from the image captured by the 3D camera 10 and gives it to the control device 200 (sequence SQ120). Controller 200 arranges marker 22 on reference plate 20 at the selected position based on the difference between the position selected in sequence SQ200 and the position of marker 22 on reference plate 20 on the camera coordinate system. A command (position on the robot reference coordinate system) is calculated (sequence SQ202), and the calculated command is given to the robot controller 300 (sequence SQ204).

The processing of sequences SQ120, SQ202, and SQ204 (the processing of * 1 in FIG. 9) is repeated until the position selected in sequence SQ200 substantially matches the position of marker 22 on reference plate 20 on the camera coordinate system. It is. When the position selected in sequence SQ200 substantially matches the position of the marker 22 on the reference plate 20 on the camera coordinate system, the control device 200 instructs the corresponding point in time, that is, on the robot reference coordinate system. Is stored as a position corresponding to one vertex of the calibration area (sequence SQ206).

The processing of the sequences SQ200, SQ120, SQ202, SQ204, and SQ206 (the processing of * 2 in FIG. 9) is repeated by the number of vertices in the calibration area acquired from the image measuring device 100.

Through the above processing, a set of positions on the robot reference coordinate system corresponding to each vertex of the calibration area (eight positions if a rectangular parallelepiped) is acquired. If necessary, the posture on the robot reference coordinate system corresponding to each vertex of the calibration area is also acquired.

Subsequently, the control device 200 determines a set of positions where the reference plate 20 should be arranged based on the set of positions on the robot reference coordinate system acquired in the sequence SQ206 (sequence SQ208). In sequence SQ208, a plurality of positions on the robot reference coordinate system are determined. The control device 200 may determine a trajectory or the like for moving the reference plate 20 in addition to a set of positions where the reference plate 20 should be arranged.

The set of positions where the reference plate 20 should be arranged may be determined, for example, by equally dividing the vertices of the calibration area into a predetermined number. Or you may make it set more positions as the area | region far from the 3D camera 10 is not equal, and is far.

As described above, since 10 to 20 data sets are required for the camera / robot calibration, it is preferable to determine 10 to 20 positions also in the sequence SQ208.

Then, control device 200 selects one position among a plurality of positions where reference plate 20 determined in sequence SQ208 is to be arranged (sequence SQ210). Then, control device 200 gives a command to robot control device 300 to instruct the placement of reference plate 20 at the selected position (position on the robot reference coordinate system) (sequence SQ212). The processing of sequences SQ210 and SQ212 executed by control device 200 corresponds to an arrangement control function for sequentially giving commands to robot 2 and sequentially arranging reference plates 20 (reference objects) in the calibration area.

When the reference plate 20 is placed at the designated position by the robot 2, the image measurement device 100 acquires the position on the camera coordinate system of the marker 22 on the reference plate 20 measured from the image captured by the 3D camera 10. To control device 200 (sequence SQ122).

The control device 200 includes a position (and posture) on the robot reference coordinate system that indicates the position of the reference plate 20 that is currently arranged, and a position (and posture) on the camera coordinate system measured by an image captured by the 3D camera 10. Are stored in association with each other (sequence SQ214).

The processing of the sequences SQ210, SQ122, SQ212, and SQ214 (the processing of * 3 in FIG. 9) is repeated for the number of positions where the reference plate 20 determined in the sequence SQ210 is to be arranged.

Through the above processing, it is possible to acquire a data set indicating the correspondence between the position on the camera coordinate system and the position on the robot reference coordinate system, which is necessary for camera / robot calibration. Finally, the control apparatus 200 estimates a matrix that defines the correspondence between the camera coordinate system and the robot reference coordinate system by executing a predetermined calculation process based on the acquired data set (sequence). SQ216). This estimated matrix is stored as a result of camera / robot calibration. In other words, the control device 200 sets a combination of the three-dimensional coordinates of the reference plate 20 and the position and orientation of the action part of the robot 2 acquired when the reference plate 20 (reference object) is sequentially arranged in the calibration area. Based on the above, a calibration parameter (correspondence) is calculated.

The execution of the camera / robot calibration is completed by the processing as described above.
FIG. 10 is a schematic diagram showing an example of a user interface screen provided in the automatic camera / robot calibration process in the robot control system 1 according to the present embodiment. A user interface screen 420 shown in FIG. 10 simultaneously displays a three-dimensional measurement result of the reference plate 20 obtained by imaging with the 3D camera 10 and a two-dimensional image of the reference plate 20 obtained by imaging with the 3D camera 10. To do. Further, the user interface screen 420 displays the calibration area and the height of the reference plate 20 preset by the user.

Specifically, on the user interface screen 420, an effective area frame 422 indicating a preset calibration area is displayed for an image captured by the 3D camera 10. The user interface screen 420 includes a height display bar 424. The height display bar 424 displays the measurable range of the Z axis. It is assumed that the distance between the 3D camera 10 and the base of the robot 2 is input as an initial setting. Then, an indicator 426 indicating the height direction position of the reference plate 20 measured by the image measuring device 100 is displayed in association with the height display bar 424.

As shown in FIGS. 10A and 10B, since the size of the cross section of the calibration area varies depending on the height of the reference plate 20, the size of the effective area frame 422 is also displayed on the user interface screen 420. The height changes according to the height of the reference plate 20.

When the user refers to the user interface screen 420 and the reference plate 20 does not exist within the set calibration area, the user operates the teaching pendant or the like so that the reference plate 20 is arranged within the calibration area. Adjust to.

When the user completes the initial setting with reference to the user interface screen 420 as shown in FIG. 10, the range in which the robot 2 should move is automatically determined according to the set calibration area as described above. Is done. At this time, by using the two-dimensional image captured by the 3D camera 10 and the three-dimensional measurement result as feedback information, a movement amount command for the robot 2 is sequentially determined.

Then, the movement step amount is determined equally from the upper limit and the lower limit of the determined movement range of the robot 2. Then, by acquiring the two-dimensional image and performing the three-dimensional measurement while moving the reference plate 20 by the determined step amount, the correspondence between the position on the camera coordinate system and the position and orientation on the robot reference coordinate system To get. Based on the correlation between the acquired position on the camera coordinate system and the position and orientation on the robot reference coordinate system, camera / robot calibration is executed to calculate necessary parameters.

<F. Camera / Robot Calibration (Manual)>
Next, a processing example in the case of manually executing camera / robot calibration in the robot control system 1 according to the present embodiment will be described.

FIG. 11 is a sequence diagram showing an example of a manual processing procedure of camera / robot calibration in the robot control system 1 according to the present embodiment. As shown in FIG. 11, manual processing of camera / robot calibration is mainly executed by the image measurement device 100 and the control device 200.

First, when the execution of camera / robot calibration is instructed, the image measuring apparatus 100 displays a user interface screen 400 for accepting the setting of the calibration area on the display unit 110 (sequence SQ150). The user operates input unit 108 such as a mouse to set a calibration area on user interface screen 400 (sequence SQ152). Image measuring apparatus 100 accepts the calibration area set by the user in sequence SQ152 (sequence SQ154).

Subsequently, the image measuring apparatus 100 determines a set of positions where the reference plate 20 is to be arranged based on the set calibration area (sequence SQ156). That is, the image measuring apparatus 100 has a position determination function that determines a plurality of positions where the reference plate 20 (reference object) is to be arranged in the calibration area as a calibration execution function. In sequence SQ156, a plurality of positions on the camera coordinate system are determined. The control device 200 may determine a trajectory or the like for moving the reference plate 20 in addition to a set of positions where the reference plate 20 should be arranged.

As described above, since 10 to 20 data sets are required for camera / robot calibration, it is preferable to determine 10 to 20 positions also in the sequence SQ156.

Subsequently, the image measuring apparatus 100 selects one position as a target position among a plurality of positions where the reference plate 20 determined in the sequence SQ156 is to be arranged (sequence SQ158). The image measuring apparatus 100 acquires the current position on the camera coordinate system of the marker 22 on the reference plate 20 based on the image captured by the 3D camera 10 (sequence SQ160), and the selected target position and Based on the difference from the acquired current position, it is determined whether or not the reference plate 20 is disposed at the target position (sequence SQ162).

When the reference plate 20 is not arranged at the target position (NO in sequence SQ162), the image measuring device 100 notifies the user of information for arranging the reference plate 20 at the target position (sequence SQ164). The user refers to this notification and operates the teaching pendant or the like to adjust the position and posture of the robot 2 so that the reference plate 20 is disposed at the target position. Then, the processes after sequence SQ160 are repeated.

On the other hand, when reference plate 20 is disposed at the target position (YES in sequence SQ162), image measurement apparatus 100 notifies the user that reference plate 20 is disposed at the target position. At the same time (sequence SQ166), the position (and orientation) of the marker 22 on the reference plate 20 on the camera coordinate system is transmitted to the control device 200 (sequence SQ168).

Then, control device 200 acquires a position (and posture) on the robot reference coordinate system indicating the position of reference plate 20 currently arranged from robot control device 300 (sequence SQ250). Then, the control device 200 determines the position (and orientation) on the robot reference coordinate system indicating the position of the reference plate 20 currently arranged and the position (and orientation) on the camera coordinate system from the image measuring device 100. Correspondingly stored (sequence SQ252).

The processing of the sequences SQ158 to SQ168, SQ250, SQ252 (the processing of * 4 in FIG. 11) is repeated by the number of positions where the reference plate 20 determined in the sequence SQ156 is to be arranged.

Through the above processing, it is possible to acquire a data set indicating the correspondence between the position on the camera coordinate system and the position on the robot reference coordinate system, which is necessary for camera / robot calibration. Finally, the control apparatus 200 estimates a matrix that defines the correspondence between the camera coordinate system and the robot reference coordinate system by executing a predetermined calculation process based on the acquired data set (sequence). SQ254). This estimated matrix is stored as a result of camera / robot calibration. In other words, the control device 200 sets a combination of the three-dimensional coordinates of the reference plate 20 and the position and orientation of the action part of the robot 2 acquired when the reference plate 20 (reference object) is sequentially arranged in the calibration area. Based on the above, a calibration parameter (correspondence) is calculated.

The execution of the camera / robot calibration is completed by the processing as described above.
FIG. 12 is a schematic diagram showing an example of a user interface screen provided in the camera / robot calibration manual process in the robot control system 1 according to the present embodiment.

FIG. 12 is a schematic diagram showing an example of a user interface screen provided in the camera / robot calibration manual process in the robot control system 1 according to the present embodiment. A user interface screen 430 shown in FIG. 12 simultaneously displays a three-dimensional measurement result of the reference plate 20 obtained by imaging with the 3D camera 10 and a two-dimensional image of the reference plate 20 obtained by imaging with the 3D camera 10. To do. Further, the user interface screen 430 displays the calibration area and the height of the reference plate 20 preset by the user.

Specifically, on the user interface screen 430, an effective area frame 422 indicating a preset calibration area is displayed for an image captured by the 3D camera 10. In association with the effective area frame 422, an indicator 432 indicating the two-dimensional position of the selected target position is displayed.

Note that since the cross-sectional size of the calibration area varies depending on the height of the reference plate 20, the size of the effective area frame 422 also changes on the user interface screen 430 according to the height of the reference plate 20. It will be.

The user interface screen 420 includes a height display bar 424. The height display bar 424 displays the measurable range of the Z axis. It is assumed that the distance between the 3D camera 10 and the base of the robot 2 is input as an initial setting. An indicator 426 indicating the position in the height direction of the reference plate 20 measured by the image measuring apparatus 100 and an indicator 428 indicating the height of the selected target position are associated with the height display bar 424. Is displayed.

As shown in FIG. 12, the image measuring apparatus 100 provides a position display function that indicates the relationship between one of a plurality of positions where the determined reference plate 20 should be placed and the current position of the reference plate 20 (reference object). To do.

The user can place the reference plate 20 at an appropriate position by referring to the user interface screen 420. That is, the user operates the robot 2 so that the arrangement height of the reference plate 20 matches a predetermined height and the indicator 432 indicating the two-dimensional position of the reference plate 20 coincides with the center of the reference plate 20. To do. Even during the operation of the robot 2, the acquisition of the two-dimensional image and the three-dimensional measurement by the 3D camera 10 are repeatedly executed.

When the reference plate 20 is placed at the selected target position, the position on the camera coordinate system and the position and orientation on the robot reference coordinate system at that time are stored in association with each other. This series of operations and processes is repeated for the number of times for executing the camera / robot calibration. Finally, based on the correlation between the obtained position on the camera coordinate system and the position and orientation on the robot reference coordinate system, camera / robot calibration is executed to calculate necessary parameters.

It should be noted that if the user mistakenly operates and the reference plate 20 comes out of the calibration area, some notification may be given to the user. As such a notification form, a change in display color, display of a warning message, generation of an alarm sound, or the like may be employed. An example in which a warning message 434 is displayed on the user interface screen 430 illustrated in FIG. By giving such a warning, the camera / robot calibration can be executed efficiently.

Alternatively, when the position of the reference plate 20 greatly deviates from an appropriate region due to a user's erroneous operation, a protection function implemented in the robot control device 300 is given by giving a command to the robot control device 300 from the control device 200. For example, the robot 2 may be forcibly stopped.

<G. Modification>
The following modifications can be made to the above-described embodiment.

(G1: Mounting form)
In the above description, the mounting form that realizes the camera / robot calibration according to the present embodiment mainly by linking the image measuring apparatus 100 and the control apparatus 200 has been described. However, the mounting form is not limited thereto. It is not a thing.

For example, the functions provided by the image measurement device 100 and the control device 200 may be interchanged, or the image measurement device 100 and the control device 200 may be configured as an integrated device. Furthermore, the robot control apparatus 300 can be configured as an integrated apparatus.

That is, any implementation may be adopted as long as it can provide the processing and functions as described above.

(G2: Automatic processing and manual processing)
For convenience of explanation, the robot control system 1 capable of performing both automatic processing and manual processing of camera / robot calibration is illustrated, but it is not always necessary to implement both processing. Only one of the processes may be implemented according to the required specification or application.

(G3: position and path of the reference plate 20)
The position at which the reference plate 20 is arranged for acquiring a data set necessary for executing the camera / robot calibration can be arbitrarily set. Further, a route for sequentially arranging the reference plates 20 at a plurality of set positions can be arbitrarily set. However, with respect to the path where the reference plates 20 are sequentially arranged, the shortest path may be set according to a predetermined optimization algorithm in order to improve work efficiency.

<H. Addendum>
The present embodiment as described above includes the following technical idea.
[Configuration 1]
A robot control system (1),
An imaging unit (10) disposed so as to include the action unit (8) of the robot (2) in the field of view;
A measurement unit (52; 100) for measuring a three-dimensional coordinate of an arbitrary object existing in the field of view of the imaging unit based on an image captured by the imaging unit;
A command generation unit (54) that generates a command for positioning the action part of the robot according to a pre-calculated correspondence relationship between the measured three-dimensional coordinates and the position and posture of the action part of the robot; 200),
A calibration execution unit (58; 100; 200) for executing calibration for calculating the correspondence relationship;
In the calibration, a robot control including a setting reception unit (60; 400) that receives a setting of a calibration region (418) that is a region where a reference object (20) associated with the action unit of the robot is to be arranged system.
[Configuration 2]
The robot control system according to Configuration 1, wherein the setting reception unit indicates a range of a calibration area that is set with the imaging unit as a reference.
[Configuration 3]
The robot control system according to Configuration 2, wherein the setting reception unit also displays a visual field range (416) of the imaging unit.
[Configuration 4]
The robot control system according to any one of configurations 1 to 3, wherein the calibration area is set as a rectangular parallelepiped based on an optical axis (AX) of the imaging unit.
[Configuration 5]
The robot according to configuration 4, wherein a size of a cross section of a rectangular parallelepiped set as the calibration area is determined according to a field of view of the imaging unit and a distance from the imaging unit to an end surface of the calibration area. Control system.
[Configuration 6]
The robot control system according to Configuration 4 or 5, wherein the setting reception unit receives a range setting on the optical axis (AX) of the imaging unit as the setting of the calibration area.
[Configuration 7]
The calibration execution unit
An arrangement controller (200; SQ210, SQ212) for sequentially giving instructions to the robot and sequentially arranging the reference objects in the calibration area;
The correspondence is calculated based on a set of the three-dimensional coordinates of the reference object and the position and orientation of the action part of the robot, which are acquired when the reference object is sequentially arranged in the calibration area. The robot control system according to any one of configurations 1 to 6, including a calculation unit (200; SQ216).
[Configuration 8]
The calibration execution unit
A position determination unit (100; SQ156) for determining a plurality of positions where the reference object is to be arranged in the calibration area;
A position display unit (100; SQ164) showing a relationship between one of the determined positions and the current position of the reference object;
Based on a set of the three-dimensional coordinates of the reference object and the position and orientation of the action part of the robot, which are obtained when the reference object is sequentially arranged at the determined positions, the correspondence relationship is obtained. The robot control system according to any one of configurations 1 to 6, further comprising a calculation unit (200; SQ254) for calculating.

<I. Advantage>
When performing the camera / robot calibration between the camera coordinate system associated with the image captured by the 3D camera and the robot reference coordinate system for controlling the robot, as in the robot control system according to the present embodiment. The reference object needs to be arranged at a predetermined position by the robot, and it is not easy to specify the position and orientation of the reference object.

That is, it is necessary to place the reference object in the measurement field of view, and the reference object should not be brought into contact with the floor surface, and an inexperienced user may find it difficult to operate the robot while satisfying such requirements.

Furthermore, when camera calibration is executed based on a two-dimensional image captured by a 3D camera, an area that cannot be three-dimensionally measured on the far side of the 3D camera may be included in the target. The camera calibration is executed for such a region. Also, the size of the effective field of view is different between the near side and the far side of the 3D camera, and it is difficult to automatically set the range to be operated when positioning the reference object along the optical axis direction. .

For such a problem, in the robot control system according to the present embodiment, an area (calibration area) where the camera / robot calibration should be executed can be arbitrarily set based on the 3D camera. At this time, since the effective visual field of the 3D camera can be displayed together, an appropriate calibration area can be set. Then, camera / robot calibration can be executed automatically or manually for the set calibration area.

In the present embodiment, the position where the reference object is arranged can be automatically determined by setting the cross section of the calibration area to substantially the same size at any position in the optical axis direction. At the same time, the setting according to the application can be facilitated.

The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

DESCRIPTION OF SYMBOLS 1 Robot control system, 2 Robot, 4 Arm, 6 Handpiece, 8 Action part, 10 3D camera, 11 Processing part, 12 Flooding part, 13,50 Imaging part, 14,110 Display part, 15 Storage part, 16, 114, 208, 308 Communication interface (I / F) section, 20 reference plate, 22 marker, 52 measurement section, 54 command generation section, 56, 2060 calibration parameter, 58 calibration execution section, 60 setting reception section, 100 images Measuring device, 102, 202, 302 processor, 104, 204, 304 main memory, 106, 206, 306 storage, 108 input unit, 112 optical drive, 113 recording medium, 116, 230, 312 processor bus, 200 control device, 210 , 12 field network controller, 214 USB controller, 216 memory card interface, 218 memory card, 220 local bus controller, 222 local bus, 300 robot controller, 310 drive controller, 320 servo driver, 330 motor, 400, 420, 430 user interface Screen, 401, 402 Cross section setting object, 403 Effective field display, 410 Measurement bottom surface setting bar, 411 Measurement top surface setting bar, 412, 414 Axial width setting bar, 416 Field of view range, 418 Calibration area, 422 Effective area frame 424 display bar, 426, 428, 432 indicator, 434 warning message, 1060 OS, 1062 tertiary Measuring program, 1064 model data 1066 setting reception program 2062 command generation program, 2064 calibration execution program, AX optical axis.

Claims

A robot control system,
An imaging unit disposed so as to include the robot's action unit in the field of view;
A measurement unit that measures the three-dimensional coordinates of an arbitrary object existing in the field of view of the imaging unit, based on the image captured by the imaging unit;
A command generation unit that generates a command for positioning the action part of the robot according to a pre-calculated correspondence relationship between the measured three-dimensional coordinates and the position and posture of the action part of the robot;
A calibration execution unit for executing calibration for calculating the correspondence relationship;
In the calibration, a robot control system comprising: a setting reception unit that receives a setting of a calibration region that is a region in which a reference object associated with the operation unit of the robot is to be placed.
The robot control system according to claim 1, wherein the setting reception unit indicates a set calibration area range with the imaging unit as a reference.
The robot control system according to claim 2, wherein the setting receiving unit also displays a field of view range of the imaging unit.
The robot control system according to any one of claims 1 to 3, wherein the calibration area is set as a rectangular parallelepiped based on an optical axis of the imaging unit.
The size of the cross section of the rectangular parallelepiped set as the calibration area is determined according to a field of view of the imaging unit and a distance from the imaging unit to an end surface of the calibration area. Robot control system.
The robot control system according to claim 4 or 5, wherein the setting reception unit receives a range setting on the optical axis of the imaging unit as the calibration area setting.
The calibration execution unit
An arrangement controller that sequentially gives instructions to the robot and sequentially arranges the reference objects in the calibration area;
The correspondence is calculated based on a set of the three-dimensional coordinates of the reference object and the position and orientation of the action part of the robot, which are acquired when the reference object is sequentially arranged in the calibration area. The robot control system according to any one of claims 1 to 6, further comprising a calculation unit.
The calibration execution unit
A position determining unit that determines a plurality of positions where the reference object is to be arranged in the calibration area;
A position display unit indicating a relationship between one of the determined positions and the current position of the reference object;
Based on a set of the three-dimensional coordinates of the reference object and the position and orientation of the action part of the robot, which are obtained when the reference object is sequentially arranged at the determined positions, the correspondence relationship is obtained. The robot control system according to any one of claims 1 to 6, further comprising a calculation unit for calculating.