US20180194008A1 - Calibration device, calibration method, and computer readable medium for visual sensor - Google Patents
Calibration device, calibration method, and computer readable medium for visual sensor Download PDFInfo
- Publication number
- US20180194008A1 US20180194008A1 US15/862,103 US201815862103A US2018194008A1 US 20180194008 A1 US20180194008 A1 US 20180194008A1 US 201815862103 A US201815862103 A US 201815862103A US 2018194008 A1 US2018194008 A1 US 2018194008A1
- Authority
- US
- United States
- Prior art keywords
- camera
- range
- calibration
- robot
- target mark
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1679—Programme controls characterised by the tasks executed
- B25J9/1692—Calibration of manipulator
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
- B25J19/0095—Means or methods for testing manipulators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- H04N13/0239—
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/30—Nc systems
- G05B2219/39—Robotics, robotics to robotics hand
- G05B2219/39008—Fixed camera detects reference pattern held by end effector
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/30—Nc systems
- G05B2219/39—Robotics, robotics to robotics hand
- G05B2219/39016—Simultaneous calibration of manipulator and camera
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/30—Nc systems
- G05B2219/39—Robotics, robotics to robotics hand
- G05B2219/39024—Calibration of manipulator
-
- G06K9/4604—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S901/00—Robots
- Y10S901/02—Arm motion controller
- Y10S901/09—Closed loop, sensor feedback controls arm movement
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S901/00—Robots
- Y10S901/46—Sensing device
- Y10S901/47—Optical
Definitions
- the present invention relates to calibration on a visual sensor, particularly, to a calibration device, a calibration method, and computer readable medium for measuring a range in which a target mark is detected during implementation of calibration using a stereo camera with multiple cameras.
- a robot In a robot system, a robot is given a visual function. Operation such as handling or machining on a work is done by making the robot recognize the position information of a subject.
- This visual function is fulfilled by capturing an image of the subject with a visual sensor attached to a hand or a neighboring part of the robot or a visual sensor provided around the robot.
- calibration data is required for converting the position information of the subject in an image to the position information of the subject viewed from the robot.
- Patent document 1 suggests a method of attaching a lattice pattern to the end of a robot arm and measuring the pattern with a fixedly arranged visual sensor (this method will be called a “method A”).
- Patent document 2 suggests a method of performing calibration by attaching a target mark having a position and a posture determined in advance in an end point coordinate system at a robot to the end of an arm, and determining the position of the target mark at multiple points in an image of the target mark captured by a visual sensor (this method will be called a “method B”).
- a pattern to be used for the calibration should be prepared in advance. If the view of a camera is too wide or too narrow for the prepared pattern, the method A fails to perform high-precision calibration. In contrast, calibration by the method B allows calibration in a wider view or calibration in a narrower view than in the method A, thereby advantageously increasing the degree of freedom of the calibration.
- a stereo camera has been used in some cases as a visual sensor as described in patent document 3, for example. There have been the following systems for the stereo camera: a passive stereo system of matching corresponding points by using the texture of a subject; and an active stereo system of matching corresponding points by using a pattern projected on a subject. In either case, calibration is required on two or more cameras forming the stereo camera.
- a target mark is desirably moved through a wide area in a view range while the visual sensor (camera, for example) is calibrated. If the target mark is to collide with an obstacle while being moved in the view range, for example, a range for move of the target mark should be limited.
- a range of move of the target mark hereinafter also called a “calibration range” for capturing an image of the target mark having been moved is required to be set in some way in a plane on which the target mark is to be moved during implementation of the calibration so as to allow the target mark to be moved through a wide area in the view range.
- the calibration range can also be designated in a robot coordinate system.
- a user is required to check the target mark in an image of the target mark captured by the camera while operating a robot by hand.
- the calibration range is desirably set in a captured image.
- the multiple cameras are attached at separate positions, so that each camera is required to be calibrated independently.
- a range of move of the target mark (calibration range) is required to be set for each of a first camera 21 and a second camera 22 in a plane on which the target mark is to be moved in such a manner that an image of the target mark having been moved can be captured in a range of image capture (angle of view) by each camera.
- patent document 2 does not recite that a range of move of a work 1 (calibration range) is set on a plane on which the work 1 is to move and in a range of image capture (angle of view) by an image capture unit 40 before the image capture unit 40 is calibrated.
- the calibration described in patent document 2 is not to calibrate multiple cameras such as those of a stereo camera but is merely to calibrate a single camera.
- the calibration described in patent document 3 is performed by attaching a checker board as a basic matrix calculation tool to the end of a robot arm and capturing images of the checker board with a stereo camera.
- this calibration inherently does not correspond to calibration performed by attaching a target mark to the end of an arm, and determining the position of the target mark at multiple points where the target mark has been moved in the images of the target mark captured by the stereo camera.
- the present invention provides a calibration device, a calibration method, and a program capable of setting a range of move of a target mark (calibration range) in advance in space for move of the target mark so as to make the range of move fall in a view range for a single visual sensor (camera, for example) or in a view range for each camera forming a stereo camera.
- a calibration device (“visual sensor controller 1 ” described later, for example) is a calibration device that associates a robot coordinate system at a robot (“robot 4 ” described later, for example) and an image coordinate system at a camera (“camera 2 ” described later, for example) by placing a target mark (“target mark 5 ” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the camera.
- the calibration device comprises: an image range setting unit (“first image range setting unit 105 ” described later, for example) that sets an image range in the image coordinate system; and a calibration range measurement unit (“first calibration range measurement unit 106 ” described later, for example) that measures a calibration range as an operation range for the robot corresponding to the image range before implementation of calibration by controlling the robot to move the target mark and detecting the target mark. During implementation of the calibration, the robot is controlled so as to move the target mark in the calibration range.
- the calibration range may be configured to be set on a plane.
- the calibration range may be configured to be set on a plane vertical to an optical axis of the camera.
- the calibration range may be configured to be set on a plane tilted from an optical axis of the camera.
- the calibration range may be configured to be set on each of at least two planes.
- a calibration method according to the present invention (“visual sensor control method” described later, for example) is a calibration method implemented by a calibration device that associates a robot coordinate system at a robot (“robot 4 ” described later, for example) and an image coordinate system at a camera (“camera 2 ” described later, for example) by placing a target mark (“target mark 5 ” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the camera.
- the calibration method comprises: an image range setting step (“first image range setting step” described later, for example) of setting an image range in the image coordinate system; and a calibration range measurement step (“calibration range measurement step” described later, for example) of measuring a calibration range as an operation range for the robot corresponding to the image range before implementation of calibration by controlling the robot to move the target mark and detecting the target mark.
- the robot is controlled so as to move the target mark in the calibration range.
- a program according to the present invention (“program” described later, for example) causes a computer to execute the following steps.
- the computer controls a calibration device that associates a robot coordinate system at a robot (“robot 4 ” described later, for example) and an image coordinate system at a camera (“camera 2 ” described later, for example) by placing a target mark (“target mark 5 ” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the camera.
- the steps comprise: an image range setting step (“first image range setting step” described later, for example) of setting an image range in the image coordinate system; and a calibration range measurement step (“calibration range measurement step” described later, for example) of measuring a calibration range as an operation range for the robot corresponding to the image range before implementation of calibration by controlling the robot to move the target mark and detecting the target mark.
- the robot is controlled so as to move the target mark in the calibration range.
- a calibration device (“visual sensor controller 1 A” described later, for example) is a calibration device that associates a robot coordinate system at a robot (“robot 4 ” described later, for example), position information in an image coordinate system at a first camera (“first camera 21 ” described later, for example) of a stereo camera (“stereo camera 2 A” described later, for example), and position information in an image coordinate system at a second camera (“second camera 22 ” described later, for example) of the stereo camera by placing a target mark (“target mark 5 ” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the stereo camera including at least the first camera and the second camera.
- the calibration device comprises: a first image range setting unit (“first image range setting unit 105 ” described later, for example) that sets a first image range in the image coordinate system at the first camera; a second image range setting unit (“second image range setting unit 1052 ” described later, for example) that sets a second image range in the image coordinate system at the second camera; a first calibration range measurement unit (“first calibration range measurement unit 106 ” described later, for example) that measures a first calibration range as a first operation range for the robot corresponding to the first image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the first camera; and a second calibration range measurement unit (“second calibration range measurement unit 1062 ” described later, for example) that measures a second calibration range as a second operation range for the robot corresponding to the second image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the second camera. While the first camera and the second camera are calibrated,
- each of the first calibration range and the second calibration range may be configured to be set on a plane.
- a calibration method according to the present invention (“visual sensor control method” described later, for example) is a calibration method implemented by a calibration device that associates a robot coordinate system at a robot (“robot 4 ” described later, for example), position information in an image coordinate system at a first camera (“first camera 21 ” described later, for example) of a stereo camera (“stereo camera 2 A” described later, for example), and position information in an image coordinate system at a second camera (“second camera 22 ” described later, for example) of the stereo camera by placing a target mark (“target mark 5 ” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the stereo camera including at least the first camera and the second camera.
- the calibration method comprises: a first image range setting step (“first image range setting step” described later, for example) of setting a first image range in the image coordinate system at the first camera; a second image range setting step (“second image range setting step” described later, for example) of setting a second image range in the image coordinate system at the second camera; a first calibration range measurement step (“first calibration range measurement step” described later, for example) of measuring a first calibration range as a first operation range for the robot corresponding to the first image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the first camera; and a second calibration range measurement step (“second calibration range measurement step” described later, for example) of measuring a second calibration range as a second operation range for the robot corresponding to the second image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the second camera. While the first camera and the second camera are calibrated, the robot is controlled so as to move the target mark in
- a program according to the present invention (“program” described later, for example) causes a computer to execute the following steps.
- the computer controls a calibration device that associates a robot coordinate system at a robot (“robot 4 ” described later, for example), position information in an image coordinate system at a first camera (“first camera 21 ” described later, for example) of a stereo camera (“stereo camera 2 A” described later, for example), and position information in an image coordinate system at a second camera (“second camera 22 ” described later, for example) of the stereo camera by placing a target mark (“target mark 5 ” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the stereo camera including at least the first camera and the second camera.
- the steps comprise: a first image range setting step (“first image range setting step” described later, for example) of setting a first image range in the image coordinate system at the first camera; a second image range setting step (“second image range setting step” described later, for example) of setting a second image range in the image coordinate system at the second camera; a first calibration range measurement step (“first calibration range measurement step” described later, for example) of measuring a first calibration range as a first operation range for the robot corresponding to the first image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the first camera; and a second calibration range measurement step (“second calibration range measurement step” described later, for example) of measuring a second calibration range as a second operation range for the robot corresponding to the second image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the second camera. While the first camera and the second camera are calibrated, the robot is controlled so as to move the target mark in at
- a calibration device, a calibration method, and a program provided by the present invention are capable of performing calibration with an increased degree of freedom by measuring an operation range for a robot (calibration range) corresponding to a range of move of a target mark in advance before implementation of the calibration in space where the target mark is to be moved during implementation of the calibration, based on a view range for a single visual sensor or a view range for each camera forming a stereo camera.
- FIG. 1 shows the configuration of a robot system 1000 entirely
- FIG. 2 shows examples of a target mark 5 ;
- FIG. 3 is a functional block diagram showing the functional configuration of a visual sensor controller 1 ;
- FIG. 4 is a block diagram showing the functional configuration of a CPU 10 in the visual sensor controller 1 ;
- FIG. 5 shows a flowchart for generating a model pattern
- FIG. 6 shows an example of a model pattern designation area
- FIG. 7 shows an example of a point P where the three-dimensional position information of the target mark 5 is to be measured
- FIG. 8A shows a flowchart showing a flow followed by a first calibration range measurement unit 106 for measuring a coordinate position information in a robot coordinate system showing a boundary of a calibration range;
- FIG. 8B shows a flowchart showing the flow followed by the first calibration range measurement unit 106 for measuring the coordinate position information in the robot coordinate system showing the boundary of the calibration range;
- FIG. 9A shows an example of a path traced by the target mark 5 attached to an end portion of an arm 41 while the target mark 5 is moved in a calibration range
- FIG. 9B shows an example of a path traced by the target mark 5 attached to the end portion of the arm 41 while the target mark 5 is moved in the calibration range;
- FIG. 10 is a flowchart showing calibration process on a camera 2 according to a first embodiment
- FIG. 11A shows an example of a plane on which a calibration range is set
- FIG. 11B shows examples of planes on which a calibration range is set
- FIG. 12 shows the configuration of a robot system 1000 entirely for performing calibration by using a stereo camera 2 A according to a second embodiment
- FIG. 13A shows an example of arrangement of the stereo camera 2 A
- FIG. 13B shows an example of arrangement of the stereo camera 2 A
- FIG. 14 is a block diagram showing the functional configuration of a CPU 10 in a visual sensor controller 1 A.
- FIG. 15 shows a flowchart showing calibration process on the stereo camera 2 A according to the second embodiment.
- FIG. 1 shows the configuration of a robot system 1000 entirely for calibration on a visual sensor.
- the robot system 1000 is for performing calibration using a single camera 2 .
- the robot system 1000 includes: a visual sensor controller 1 that makes three-dimensional measurement through image processing on data about an image captured by the single camera 2 ; a robot 4 having an arm 41 with an end portion to which a target mark 5 is attached; and a robot controller 3 for control over the robot 4 .
- the camera 2 is fixed to a pedestal (not shown in the drawings).
- the visual sensor controller 1 is described as an example of a calibration device.
- FIG. 2 shows examples of the target mark 5 .
- the target mark 5 is not limited to these examples. Any shape is applicable to the target mark 5 . Meanwhile, the shape of the target mark 5 is desirably such that the characteristics of the target mark 5 used as a model pattern are expressed on a two-dimensional plane.
- the target mark 5 may be a mark printed on paper or a seal and attached to the end of the arm 41 of the robot 4 , for example.
- the robot controller 3 recognizes the coordinate position information of the end of the arm 41 in a robot coordinate system as a current position. Thus, based on the coordinate position of the end of the arm 41 in the robot coordinate system and the known three-dimensional position information and the known three-dimensional posture of the target mark 5 in an end point coordinate system at the robot 4 , the robot controller 3 can always recognize the coordinate position of the target mark 5 in the robot coordinate system while the robot controller 3 controls drive of the arm 41 .
- the robot controller 3 includes a CPU (not shown in the drawings) for controlling the robot controller 3 entirely in an integrated fashion.
- the visual sensor controller 1 is connected to the robot controller 3 through an external equipment interface (not shown in the drawings).
- the robot controller 3 transmits a signal for triggering image processing to the visual sensor controller 1 . Further, the robot controller 3 receives a result of image processing obtained by execution of the image processing (detection of the target mark 5 , for example) by the visual sensor controller 1 , etc.
- the robot controller 3 controls drive of the arm 41 so as to move the target mark 5 attached to the end of the arm 41 in a range set in advance for calibration (“calibration range”) during implementation of the calibration. Process of measuring the calibration range will be described in detail later.
- the robot controller 3 measures the coordinate position of the target mark 5 in the robot coordinate system attached to the end of the arm 41 of the robot 4 . Specifically, the robot controller 3 can measure the coordinate position of the target mark 5 in the robot coordinate system while the target mark 5 is at a destination.
- the camera 2 is connected to the visual sensor controller 1 functioning as the calibration device.
- the visual sensor controller 1 makes the camera 2 capture an image of the target mark 5 to calibrate the camera 2 .
- FIG. 3 is a functional block diagram showing the functional configuration of the visual sensor controller 1 .
- the visual sensor controller 1 includes a central processing unit (CPU) 10 for controlling the visual sensor controller 1 entirely in an integrated fashion.
- the CPU 10 is connected through a bus 11 to multiple frame memories 12 , a read-only memory (ROM) 13 , a random access memory (RAM) 14 , and a nonvolatile RAM 15 .
- the camera 2 is connected to the bus 11 through a camera interface 16 . Further, a monitor 19 is connected to the bus 11 through a monitor interface 17 .
- the CPU 10 is further connected through the bus 11 to an external equipment interface 18 .
- the external equipment interface 18 is connected to the robot controller 3 to receive the coordinate position of the target mark 5 from the robot controller 3 and transmit a result of image processing obtained by execution of the image processing (detection of the target mark 5 , for example) by the visual sensor controller 1 , etc. to the robot controller 3 .
- the ROM 13 stores programs used for execution of various types of process by the visual sensor controller 1 .
- access is made faster to a RAM than to a ROM.
- the CPU 10 may develop the programs stored in the ROM 13 in advance on the RAM 14 . Then, the CPU 10 may read the programs from the RAM 14 and execute the read programs.
- the RAM 14 stores temporarily saved data necessary for execution of the programs.
- the nonvolatile RAM 15 is a magnetic storage unit, a flash memory, an MRAM, FRAM (registered trademark), or an EEPROM, for example.
- the nonvolatile RAM 15 is an SRAM or a DRAM backed up by a battery, for example.
- the nonvolatile RAM 15 is configured as a nonvolatile memory to hold its storage state even after the visual sensor controller 1 is powered off.
- the nonvolatile RAM 15 stores setting necessary for execution of the programs, for example.
- the frame memory 12 stores image data.
- the nonvolatile RAM 15 includes a reference information storage 151 and a detection result storage 152 .
- the reference information storage 151 stores reference information (also called a “model pattern” or a “template”) indicating a subject (target mark 5 ).
- the reference information may be a group of edge points (also called an “edge point group”) in a subject (target mark 5 ), for example.
- the edge point is a point where brightness changes largely in an image.
- the edge point group may be formed by converting the subject (target mark 5 ) to an edge image through publicly-known Sobel filtering, and extracting a pixel (edge point) having an intensity of a predetermined threshold or more from the edge image.
- the edge point group extracted in this way from the image including the subject (target mark 5 ) to be detected is stored as the model pattern into the reference information storage 151 .
- the model pattern is not limited to edge points.
- feature points such as those extracted by publicly-known SIFT may be used as the model pattern.
- the model pattern may be generated by arranging a geometric graphic such as a line segment, a rectangle, or a circle so as to match the contour of the subject (target mark 5 ). In this case, feature points may be provided at proper intervals on the geometric graphic forming the contour.
- the model pattern may also be a template image generated by cutting out a part corresponding to a model pattern designation area from a captured image of the target mark 5 . Storing the model pattern in advance generated in the above-described way into the reference information storage 151 is also called “teaching the model pattern.” Teaching of the model pattern will be described later.
- the detection result storage 152 stores a result of detection of the target mark 5 detected by using the taught model pattern from data about an image captured by the camera 2 while the target mark 5 having been moved in the set calibration range is at each destination.
- the camera 2 In response to a command from the CPU 10 , the camera 2 captures an image of the subject to acquire the image, and outputs a signal about the acquired image.
- the camera interface 16 has the function of generating a synchronization signal for controlling timing of exposure for the camera 2 in response to a command from the CPU 10 , and the function of amplifying a signal received from the camera 2 .
- the camera 2 and the camera interface 16 are not limited to any particular parts but are commercially-available and common-used parts.
- the signal about the image taken from the camera 2 is A/D converted by the camera interface 16 , and then stored temporarily as digital image data through the bus 11 into the frame memory 12 .
- the CPU 10 processes the image by using data stored in the frame memory 12 , the ROM 13 , the RAM 14 , and the nonvolatile RAM 15 . Data resulting from the image processing is stored again into the frame memory 12 .
- the CPU 10 may transfer the data stored in the frame memory 12 to the monitor interface 17 and display the data on the monitor 19 in order to allow check of the substance of the data by an operator, for example.
- the external equipment interface 18 is connected to various types of external equipment.
- the external equipment interface 18 is connected to the robot controller 3 to receive a signal for triggering image processing from the robot controller 3 and supply the robot controller 3 with position information data obtained by image processing, etc.
- a keyboard or a mouse may also be connected to the external equipment interface 18 as an input unit 33 for an operator, for example.
- FIG. 4 is a block diagram showing the functional configuration of the CPU 10 in the visual sensor controller 1 .
- the CPU 10 includes a model pattern generation unit 101 , a first parameter setting unit 102 , a first detection unit 103 , a first image range setting unit 105 , a first calibration range measurement unit 106 , an image capture control unit 107 , and a calibration unit 108 .
- Each of these functional units and functional steps is realized by execution of a system program in the ROM 13 by the CPU 10 .
- step S 1 the model pattern generation unit 101 exerts control to make the camera 2 capture an image of the target mark 5 arranged in the view of the camera 2 .
- a relationship between the position of the camera 2 and that of the target mark 5 is desirably set to be the same as that during detection of the target mark 5 .
- the model pattern generation unit 101 sets an area in the captured image of the target mark 5 in the form of a rectangular area or a circular area, for example, as a model pattern designation area in which the target mark 5 appears. Further, the model pattern generation unit 101 defines a model pattern coordinate system (image coordinate system) in the model pattern designation area.
- FIG. 6 shows an example of the model pattern designation area, that of the model pattern coordinate system (image coordinate system), and that of the model pattern.
- the model pattern generation unit 101 may set an area instructed by an operator as the model pattern designation area. Alternatively, the model pattern generation unit 101 may determine spots of large brightness gradients in the image as the contour of the image of the target mark 5 , and set the model pattern designation area so as to contain the contour of the image of the target mark 5 inside the model pattern designation area.
- step S 3 the model pattern generation unit 101 extracts an edge point in the model pattern designation area as a feature point, obtains physical quantities such as the position of the edge point, the direction and the magnitude of a brightness gradient at the edge point, etc., and converts the edge point to a value expressed in the model pattern coordinate system defined in the model pattern designation area. Further, the model pattern generation unit 101 sets a point instructed by the operator as the point P where the three-dimensional position information of the target mark 5 is to be measured, and stores the point P into the reference information storage 151 .
- the model pattern generation unit 101 may set a center point of the model pattern as a point where the three-dimensional position information of the target mark 5 is to be measured, for example.
- FIG. 7 shows an example of the point P where the three-dimensional position information of the target mark 5 is to be measured.
- a center point of the target mark 5 is set as the point P where the three-dimensional position information of the target mark 5 is to be measured.
- the model pattern is not limited to edge points.
- feature points such as those extracted by publicly-known SIFT may be used as the model pattern.
- the model pattern may be generated by arranging a geometric graphic such as a line segment, a rectangle, or a circle so as to match the contour of a subject (target mark 5 ) In this case, feature points may be provided at proper intervals on the geometric graphic forming the contour.
- the model pattern may also be a template image generated by cutting out a part corresponding to the model pattern designation area from the captured image of the target mark 5 .
- step S 4 the model pattern generation unit 101 stores the generated model pattern into the reference information storage 151 .
- the model pattern is generated by using the image captured by the camera 2 .
- the following describes a parameter for detecting the target mark 5 from data about the image captured by the camera 2 by using the model pattern stored in the reference information storage 151 .
- a difference is generated in a distance between the camera 2 and the position of the target mark 5 at a destination or in an angle from an optical axis, for example.
- an image of the target mark 5 captured by the camera 2 may differ in a manner that depends on a destination of the target mark 5 in the calibration range from the model pattern generated by the model pattern generation unit 101 in terms of appearance such as a size or the occurrence of rotation or distortion.
- the first parameter setting unit 102 sets a parameter for detecting the target mark 5 from data about an image captured by the camera 2 so as to allow detection of a model pattern at any destination of the target mark 5 in the calibration range even in the above-described situation. More specifically, the parameter set by the first parameter setting unit 102 depends on a detection algorithm. For example, the parameter may be set to have a predetermined range with respect to a model in terms of a size, rotation, distortion, a position range or an angle range for detection, for example. Alternatively, the parameter may be set as a single numerical value or an on/off value. The parameter is not limited to these examples.
- the first detection unit 103 described later is configured to be capable of detecting the target mark 5 matching the model pattern from captured image data by using a single numerical value, an on/off value, or a parameter value in a predetermined range so as to allow detection of the model pattern at any destination of the target mark 5 .
- the first detection unit 103 described later becomes capable of detecting the model pattern from the captured image data by using a proper parameter value.
- the parameter may be a parameter to be applied to data about an image captured by the camera 2 .
- the first detection unit 103 is configured to detect the model pattern about the target mark 5 from image data generated by applying the parameter to the data about the image captured by the camera 2 .
- the model pattern can be detected from image data generated by smoothing the image data by a Gaussian filter.
- the parameter may be a parameter to be applied to the model pattern about the target mark 5 .
- the first detection unit 103 is configured to detect the target mark 5 from the data about the image captured by the camera 2 by using the model pattern to which the parameter is applied.
- One example of the parameter set by the first parameter setting unit 102 and applied to the model pattern may be a transformation matrix for projection transformation, affine transformation, or homothetic transformation.
- a value of the parameter is set as a single numerical value, a single transformation matrix is selectable.
- a value of the parameter is set to have a predetermined range, a transformation matrix of a predetermined range is selectable.
- the following describes specific examples applied if the parameter is set to have a predetermined range.
- the parameter may be set in such a manner that a parameter range covers a projection transformation matrix with an element having a deviation of a predetermined threshold or less from a corresponding element in a projection transformation matrix as a basis.
- a range for a rotation angle may be set based on one rotation angle.
- a range for a homothetic ratio may be set based on one homothetic ratio. For example, based on an angle between the optical axis of the camera 2 and a plane on which the target mark 5 is to move during implementation of calibration (particularly if this plane is not at a right angle to the optical axis but is tilted diagonally from the optical axis), a size range may be set from 0.9 to 1.1 for the camera 2 by using the parameter. By doing so, robust detection can be realized on the occurrence of a difference in appearance of the target mark 5 resulting from different destinations of the target mark 5 , for example.
- a parameter value for exposure time is desirably set in consideration of an angle between the camera 2 and a plane on which the target mark 5 is arranged or a relationship with illumination, for example.
- the first detection unit 103 detects the target mark 5 from the data about the image captured by the camera 2 , and measures the coordinate position of the detected target mark 5 in an image coordinate system at the camera 2 . More specifically, the first detection unit 103 selects a parameter from a single numerical value, an on/off value, or a predetermined range for the parameter. In selecting the parameter from the predetermined range, a center value in the parameter range may be selected first, for example. Then, a value shifted in the plus or minus direction from the center value may be selected as a next parameter, for example.
- the first detection unit 103 converts the data about the image captured by the camera 2 by using the selected parameter so as to allow detection of the target mark 5 from the image data. In this way, the first detection unit 103 can detect the target mark 5 from the converted image data by the publicly-known detection technique.
- the first detection unit 103 extracts a feature point from the image data to which the parameter is applied by the same method as applied for extracting a feature point from the taught model pattern, and conducts publicly-known matching between a feature point resulting from the conversion with the parameter and a feature point forming the model pattern, thereby detecting the target mark 5 .
- the first detection unit 103 may convert the model pattern about the target mark 5 by using the selected parameter.
- the first detection unit 103 can detect the target mark 5 matching the converted model pattern from the captured image data by the above-described publicly-known detection technique. More specifically, the first detection unit 103 extracts a feature point from the data about the image captured by the camera 2 by the same method as applied for extracting a feature point from the taught model pattern, and conducts publicly-known matching between the extracted feature point and a feature point in the model pattern to which the parameter is applied, thereby detecting the target mark 5 .
- the first detection unit 103 measures the coordinate position of the detected target mark 5 in the image coordinate system at the camera 2 .
- the target mark 5 is moved on a plane designated in advance.
- This plane may include multiple parallel planes or may be formed by coupling multiple partial planes.
- the first image range setting unit 105 sets an image range in the image coordinate system at the camera 2 corresponding to this calibration range. More specifically, in response to an instruction from an operator, the first image range setting unit 105 sets an image range in the image captured by the camera 2 based on the image coordinate system at the camera 2 . In this case, the first image range setting unit 105 can designate the image range as a rectangular range, for example. The first image range setting unit 105 may set the image entirely as the image range.
- the first image range setting unit 105 may limit the image range in response to an instruction from the operator so as to avoid the obstacle existing in the view of the camera 2 .
- the first image range setting unit 105 may designate the image range as a closed graphic drawn with multiple line segments.
- the first calibration range measurement unit 106 measures a calibration range (a coordinate position in the robot coordinate system showing a boundary of the calibration range, for example) as an operation range for the robot 4 corresponding to the image range set by the first image range setting unit 105 before implementation of calibration by controlling the robot 4 so as to move the target mark 5 and making the first detection unit 103 detect the target mark 5 .
- a calibration range a coordinate position in the robot coordinate system showing a boundary of the calibration range, for example
- the first calibration range measurement unit 106 determines whether or not the target mark 5 is detectable in the image range designated in the image coordinate system at the camera 2 by the first image range setting unit 105 by moving the target mark 5 and detecting the target mark 5 repeatedly, thereby measuring a calibration range (a coordinate position in the robot coordinate system showing a boundary of the calibration range, for example) as an operation range for the robot 4 (specifically, target mark 5 ) corresponding to the designated image range.
- a calibration range a coordinate position in the robot coordinate system showing a boundary of the calibration range, for example
- FIGS. 8A and 8B show flowcharts showing a flow followed by the first calibration range measurement unit 106 for measuring a coordinate position in the robot coordinate system showing a boundary of a calibration range.
- the function of the first calibration range measurement unit 106 will be described by referring to FIGS. 8A and 8B .
- a plane on which the target mark 5 is to be moved during implementation of calibration is defined as an XY plane in a three-dimensional coordinate system designated in advance. However, this plane is not limited to the XY plane in the three-dimensional coordinate system.
- step S 11 the first calibration range measurement unit 106 acquires the three-dimensional coordinate position of the target mark 5 in the robot coordinate system arranged at an initial position on a plane in the view of the camera 2 and the coordinate position of this target mark 5 in the image coordinate system at the camera 2 through the robot controller 3 and the first detection unit 103 respectively.
- the target mark 5 may be moved to the initial position in response to jog operation by an operator, for example.
- the target mark 5 may be moved to an initial position stored in the robot controller 3 .
- step S 12 the first calibration range measurement unit 106 moves the robot 4 (specifically, target mark 5 ) through the robot controller 3 on the plane from the initial position by a unit stroke in an X direction set in advance in an X-axis direction. Then, the first calibration range measurement unit 106 acquires the three-dimensional coordinate position of the target mark 5 in the robot coordinate system and the coordinate position of the target mark 5 in the image coordinate system at the camera 2 through the robot controller 3 and the first detection unit 103 respectively.
- step S 13 the first calibration range measurement unit 106 moves the robot 4 (specifically, target mark 5 ) through the robot controller 3 on the plane from the initial position by a unit stroke in a Y direction set in advance in a Y-axis direction. Then, the first calibration range measurement unit 106 acquires the three-dimensional coordinate position of the target mark 5 in the robot coordinate system and the coordinate position of the target mark 5 in the image coordinate system at the camera 2 through the robot controller 3 and the first detection unit 103 respectively.
- step S 14 the first calibration range measurement unit 106 calculates a three-dimensional vector VX (magnitude and direction) in the robot coordinate system corresponding to the X-direction unit stroke in the X-axis direction on the plane, and a two-dimensional vector vx (magnitude and direction) in the image coordinate system at the camera 2 showing the stroke of the target mark 5 .
- the first calibration range measurement unit 106 further calculates a three-dimensional vector VY (magnitude and direction) in the robot coordinate system corresponding to the Y-direction unit stroke in the Y-axis direction on the plane, and a two-dimensional vector vy (magnitude and direction) in the image coordinate system at the camera 2 showing the stroke of the target mark 5 .
- the first calibration range measurement unit 106 can acquire approximate association between the stroke in the image coordinate system at the camera 2 and the stroke in the robot coordinate system.
- the first calibration range measurement unit 106 can convert a position information in the image coordinate system to an approximate position information in the robot coordinate system by using a transformation matrix R calculated based on the following formula (1):
- a start point for the vector is not limited to the initial position.
- the vector may be started from a point to which the robot 4 has been moved from the initial position by the X-direction unit stroke in step S 12 .
- step S 15 based on the association calculated in step S 14 , the first calibration range measurement unit 106 calculates and sets the magnitude and the direction (V) of the stroke in the robot coordinate system from the initial position on the plane corresponding to the magnitude and the direction (v) of the stroke in the image coordinate system at the camera 2 from the initial position to a boundary position (corner, for example) in the image coordinate system at the camera 2 .
- the magnitude and the direction (V) are desirably set so as to increase the probability of detection of the target mark 5 in a range in the image coordinate system at the camera 2 by being multiplied by a factor less than 1. Coordinate values in the image range in the image coordinate system at the camera 2 (boundary positions including four corners, for example) are calculated in advance.
- step S 16 the first calibration range measurement unit 106 moves the target mark 5 . More specifically, based on the set magnitude and the set direction (V) of the stroke, the first calibration range measurement unit 106 moves the robot 4 (specifically, target mark 5 ) through the robot controller 3 from the initial position on the plane.
- V set direction
- step S 17 the first calibration range measurement unit 106 determines through the first detection unit 103 whether or not the target mark 5 is detectable. If the target mark 5 has been detected successfully (Yes), the flow goes to step S 18 . If the target mark 5 has not been detected successfully (No), the flow goes to step S 19 .
- step S 18 the first calibration range measurement unit 106 determines through the first detection unit 103 whether or not a distance from the coordinate position of the target mark 5 in the image coordinate system at the camera 2 to the boundary position (corner, for example) in the image coordinate system at the camera 2 is small (specifically, whether or not the distance is a predetermined threshold or less). If the distance is determined to be small (Yes), the flow goes to step S 20 . If the distance is determined not to be small (No), the flow goes to step S 22 .
- step S 19 the first calibration range measurement unit 106 sets the magnitude and the direction of the stroke again. More specifically, if the first calibration range measurement unit 106 determines that the target mark 5 has not been detected successfully during move from the initial position, the first calibration range measurement unit 106 multiplies the set magnitude and the set direction (V) of the stroke by a factor less than 1, thereby setting the magnitude and the direction (V) of the stroke again. The factor used in this step should be lower than the factor having been applied to the move from the initial position. Then, the flow goes to step S 16 .
- step S 20 the first calibration range measurement unit 106 stores a current position information of the target mark 5 in the robot coordinate system and a current position information of the target mark 5 in the image coordinate system at the camera 2 .
- step S 21 the first calibration range measurement unit 106 determines whether or not measurement has been made for all the boundary positions. If measurement has been made for all the boundary positions (Yes), the flow ends. If there is a boundary position to be subject to next measurement (if No), the flow goes to step S 15 .
- step S 22 the first calibration range measurement unit 106 sets the magnitude and the direction (V′) of the stroke from the current position on the plane in the robot coordinate system corresponding to the magnitude and the direction (v′) of the stroke from the current position to the boundary position.
- step S 23 based on the set magnitude and the set direction (V′) of the stroke from the current position, the first calibration range measurement unit 106 moves the robot 4 (specifically, target mark 5 ).
- step S 24 the first calibration range measurement unit 106 determines through the first detection unit 103 whether or not the target mark 5 is detectable. If the target mark 5 has been detected successfully (Yes), the flow goes to step S 18 . If the target mark 5 has not been detected successfully (No), the flow goes to step S 25 .
- step S 25 the first calibration range measurement unit 106 multiplies the set magnitude and the set direction (V′) of the stroke by a factor less than 1, thereby setting the magnitude and the direction (V′) of the stroke again from a position where the robot 4 existed before moving.
- the factor used in this step should be lower than the factor having been applied to the move from the position where the robot 4 existed before moving.
- the flow goes to step S 23 .
- the first calibration range measurement unit 106 can measure a coordinate position in the robot coordinate system showing a boundary of a calibration range in which the target mark 5 is to be detected.
- the visual sensor controller 1 functioning as the calibration device controls the robot 4 so as to move the target mark 5 in the calibration range.
- the arm 41 desirably moves through the set calibration range uniformly.
- the visual sensor controller 1 may control the robot 4 in such a manner that the target mark 5 moves by tracing a path shown in FIG. 9A or 9B .
- the image capture control unit 107 makes the camera 2 capture an image of the target mark 5 attached to the end of the arm 41 of the robot 4 and moved by the robot 4 in a calibration range at each of multiple destination positions during implementation of calibration.
- the number of the destinations is desirably set to be larger than a number minimum for allowing calibration (a desired number is 20 or more, for example). By doing so, calibration can be performed more precisely.
- the calibration unit 108 stores the following coordinate positions into the detection result storage 152 at each of multiple destinations of the target mark 5 attached to the end of the arm 41 of the robot 4 and moved by the robot controller 3 during implementation of calibration: the coordinate position of the target mark 5 in the image coordinate system at the camera 2 appearing in data about an image captured by the camera 2 ; and the coordinate position of the target mark 5 in the robot coordinate system while the image of the target mark 5 is captured by the camera 2 . Then, the calibration unit 108 calibrates the camera 2 based on the coordinate position of the target mark 5 in the image coordinate system at the camera 2 stored in the detection result storage 152 , and the coordinate position of the target mark 5 in the robot coordinate system while the image of the target mark 5 is captured by the camera 2 stored in the detection result storage 152 .
- FIG. 10 is a flowchart showing calibration process on the camera 2 executed by the visual sensor controller 1 (CPU 10 ) according to this embodiment.
- step S 31 a model pattern is generated and a parameter is set.
- step S 32 the CPU 10 (first image range setting unit 105 ) sets an image range (corresponding to a calibration range) for the camera 2 in the image coordinate system in response to designation by an operator.
- the operator can designate the image range (corresponding to the calibration range) in an image captured by the camera 2 .
- step S 33 the first calibration range measurement unit 106 measures a coordinate position in the robot coordinate system showing a boundary of the calibration range. More specifically, as described above, the first calibration range measurement unit 106 determines whether or not the target mark 5 is detectable in the image range by moving the target mark 5 and detecting the target mark 5 repeatedly, thereby measuring the calibration range (a coordinate position in the robot coordinate system showing a boundary of an operation range for the robot 4 , for example). (The process in step S 33 is performed based on the above-described process flow followed by the first calibration range measurement unit 106 .)
- step S 34 the CPU 10 (calibration unit 108 ) sets a measurement counter for counting the number of times measurement is made at 1 .
- step S 35 the CPU 10 (calibration unit 108 ) acquires the three-dimensional coordinate position of the target mark 5 in the robot coordinate system measured by the robot controller 3 .
- step S 36 the CPU 10 (first detection unit 103 ) detects the target mark 5 from data about the image captured by the camera 2 , and measures the coordinate position of the detected target mark 5 in the image coordinate system at the camera 2 .
- step S 37 the CPU 10 (calibration unit 108 ) stores the three-dimensional coordinate position of the target mark 5 in the robot coordinate system and the coordinate position of the target mark 5 in the image coordinate system at the camera 2 in association with each other while the target mark 5 is at a current position.
- step S 38 the CPU 10 (calibration unit 108 ) increments the measurement counter for counting the number of times measurement is made by 1.
- step S 39 it is determined whether or not the measurement counter exceeds a predetermined value. If the measurement counter exceeds the predetermined value (Yes), the flow goes to step S 41 . If the measurement counter does not exceed the predetermined value (No), the flow goes to step S 40 .
- step S 40 the robot controller 3 moves the target mark 5 attached to the end of the arm 41 of the robot 4 to a place in the calibration range where the target mark 5 can be measured from the camera 2 . Then, the flow goes to step S 35 .
- step S 41 the CPU 10 (calibration unit 108 ) calibrates the camera 2 based on the association stored in step S 37 between the three-dimensional coordinate position of the target mark 5 in the robot coordinate system and the coordinate position of the target mark 5 in the image coordinate system at the camera 2 .
- the visual sensor controller 1 of the first embodiment includes: the first image range setting unit 105 that sets an image range in the image coordinate system at the camera 2 ; and the first calibration range measurement unit 106 that measures a calibration range as an operation range for the robot 4 corresponding to the set image range before implementation of calibration by controlling the robot 4 to move the target mark 5 and detecting the target mark 5 .
- an operation range for the robot 4 (calibration range) corresponding to a range of move of the target mark 5 is measured in advance before implementation of calibration based on a view range for a single visual sensor, so that the calibration can be performed with an increased degree of freedom.
- the target mark 5 is allowed to move in a range covering the view of the camera 2 entirely, so that the calibration can be performed more precisely. If a plane on which the target mark 5 is to move includes an obstacle, for example, an image range is designated in response to an instruction from an operator so as to avoid the obstacle existing in the view of the camera 2 . By doing so, the calibration range can be set efficiently.
- a calibration range can be set on a plane.
- the plane on which the target mark 5 is to be moved during implementation of calibration can be a plane vertical to the optical axis of the camera 2 or a plane tilted from the optical axis of the camera 2 as shown in FIG. 11A .
- a calibration range can be set on each of at least two planes.
- a calibration range can be extended by making such settings. This increases the degree of freedom further in calibration, so that the calibration is performed more precisely.
- a calibration method of the first embodiment and a program of the first embodiment achieve effects comparable to those achieved by the visual sensor controller 1 of the first embodiment.
- the first embodiment of the present invention has been described, but the present invention is not limited to the above-described first embodiment.
- the effects described in the first embodiment are merely a list of most preferred effects resulting from the present invention. Effects achieved by the present invention are not limited to those described in the first embodiment.
- the model pattern generation unit 101 is configured to generate a model pattern by using the camera 2 and store the generated model pattern.
- the model pattern generation unit 101 may be configured to use an existing shape (a shape such as a circle, for example) as the target mark 5 .
- the target mark 5 is moved on a plane designated in advance and a calibration range is measured on this plane.
- the calibration range may be defined in any three-dimensional space. This is expected to increase the degree of freedom further in calibration.
- the plane to be designated in advance may include multiple parallel planes or may be formed by coupling multiple partial planes.
- the plane to be designated may include at least two planes. If the plane includes multiple planes, the first calibration range measurement unit 106 measures a coordinate position in the robot coordinate system showing a boundary of a calibration range on each of the planes.
- two planes may be defined in a three-dimensional coordinate system.
- a calibration range may be set as space inside a hexahedron defined by a point of intersection between each plane and a line of sight pointing to a corner in an image captured by the camera 2 .
- the first calibration range measurement unit 106 measures a coordinate position in the robot coordinate system showing a boundary of a calibration range in this hexahedron.
- a visual sensor controller 1 A functioning as a calibration device sets a calibration range as an operation range for a robot 4 for allowing image capture of a target mark 5 in an image range for a stereo camera 2 A including at least a first camera 21 and a second camera 22 after the target mark 5 is moved.
- the description of the second embodiment does not cover structures and functions common to those of the first embodiment but is directed to issues peculiar to the second embodiment.
- FIG. 12 shows the configuration of a robot system 1000 entirely for calibration on a stereo camera.
- the robot system 1000 is for performing calibration using the stereo camera 2 A with the first camera 21 and the second camera 22 .
- the number of cameras forming the stereo camera 2 A is not limited to two but can be any number of two or more. Each of the cameras forming the stereo camera 2 A is certainly applicable as a single camera.
- the stereo camera 2 A is fixed to a pedestal (not shown in the drawings).
- the first camera 21 and the second camera 22 may be arranged parallel to each other.
- each of the first camera 21 and the second camera 22 may be arranged at a tilt.
- Tilting each of the first camera 21 and the second camera 22 makes it possible to increase an area of an overlap between an area of image captured by the first camera 21 and an area of image captured by the second camera 22 , compared to arranging the first camera 21 and the second camera 22 parallel to each other. Specifically, tilting each of the first camera 21 and the second camera 22 makes it possible to increase an area where three-dimensional measurement is allowed by the stereo camera 2 A, compared to arranging the first camera 21 and the second camera 22 parallel to each other.
- the first camera 21 and the second camera 22 desirably have the same configuration in terms of a view range, a lens, etc. By doing so, the first camera 21 and the second camera 22 are expected to catch the target mark 5 in the same way.
- the stereo camera 2 A is connected to the visual sensor controller 1 A.
- the visual sensor controller 1 A makes the first camera 21 and the second camera 22 capture images of the target mark 5 , and calibrates each of the first camera 21 and the second camera 22 .
- the functional configuration of the visual sensor controller 1 A functioning as the calibration device is the same as the functional configuration of the visual sensor controller 1 of the first embodiment ( FIG. 3 ), except that the stereo camera 2 A (first camera 21 and second camera 22 ) is connected to the bus 11 through the camera interface 16 .
- the functional configuration of the visual sensor controller 1 A can be understood by referring to FIG. 3 mentioned above.
- FIG. 14 is a block diagram showing the functional configuration of a CPU 10 in the visual sensor controller 1 A.
- the CPU 10 in the visual sensor controller 1 A includes a model pattern generation unit 101 , a first parameter setting unit 102 , a first detection unit 103 , a first image range setting unit 105 , a first calibration range measurement unit 106 , a second parameter setting unit 1022 , a second detection unit 1032 , a second image range setting unit 1052 , a second calibration range measurement unit 1062 , an image capture control unit 107 , and a calibration unit 108 .
- the model pattern generation unit 101 generates a model pattern by making the first camera 21 capture an image of the target mark 5 arranged in the view of the first camera 21 .
- the model pattern generated by using the image captured by the first camera 21 is also used as a model pattern for the second camera 22 .
- a model pattern for the second camera 22 may be generated individually by making the second camera 22 capture an image of the target mark 5 arranged in the view of the second camera 22 .
- the first parameter setting unit 102 sets a first parameter for detecting a model pattern about the target mark 5 from data about an image captured by the first camera 21 .
- the function of the first parameter setting unit 102 is comparable to that of the first parameter setting unit 102 in the first embodiment.
- the following description is for the second parameter setting unit 1022 that uses the model pattern about the target mark 5 generated by using the first camera 21 to set a second parameter for detecting this model pattern from data about an image captured by the second camera 22 .
- the second parameter setting unit 1022 sets the second parameter for detecting the model pattern about the target mark 5 from the data about the image captured by the second camera 22 based on the first parameter. More specifically, the second parameter setting unit 1022 uses the first parameter as it is for initially setting a value of the second parameter. Meanwhile, if a value of the second parameter is set to have a predetermined range during initial setting of the second parameter, for example, the second parameter setting unit 1022 may employ the same range as the first parameter.
- the second parameter setting unit 1022 may set a wide range for the second parameter covering a predetermined range for the first parameter set by the first parameter setting unit 102 .
- the second detection unit 1032 described later has detected the model pattern about the target mark 5 successfully from the data about the image captured by the second camera 22 by applying a given value of the second parameter
- the second parameter setting unit 1022 can set a range for a value of the second parameter again with respect to this value of the second parameter as a center based on a deviation from a center value in the predetermined range for the first parameter. For example, if the first parameter has a size range from 0.9 to 1.1, a center value is 1.0 and a deviation from the center value in the predetermined range for the first parameter is 0.1.
- a center value of the second parameter is set at 0.95 and the deviation in the first parameter is applied to the second parameter.
- a value of the second parameter is set in a range [0.85 to 1.05] with respect to 0.95 as a center value. In this way, the range for a value of the second parameter during initial setting of the second parameter can be readjusted, so that the model pattern can be detected more efficiently from the data about the image captured by the second camera 22 .
- the second parameter is not limited to these examples.
- the first detection unit 103 detects the model pattern about the target mark 5 from the data about the image captured by the first camera 21 , and measures the coordinate position of the detected target mark 5 in an image coordinate system at the first camera 21 .
- the function of the first detection unit 103 is comparable to that of the first detection unit 103 in the first embodiment.
- the second detection unit 1032 detects the model pattern about the target mark 5 from the data about the image captured by the second camera 22 , and measures the coordinate position of the detected target mark 5 in an image coordinate system at the second camera 22 . Detection process by the second detection unit 1032 can be understood by replacing the camera 2 , the first parameter setting unit 102 , and the parameter in the description given above in the first embodiment relating to the first detection unit 103 by the second camera 22 , the second parameter setting unit 1022 , and the second parameter respectively.
- the first image range setting unit 105 sets a first image range in the image captured by the first camera 21 based on the image coordinate system at the first camera 21 .
- the first image range may cover the captured image entirely.
- the first image range setting unit 105 can designate the first image range as a rectangular range, for example. If a plane on which the target mark 5 attached to an end portion of an arm 41 is to move includes an obstacle, for example, the first image range setting unit 105 may limit the first image range in response to an instruction from the operator so as to avoid the obstacle existing in the view of the first camera 21 . In this case, the first image range setting unit 105 may designate the first image range as a closed graphic drawn with multiple line segments. The function of the first image range setting unit 105 is comparable to that of the first image range setting unit 105 in the first embodiment.
- the second image range setting unit 1052 sets a second image range in the image captured by the second camera 22 based on the image coordinate system at the second camera 22 .
- Process by the second image range setting unit 1052 can be understood by replacing the camera 2 in the description given above in the first embodiment relating to the first image range setting unit 105 by the second camera 22 .
- the first calibration range measurement unit 106 measures a first calibration range as an operation range for the robot 4 corresponding to the first image range set by the first image range setting unit 105 before implementation of calibration by controlling the robot 4 so as to move the target mark 5 and making the first detection unit 103 detect the target mark 5 . More specifically, the first calibration range measurement unit 106 determines whether or not the target mark 5 is detectable in the first image range by the first camera 21 by moving the target mark 5 and detecting the target mark 5 repeatedly, thereby measuring the first calibration range as an operation range for the robot 4 corresponding to the first image range.
- the function of the first calibration range measurement unit 106 is comparable to that of the first calibration range measurement unit 106 in the first embodiment.
- the second calibration range measurement unit 1062 determines whether or not the target mark 5 is detectable in the second image range by the second camera 22 by moving the target mark 5 and detecting the target mark 5 repeatedly, thereby measuring a second calibration range as an operation range for the robot 4 corresponding to the second image range.
- Process by the second calibration range measurement unit 1062 can be understood by replacing the camera 2 , the first parameter setting unit 102 , the parameter, and the image range in the description given above in the first embodiment relating to the first calibration range measurement unit 106 by the second camera 22 , the second parameter setting unit 1022 , the second parameter, and the second image range respectively.
- the target mark 5 is moved on a plane designated in advance in this embodiment.
- the first calibration range and the second calibration range are set on the same plane.
- a flowchart about process of measuring the first calibration range followed by the first calibration range measurement unit 106 can be understood by the replacing the camera 2 , the first parameter setting unit 102 , the parameter, and the image range referred to in the description in the first embodiment for the flowchart relating to the calibration range measurement unit 106 ( FIGS. 8A and 8B ) by the first camera 21 , the first parameter setting unit 102 , the first parameter, and the first image range respectively, and by the second camera 22 , the second parameter setting unit 1022 , the second parameter, and the second image range respectively.
- the visual sensor controller 1 A functioning as the calibration device controls the robot 4 so as to move the target mark 5 in at least one of the first calibration range and the second calibration range, or in a range covered by both the first calibration range and the second calibration range.
- the target mark 5 may be moved only in a range covered by both the first calibration range and the second calibration range, only in a range covered by either the first calibration range or the second calibration range, or in a range including the first calibration range and the second calibration range.
- the image capture control unit 107 makes the first camera 21 and the second camera 22 capture images of the target mark 5 at each of multiple destination positions attached to the end of the arm 41 of the robot 4 and to be moved by the robot 4 in at least one of the first calibration range and the second calibration range, or in a range covered by both the first calibration range and the second calibration range during implementation of calibration.
- the number of the destinations is desirably set to be larger than a number minimum for allowing calibration (a desired number is 20 or more, for example) on each of the first camera 21 and the second camera 22 . By doing so, calibration can be performed more precisely.
- the calibration unit 108 stores the following coordinate positions into the detection result storage 152 at each of multiple destinations of the target mark 5 moved in at least one of the first calibration range and the second calibration range, or in a range covered by both the first calibration range and the second calibration range during implementation of calibration: the coordinate position of the target mark 5 in the image coordinate system at the first camera 21 ; the coordinate position of the target mark 5 in the image coordinate system at the second camera 22 ; and the coordinate position of the target mark 5 in the robot coordinate system while the images of the target mark 5 are captured by the first camera 21 and the second camera 22 .
- the calibration unit 108 calibrates the first camera 21 and the second camera 22 based on the coordinate position of the target mark 5 in the image coordinate system at the first camera 21 stored in the detection result storage 152 , the coordinate position of the target mark 5 in the image coordinate system at the second camera 22 stored in the detection result storage 152 , and the coordinate position of the target mark 5 in the robot coordinate system while the images of the target mark 5 are captured by the first camera 21 and the second camera 22 stored in the detection result storage 152 .
- the first camera 21 and the second camera 22 may be calibrated individually.
- FIG. 15 is an example of a flowchart showing calibration process on the first camera 21 and the second camera 22 performed by the visual sensor controller 1 A (CPU 10 ) according to this embodiment. This flowchart shows process performed if the target mark 5 is to be moved in a range including the first calibration range and the second calibration range.
- step S 51 a model pattern is generated. Further, the first parameter and the second parameter are set.
- step S 52 in response to designation by an operator, the CPU 10 (first image range setting unit 105 and second image range setting unit 1052 ) sets the first image range in an image captured by the first camera 21 in the image coordinate system at the first camera 21 , and sets the second image range in an image captured by the second camera 22 in the image coordinate system at the second camera 22 .
- step S 53 the CPU 10 (first calibration range measurement unit 106 and second calibration range measurement unit 1062 ) measures a coordinate position in the robot coordinate system showing a boundary of the first calibration range corresponding to the first image range for the first camera 21 , and measures a coordinate position in the robot coordinate system showing a boundary of the second calibration range corresponding to the second image range for the second camera 22 .
- the first calibration range and the second calibration range are set to allow calibration process on the first camera 21 and the second camera 22 to be started.
- step S 54 the CPU 10 (calibration unit 108 ) sets a measurement counter for counting the number of times measurement is made at 1 .
- step S 55 the CPU 10 (calibration unit 108 ) acquires the three-dimensional coordinate position of the target mark 5 in the robot coordinate system measured by the robot controller 3 .
- step S 56 the CPU 10 (first detection unit 103 and second detection unit 1032 ) detects the target mark 5 from data about the image captured by the first camera 21 and from data about the image captured by the second camera 22 , and measures the coordinate position of the detected target mark 5 in each of the image coordinate system at the first camera 21 and the image coordinate system at the second camera 22 .
- the target mark 5 may be detected only from the first camera 21 and the second camera 22 . In this case, only a coordinate position in an image coordinate system at a camera having detected the target mark 5 successfully is stored.
- step S 57 the CPU 10 (calibration unit 108 ) stores the three-dimensional coordinate position of the target mark 5 in the robot coordinate system, the coordinate position of the target mark 5 in the image coordinate system at the first camera 21 , and the coordinate position of the target mark 5 in the image coordinate system at the second camera 22 in association with each other while the target mark 5 is at a current position.
- step S 58 the CPU 10 (calibration unit 108 ) increments the measurement counter for counting the number of times measurement is made by 1.
- step S 59 if the measurement counter does not exceed a predetermined value (No), the flow goes to step S 60 . If the measurement counter exceeds the predetermined value (Yes), the flow goes to step S 61 .
- step S 60 the robot controller 3 moves the target mark 5 attached to the end of the arm 41 of the robot 4 to a place set in advance in the first calibration range or the second calibration range where the target mark 5 can be measured from at least one of the first camera 21 and the second camera 22 . Then, the flow goes to step S 55 .
- step S 61 the CPU 10 (calibration unit 108 ) calibrates the first camera 21 and the second camera 22 based on the association stored in step S 57 between the three-dimensional coordinate position of the target mark 5 in the robot coordinate system, the coordinate position of the target mark 5 in the image coordinate system at the first camera 21 , and the coordinate position of the target mark 5 in the image coordinate system at the second camera 22 .
- This process flow is not restrictive but has been described as an example. For example, if the target mark 5 is to be moved only in a range covered by both the first calibration range and the second calibration range, steps S 56 and S 60 can be replaced as follows.
- step S 56 only if the target mark 5 is detected from both the first camera 21 and the second camera 22 , the coordinate position of the target mark 5 in the image coordinate system at the first camera 21 and the coordinate position of the target mark 5 in the image coordinate system at the second camera 22 may be stored.
- step S 60 the robot controller 3 moves the target mark 5 attached to the end of the arm 41 of the robot 4 to a place set in advance in the first calibration range and the second calibration range where the target mark 5 can be measured from both the first camera 21 and the second camera 22 .
- the visual sensor controller 1 A of the second embodiment includes: the first image range setting unit 105 that sets the first image range in the image coordinate system at the first camera 21 ; the first calibration range measurement unit 106 that measures the first calibration range as an operation range for the robot 4 corresponding to the first image range before implementation of calibration by controlling the robot 4 to move the target mark 5 and detecting the target mark 5 ; the second image range setting unit 1052 that sets the second image range in the image coordinate system at the second camera 22 ; and the second calibration range measurement unit 1062 that measures the second calibration range as an operation range for the robot 4 corresponding to the second image range before implementation of the calibration by controlling the robot 4 to move the target mark 5 and detecting the target mark 5 .
- an operation range for the robot 4 (first calibration range and second calibration range) corresponding to a range of move of the target mark 5 is measured in advance before implementation of the calibration based on view ranges for multiple cameras (visual sensors), so that the calibration can be performed with an increased degree of freedom.
- the target mark 5 is allowed to move in a maximum range for each camera, so that the calibration can be performed more precisely. If space in which the target mark 5 is to move includes an obstacle, for example, the first image range and the second image range are designated in response to an instruction from an operator so as to avoid the obstacle. By doing so, the operation range for the robot 4 (first calibration and second calibration range) can be set efficiently.
- Each of the first calibration range and the second calibration range can be configured to be set on a plane.
- a calibration method of the second embodiment and a program of the second embodiment achieve effects comparable to those achieved by the visual sensor controller 1 A of the second embodiment.
- the present invention is not limited to the above-described second embodiment.
- the effects described in the second embodiment are merely a list of most preferred effects resulting from the present invention. Effects achieved by the present invention are not limited to those described in the second embodiment.
- the stereo camera 2 A includes two cameras.
- the stereo camera 2 A may be configured to include three or more cameras.
- the model pattern generation unit 101 is configured to generate a model pattern by using the first camera 21 and store the generated model pattern.
- the model pattern generation unit 101 may be configured to generate a model pattern by using the second camera 22 and store the generated model pattern.
- the model pattern generation unit 101 is configured to generate and store a model pattern.
- the model pattern generation unit 101 may be configured to use an existing shape (a shape such as a circle, for example) as the target mark 5 .
- the target mark 5 is moved on a plane designated in advance. Further, the first calibration range and the second calibration range are measured on this plane. Alternatively, the first calibration range and the second calibration range may be defined in any three-dimensional space.
- the plane to be designated in advance may include multiple parallel planes or may be formed by coupling multiple partial planes.
- the visual sensor controller 1 or 1 A functions as the calibration device.
- the calibration device is not limited to the visual sensor controller 1 or 1 A.
- the calibration device may be a controller including the visual sensor controller 1 or 1 A and the robot controller 3 integrated with each other.
- the calibration device may cover information processing devices (computers) in general.
- the calibration device may be a server, a PC, various types of controllers, etc.
- the calibration method implemented by the visual sensor controller 1 or 1 A is realized by software.
- programs constituting the software are installed on a computer (visual sensor controller 1 ). These programs may be stored in a removable medium and then distributed to a user. Alternatively, these programs may be distributed by being downloaded onto a computer of the user through a network.
Abstract
Description
- This application is based on and claims the benefit of priority from Japanese Patent Application No. 2017-003667, filed on 12 Jan. 2017, the content of which is incorporated herein by reference.
- The present invention relates to calibration on a visual sensor, particularly, to a calibration device, a calibration method, and computer readable medium for measuring a range in which a target mark is detected during implementation of calibration using a stereo camera with multiple cameras.
- In a robot system, a robot is given a visual function. Operation such as handling or machining on a work is done by making the robot recognize the position information of a subject. This visual function is fulfilled by capturing an image of the subject with a visual sensor attached to a hand or a neighboring part of the robot or a visual sensor provided around the robot. In such a robot system, to acquire the position information of the subject viewed from the robot, calibration data is required for converting the position information of the subject in an image to the position information of the subject viewed from the robot.
- Calibration data has been acquired by various conventional methods. For example,
patent document 1 suggests a method of attaching a lattice pattern to the end of a robot arm and measuring the pattern with a fixedly arranged visual sensor (this method will be called a “method A”).Patent document 2 suggests a method of performing calibration by attaching a target mark having a position and a posture determined in advance in an end point coordinate system at a robot to the end of an arm, and determining the position of the target mark at multiple points in an image of the target mark captured by a visual sensor (this method will be called a “method B”). - If calibration is to be performed by the method A, a pattern to be used for the calibration should be prepared in advance. If the view of a camera is too wide or too narrow for the prepared pattern, the method A fails to perform high-precision calibration. In contrast, calibration by the method B allows calibration in a wider view or calibration in a narrower view than in the method A, thereby advantageously increasing the degree of freedom of the calibration. For three-dimensional measurement, a stereo camera has been used in some cases as a visual sensor as described in
patent document 3, for example. There have been the following systems for the stereo camera: a passive stereo system of matching corresponding points by using the texture of a subject; and an active stereo system of matching corresponding points by using a pattern projected on a subject. In either case, calibration is required on two or more cameras forming the stereo camera. - Patent Document 1: Japanese Patent No. 2690603
- Patent Document 2: Japanese Unexamined Patent Application, Publication No. 2015-174191
- Patent Document 3: Japanese Unexamined Patent Application, Publication No. 2010-172986
- To calibrate a visual sensor (camera, for example) more precisely, a target mark is desirably moved through a wide area in a view range while the visual sensor (camera, for example) is calibrated. If the target mark is to collide with an obstacle while being moved in the view range, for example, a range for move of the target mark should be limited. Hence, before the visual sensor (camera, for example) is calibrated, a range of move of the target mark (hereinafter also called a “calibration range”) for capturing an image of the target mark having been moved is required to be set in some way in a plane on which the target mark is to be moved during implementation of the calibration so as to allow the target mark to be moved through a wide area in the view range. The calibration range can also be designated in a robot coordinate system. However, to designate a range in the robot coordinate system for making the target mark come into the view and preventing the target mark from colliding with an obstacle, a user is required to check the target mark in an image of the target mark captured by the camera while operating a robot by hand. To eliminate the need for such troublesome operation, the calibration range is desirably set in a captured image. For calibration on multiple cameras such as those of a stereo camera, the multiple cameras are attached at separate positions, so that each camera is required to be calibrated independently. Hence, also in this case, before each camera is calibrated, a range of move of the target mark (calibration range) is required to be set for each of a
first camera 21 and asecond camera 22 in a plane on which the target mark is to be moved in such a manner that an image of the target mark having been moved can be captured in a range of image capture (angle of view) by each camera. - In this regard,
patent document 2 does not recite that a range of move of a work 1 (calibration range) is set on a plane on which thework 1 is to move and in a range of image capture (angle of view) by an image capture unit 40 before the image capture unit 40 is calibrated. Further, the calibration described inpatent document 2 is not to calibrate multiple cameras such as those of a stereo camera but is merely to calibrate a single camera. The calibration described inpatent document 3 is performed by attaching a checker board as a basic matrix calculation tool to the end of a robot arm and capturing images of the checker board with a stereo camera. Thus, this calibration inherently does not correspond to calibration performed by attaching a target mark to the end of an arm, and determining the position of the target mark at multiple points where the target mark has been moved in the images of the target mark captured by the stereo camera. - The present invention provides a calibration device, a calibration method, and a program capable of setting a range of move of a target mark (calibration range) in advance in space for move of the target mark so as to make the range of move fall in a view range for a single visual sensor (camera, for example) or in a view range for each camera forming a stereo camera.
- (1) A calibration device according to the present invention (“
visual sensor controller 1” described later, for example) is a calibration device that associates a robot coordinate system at a robot (“robot 4” described later, for example) and an image coordinate system at a camera (“camera 2” described later, for example) by placing a target mark (“target mark 5” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the camera. The calibration device comprises: an image range setting unit (“first imagerange setting unit 105” described later, for example) that sets an image range in the image coordinate system; and a calibration range measurement unit (“first calibrationrange measurement unit 106” described later, for example) that measures a calibration range as an operation range for the robot corresponding to the image range before implementation of calibration by controlling the robot to move the target mark and detecting the target mark. During implementation of the calibration, the robot is controlled so as to move the target mark in the calibration range. - (2) In the calibration device described in (1), the calibration range may be configured to be set on a plane.
- (3) In the calibration device described in (1) or (2), the calibration range may be configured to be set on a plane vertical to an optical axis of the camera.
- (4) In the calibration device described in (1) or (2), the calibration range may be configured to be set on a plane tilted from an optical axis of the camera.
- (5) In the calibration device described in (1) or (4), the calibration range may be configured to be set on each of at least two planes.
- (6) A calibration method according to the present invention (“visual sensor control method” described later, for example) is a calibration method implemented by a calibration device that associates a robot coordinate system at a robot (“
robot 4” described later, for example) and an image coordinate system at a camera (“camera 2” described later, for example) by placing a target mark (“target mark 5” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the camera. The calibration method comprises: an image range setting step (“first image range setting step” described later, for example) of setting an image range in the image coordinate system; and a calibration range measurement step (“calibration range measurement step” described later, for example) of measuring a calibration range as an operation range for the robot corresponding to the image range before implementation of calibration by controlling the robot to move the target mark and detecting the target mark. During implementation of the calibration, the robot is controlled so as to move the target mark in the calibration range. - (7) A program according to the present invention (“program” described later, for example) causes a computer to execute the following steps. The computer controls a calibration device that associates a robot coordinate system at a robot (“
robot 4” described later, for example) and an image coordinate system at a camera (“camera 2” described later, for example) by placing a target mark (“target mark 5” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the camera. The steps comprise: an image range setting step (“first image range setting step” described later, for example) of setting an image range in the image coordinate system; and a calibration range measurement step (“calibration range measurement step” described later, for example) of measuring a calibration range as an operation range for the robot corresponding to the image range before implementation of calibration by controlling the robot to move the target mark and detecting the target mark. During implementation of the calibration, the robot is controlled so as to move the target mark in the calibration range. - (8) A calibration device according to the present invention (“visual sensor controller 1A” described later, for example) is a calibration device that associates a robot coordinate system at a robot (“
robot 4” described later, for example), position information in an image coordinate system at a first camera (“first camera 21” described later, for example) of a stereo camera (“stereo camera 2A” described later, for example), and position information in an image coordinate system at a second camera (“second camera 22” described later, for example) of the stereo camera by placing a target mark (“target mark 5” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the stereo camera including at least the first camera and the second camera. The calibration device comprises: a first image range setting unit (“first imagerange setting unit 105” described later, for example) that sets a first image range in the image coordinate system at the first camera; a second image range setting unit (“second imagerange setting unit 1052” described later, for example) that sets a second image range in the image coordinate system at the second camera; a first calibration range measurement unit (“first calibrationrange measurement unit 106” described later, for example) that measures a first calibration range as a first operation range for the robot corresponding to the first image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the first camera; and a second calibration range measurement unit (“second calibrationrange measurement unit 1062” described later, for example) that measures a second calibration range as a second operation range for the robot corresponding to the second image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the second camera. While the first camera and the second camera are calibrated, the robot is controlled so as to move the target mark in at least one of the first calibration range and the second calibration range, or in a range covered by both the first calibration range and the second calibration range. - (9) In the calibration device described in (8), each of the first calibration range and the second calibration range may be configured to be set on a plane.
- (10) A calibration method according to the present invention (“visual sensor control method” described later, for example) is a calibration method implemented by a calibration device that associates a robot coordinate system at a robot (“
robot 4” described later, for example), position information in an image coordinate system at a first camera (“first camera 21” described later, for example) of a stereo camera (“stereo camera 2A” described later, for example), and position information in an image coordinate system at a second camera (“second camera 22” described later, for example) of the stereo camera by placing a target mark (“target mark 5” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the stereo camera including at least the first camera and the second camera. The calibration method comprises: a first image range setting step (“first image range setting step” described later, for example) of setting a first image range in the image coordinate system at the first camera; a second image range setting step (“second image range setting step” described later, for example) of setting a second image range in the image coordinate system at the second camera; a first calibration range measurement step (“first calibration range measurement step” described later, for example) of measuring a first calibration range as a first operation range for the robot corresponding to the first image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the first camera; and a second calibration range measurement step (“second calibration range measurement step” described later, for example) of measuring a second calibration range as a second operation range for the robot corresponding to the second image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the second camera. While the first camera and the second camera are calibrated, the robot is controlled so as to move the target mark in at least one of the first calibration range and the second calibration range, or in a range covered by both the first calibration range and the second calibration range. - (11) A program according to the present invention (“program” described later, for example) causes a computer to execute the following steps. The computer controls a calibration device that associates a robot coordinate system at a robot (“
robot 4” described later, for example), position information in an image coordinate system at a first camera (“first camera 21” described later, for example) of a stereo camera (“stereo camera 2A” described later, for example), and position information in an image coordinate system at a second camera (“second camera 22” described later, for example) of the stereo camera by placing a target mark (“target mark 5” described later, for example) at the robot, controlling the robot so as to move the target mark, and detecting the target mark at multiple points in a view of the stereo camera including at least the first camera and the second camera. The steps comprise: a first image range setting step (“first image range setting step” described later, for example) of setting a first image range in the image coordinate system at the first camera; a second image range setting step (“second image range setting step” described later, for example) of setting a second image range in the image coordinate system at the second camera; a first calibration range measurement step (“first calibration range measurement step” described later, for example) of measuring a first calibration range as a first operation range for the robot corresponding to the first image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the first camera; and a second calibration range measurement step (“second calibration range measurement step” described later, for example) of measuring a second calibration range as a second operation range for the robot corresponding to the second image range before the first camera and the second camera are calibrated by controlling the robot to move the target mark and detecting the target mark by using the second camera. While the first camera and the second camera are calibrated, the robot is controlled so as to move the target mark in at least one of the first calibration range and the second calibration range, or in a range covered by both the first calibration range and the second calibration range. - A calibration device, a calibration method, and a program provided by the present invention are capable of performing calibration with an increased degree of freedom by measuring an operation range for a robot (calibration range) corresponding to a range of move of a target mark in advance before implementation of the calibration in space where the target mark is to be moved during implementation of the calibration, based on a view range for a single visual sensor or a view range for each camera forming a stereo camera.
-
FIG. 1 shows the configuration of arobot system 1000 entirely; -
FIG. 2 shows examples of atarget mark 5; -
FIG. 3 is a functional block diagram showing the functional configuration of avisual sensor controller 1; -
FIG. 4 is a block diagram showing the functional configuration of aCPU 10 in thevisual sensor controller 1; -
FIG. 5 shows a flowchart for generating a model pattern; -
FIG. 6 shows an example of a model pattern designation area; -
FIG. 7 shows an example of a point P where the three-dimensional position information of thetarget mark 5 is to be measured; -
FIG. 8A shows a flowchart showing a flow followed by a first calibrationrange measurement unit 106 for measuring a coordinate position information in a robot coordinate system showing a boundary of a calibration range; -
FIG. 8B shows a flowchart showing the flow followed by the first calibrationrange measurement unit 106 for measuring the coordinate position information in the robot coordinate system showing the boundary of the calibration range; -
FIG. 9A shows an example of a path traced by thetarget mark 5 attached to an end portion of anarm 41 while thetarget mark 5 is moved in a calibration range; -
FIG. 9B shows an example of a path traced by thetarget mark 5 attached to the end portion of thearm 41 while thetarget mark 5 is moved in the calibration range; -
FIG. 10 is a flowchart showing calibration process on acamera 2 according to a first embodiment; -
FIG. 11A shows an example of a plane on which a calibration range is set; -
FIG. 11B shows examples of planes on which a calibration range is set; -
FIG. 12 shows the configuration of arobot system 1000 entirely for performing calibration by using astereo camera 2A according to a second embodiment; -
FIG. 13A shows an example of arrangement of thestereo camera 2A; -
FIG. 13B shows an example of arrangement of thestereo camera 2A; -
FIG. 14 is a block diagram showing the functional configuration of aCPU 10 in a visual sensor controller 1A; and -
FIG. 15 shows a flowchart showing calibration process on thestereo camera 2A according to the second embodiment. - An example of an embodiment (first embodiment) of the present invention will be described below.
FIG. 1 shows the configuration of arobot system 1000 entirely for calibration on a visual sensor. Therobot system 1000 is for performing calibration using asingle camera 2. As shown inFIG. 1 , therobot system 1000 includes: avisual sensor controller 1 that makes three-dimensional measurement through image processing on data about an image captured by thesingle camera 2; arobot 4 having anarm 41 with an end portion to which atarget mark 5 is attached; and arobot controller 3 for control over therobot 4. Thecamera 2 is fixed to a pedestal (not shown in the drawings). In this embodiment, thevisual sensor controller 1 is described as an example of a calibration device. -
FIG. 2 shows examples of thetarget mark 5. Thetarget mark 5 is not limited to these examples. Any shape is applicable to thetarget mark 5. Meanwhile, the shape of thetarget mark 5 is desirably such that the characteristics of thetarget mark 5 used as a model pattern are expressed on a two-dimensional plane. Thetarget mark 5 may be a mark printed on paper or a seal and attached to the end of thearm 41 of therobot 4, for example. - The
robot controller 3 recognizes the coordinate position information of the end of thearm 41 in a robot coordinate system as a current position. Thus, based on the coordinate position of the end of thearm 41 in the robot coordinate system and the known three-dimensional position information and the known three-dimensional posture of thetarget mark 5 in an end point coordinate system at therobot 4, therobot controller 3 can always recognize the coordinate position of thetarget mark 5 in the robot coordinate system while therobot controller 3 controls drive of thearm 41. Therobot controller 3 includes a CPU (not shown in the drawings) for controlling therobot controller 3 entirely in an integrated fashion. Thevisual sensor controller 1 is connected to therobot controller 3 through an external equipment interface (not shown in the drawings). Therobot controller 3 transmits a signal for triggering image processing to thevisual sensor controller 1. Further, therobot controller 3 receives a result of image processing obtained by execution of the image processing (detection of thetarget mark 5, for example) by thevisual sensor controller 1, etc. - The
robot controller 3 controls drive of thearm 41 so as to move thetarget mark 5 attached to the end of thearm 41 in a range set in advance for calibration (“calibration range”) during implementation of the calibration. Process of measuring the calibration range will be described in detail later. - The
robot controller 3 measures the coordinate position of thetarget mark 5 in the robot coordinate system attached to the end of thearm 41 of therobot 4. Specifically, therobot controller 3 can measure the coordinate position of thetarget mark 5 in the robot coordinate system while thetarget mark 5 is at a destination. - The
camera 2 is connected to thevisual sensor controller 1 functioning as the calibration device. Thevisual sensor controller 1 makes thecamera 2 capture an image of thetarget mark 5 to calibrate thecamera 2. -
FIG. 3 is a functional block diagram showing the functional configuration of thevisual sensor controller 1. Thevisual sensor controller 1 includes a central processing unit (CPU) 10 for controlling thevisual sensor controller 1 entirely in an integrated fashion. TheCPU 10 is connected through abus 11 tomultiple frame memories 12, a read-only memory (ROM) 13, a random access memory (RAM) 14, and anonvolatile RAM 15. Thecamera 2 is connected to thebus 11 through acamera interface 16. Further, amonitor 19 is connected to thebus 11 through amonitor interface 17. TheCPU 10 is further connected through thebus 11 to anexternal equipment interface 18. Theexternal equipment interface 18 is connected to therobot controller 3 to receive the coordinate position of thetarget mark 5 from therobot controller 3 and transmit a result of image processing obtained by execution of the image processing (detection of thetarget mark 5, for example) by thevisual sensor controller 1, etc. to therobot controller 3. - The
ROM 13 stores programs used for execution of various types of process by thevisual sensor controller 1. Generally, access is made faster to a RAM than to a ROM. Thus, theCPU 10 may develop the programs stored in theROM 13 in advance on theRAM 14. Then, theCPU 10 may read the programs from theRAM 14 and execute the read programs. TheRAM 14 stores temporarily saved data necessary for execution of the programs. Thenonvolatile RAM 15 is a magnetic storage unit, a flash memory, an MRAM, FRAM (registered trademark), or an EEPROM, for example. Alternatively, thenonvolatile RAM 15 is an SRAM or a DRAM backed up by a battery, for example. Thenonvolatile RAM 15 is configured as a nonvolatile memory to hold its storage state even after thevisual sensor controller 1 is powered off. Thenonvolatile RAM 15 stores setting necessary for execution of the programs, for example. Theframe memory 12 stores image data. - The
nonvolatile RAM 15 includes areference information storage 151 and adetection result storage 152. Thereference information storage 151 stores reference information (also called a “model pattern” or a “template”) indicating a subject (target mark 5). The reference information may be a group of edge points (also called an “edge point group”) in a subject (target mark 5), for example. The edge point is a point where brightness changes largely in an image. For example, the edge point group may be formed by converting the subject (target mark 5) to an edge image through publicly-known Sobel filtering, and extracting a pixel (edge point) having an intensity of a predetermined threshold or more from the edge image. The edge point group extracted in this way from the image including the subject (target mark 5) to be detected is stored as the model pattern into thereference information storage 151. The model pattern is not limited to edge points. For example, feature points such as those extracted by publicly-known SIFT may be used as the model pattern. Alternatively, the model pattern may be generated by arranging a geometric graphic such as a line segment, a rectangle, or a circle so as to match the contour of the subject (target mark 5). In this case, feature points may be provided at proper intervals on the geometric graphic forming the contour. The model pattern may also be a template image generated by cutting out a part corresponding to a model pattern designation area from a captured image of thetarget mark 5. Storing the model pattern in advance generated in the above-described way into thereference information storage 151 is also called “teaching the model pattern.” Teaching of the model pattern will be described later. - The
detection result storage 152 stores a result of detection of thetarget mark 5 detected by using the taught model pattern from data about an image captured by thecamera 2 while thetarget mark 5 having been moved in the set calibration range is at each destination. - In response to a command from the
CPU 10, thecamera 2 captures an image of the subject to acquire the image, and outputs a signal about the acquired image. Thecamera interface 16 has the function of generating a synchronization signal for controlling timing of exposure for thecamera 2 in response to a command from theCPU 10, and the function of amplifying a signal received from thecamera 2. Thecamera 2 and thecamera interface 16 are not limited to any particular parts but are commercially-available and common-used parts. - The signal about the image taken from the
camera 2 is A/D converted by thecamera interface 16, and then stored temporarily as digital image data through thebus 11 into theframe memory 12. In thevisual sensor controller 1, theCPU 10 processes the image by using data stored in theframe memory 12, theROM 13, theRAM 14, and thenonvolatile RAM 15. Data resulting from the image processing is stored again into theframe memory 12. In response to a command, theCPU 10 may transfer the data stored in theframe memory 12 to themonitor interface 17 and display the data on themonitor 19 in order to allow check of the substance of the data by an operator, for example. - The
external equipment interface 18 is connected to various types of external equipment. For example, theexternal equipment interface 18 is connected to therobot controller 3 to receive a signal for triggering image processing from therobot controller 3 and supply therobot controller 3 with position information data obtained by image processing, etc. A keyboard or a mouse may also be connected to theexternal equipment interface 18 as an input unit 33 for an operator, for example. - The function of the
CPU 10 will be described next in terms of each processing unit. Description in terms of each processing step (method) will be omitted as each processing step (method) can be understood by replacing “unit” in the following description by “step.”FIG. 4 is a block diagram showing the functional configuration of theCPU 10 in thevisual sensor controller 1. TheCPU 10 includes a modelpattern generation unit 101, a firstparameter setting unit 102, afirst detection unit 103, a first imagerange setting unit 105, a first calibrationrange measurement unit 106, an imagecapture control unit 107, and acalibration unit 108. Each of these functional units and functional steps is realized by execution of a system program in theROM 13 by theCPU 10. - The model
pattern generation unit 101 generates a model pattern by using thecamera 2, for example.FIG. 5 shows a flowchart followed by the modelpattern generation unit 101 for generating the model pattern. The function of the modelpattern generation unit 101 will be described by referring toFIG. 5 . - In step S1, the model
pattern generation unit 101 exerts control to make thecamera 2 capture an image of thetarget mark 5 arranged in the view of thecamera 2. At this time, a relationship between the position of thecamera 2 and that of thetarget mark 5 is desirably set to be the same as that during detection of thetarget mark 5. - In step S2, the model
pattern generation unit 101 sets an area in the captured image of thetarget mark 5 in the form of a rectangular area or a circular area, for example, as a model pattern designation area in which thetarget mark 5 appears. Further, the modelpattern generation unit 101 defines a model pattern coordinate system (image coordinate system) in the model pattern designation area.FIG. 6 shows an example of the model pattern designation area, that of the model pattern coordinate system (image coordinate system), and that of the model pattern. The modelpattern generation unit 101 may set an area instructed by an operator as the model pattern designation area. Alternatively, the modelpattern generation unit 101 may determine spots of large brightness gradients in the image as the contour of the image of thetarget mark 5, and set the model pattern designation area so as to contain the contour of the image of thetarget mark 5 inside the model pattern designation area. - In step S3, the model
pattern generation unit 101 extracts an edge point in the model pattern designation area as a feature point, obtains physical quantities such as the position of the edge point, the direction and the magnitude of a brightness gradient at the edge point, etc., and converts the edge point to a value expressed in the model pattern coordinate system defined in the model pattern designation area. Further, the modelpattern generation unit 101 sets a point instructed by the operator as the point P where the three-dimensional position information of thetarget mark 5 is to be measured, and stores the point P into thereference information storage 151. By designating the point P in advance where the three-dimensional position information of thetarget mark 5 is to be measured, the three-dimensional position information and the three-dimensional posture of thetarget mark 5 are determined in advance in the end point coordinate system at therobot 4. If the point P where the three-dimensional position information of thetarget mark 5 is to be measured is not designated explicitly, the modelpattern generation unit 101 may set a center point of the model pattern as a point where the three-dimensional position information of thetarget mark 5 is to be measured, for example.FIG. 7 shows an example of the point P where the three-dimensional position information of thetarget mark 5 is to be measured. InFIG. 7 , a center point of thetarget mark 5 is set as the point P where the three-dimensional position information of thetarget mark 5 is to be measured. As described above, the model pattern is not limited to edge points. For example, feature points such as those extracted by publicly-known SIFT may be used as the model pattern. Alternatively, the model pattern may be generated by arranging a geometric graphic such as a line segment, a rectangle, or a circle so as to match the contour of a subject (target mark 5) In this case, feature points may be provided at proper intervals on the geometric graphic forming the contour. The model pattern may also be a template image generated by cutting out a part corresponding to the model pattern designation area from the captured image of thetarget mark 5. - In step S4, the model
pattern generation unit 101 stores the generated model pattern into thereference information storage 151. As described above, the model pattern is generated by using the image captured by thecamera 2. - The following describes a parameter for detecting the
target mark 5 from data about the image captured by thecamera 2 by using the model pattern stored in thereference information storage 151. - A difference is generated in a distance between the
camera 2 and the position of thetarget mark 5 at a destination or in an angle from an optical axis, for example. Hence, an image of thetarget mark 5 captured by thecamera 2 may differ in a manner that depends on a destination of thetarget mark 5 in the calibration range from the model pattern generated by the modelpattern generation unit 101 in terms of appearance such as a size or the occurrence of rotation or distortion. As a result, it is likely that, in trying to detect thetarget mark 5 completely matching the model pattern, it will be impossible to detect such a subject (target mark 5) from data about the image captured by thecamera 2. - The first
parameter setting unit 102 sets a parameter for detecting thetarget mark 5 from data about an image captured by thecamera 2 so as to allow detection of a model pattern at any destination of thetarget mark 5 in the calibration range even in the above-described situation. More specifically, the parameter set by the firstparameter setting unit 102 depends on a detection algorithm. For example, the parameter may be set to have a predetermined range with respect to a model in terms of a size, rotation, distortion, a position range or an angle range for detection, for example. Alternatively, the parameter may be set as a single numerical value or an on/off value. The parameter is not limited to these examples. Thus, thefirst detection unit 103 described later is configured to be capable of detecting thetarget mark 5 matching the model pattern from captured image data by using a single numerical value, an on/off value, or a parameter value in a predetermined range so as to allow detection of the model pattern at any destination of thetarget mark 5. By doing so, thefirst detection unit 103 described later becomes capable of detecting the model pattern from the captured image data by using a proper parameter value. The parameter may be a parameter to be applied to data about an image captured by thecamera 2. In this case, thefirst detection unit 103 is configured to detect the model pattern about thetarget mark 5 from image data generated by applying the parameter to the data about the image captured by thecamera 2. For example, the model pattern can be detected from image data generated by smoothing the image data by a Gaussian filter. The parameter may be a parameter to be applied to the model pattern about thetarget mark 5. In this case, thefirst detection unit 103 is configured to detect thetarget mark 5 from the data about the image captured by thecamera 2 by using the model pattern to which the parameter is applied. - One example of the parameter set by the first
parameter setting unit 102 and applied to the model pattern may be a transformation matrix for projection transformation, affine transformation, or homothetic transformation. For example, if a value of the parameter is set as a single numerical value, a single transformation matrix is selectable. If a value of the parameter is set to have a predetermined range, a transformation matrix of a predetermined range is selectable. The following describes specific examples applied if the parameter is set to have a predetermined range. In the case of a projection transformation matrix, the parameter may be set in such a manner that a parameter range covers a projection transformation matrix with an element having a deviation of a predetermined threshold or less from a corresponding element in a projection transformation matrix as a basis. In the case of rotation, a range for a rotation angle may be set based on one rotation angle. Likewise, in the case of homothetic transformation, a range for a homothetic ratio may be set based on one homothetic ratio. For example, based on an angle between the optical axis of thecamera 2 and a plane on which thetarget mark 5 is to move during implementation of calibration (particularly if this plane is not at a right angle to the optical axis but is tilted diagonally from the optical axis), a size range may be set from 0.9 to 1.1 for thecamera 2 by using the parameter. By doing so, robust detection can be realized on the occurrence of a difference in appearance of thetarget mark 5 resulting from different destinations of thetarget mark 5, for example. A parameter value for exposure time is desirably set in consideration of an angle between thecamera 2 and a plane on which thetarget mark 5 is arranged or a relationship with illumination, for example. - The
first detection unit 103 detects thetarget mark 5 from the data about the image captured by thecamera 2, and measures the coordinate position of the detectedtarget mark 5 in an image coordinate system at thecamera 2. More specifically, thefirst detection unit 103 selects a parameter from a single numerical value, an on/off value, or a predetermined range for the parameter. In selecting the parameter from the predetermined range, a center value in the parameter range may be selected first, for example. Then, a value shifted in the plus or minus direction from the center value may be selected as a next parameter, for example. - If the parameter is a parameter to be applied to the data about the image captured by the
camera 2 as described above, after thefirst detection unit 103 selects the parameter from a single numerical value, an on/off value, or a predetermined range for the parameter, thefirst detection unit 103 converts the data about the image captured by thecamera 2 by using the selected parameter so as to allow detection of thetarget mark 5 from the image data. In this way, thefirst detection unit 103 can detect thetarget mark 5 from the converted image data by the publicly-known detection technique. - More specifically, the
first detection unit 103 extracts a feature point from the image data to which the parameter is applied by the same method as applied for extracting a feature point from the taught model pattern, and conducts publicly-known matching between a feature point resulting from the conversion with the parameter and a feature point forming the model pattern, thereby detecting thetarget mark 5. - Conversely, the
first detection unit 103 may convert the model pattern about thetarget mark 5 by using the selected parameter. In this case, thefirst detection unit 103 can detect thetarget mark 5 matching the converted model pattern from the captured image data by the above-described publicly-known detection technique. More specifically, thefirst detection unit 103 extracts a feature point from the data about the image captured by thecamera 2 by the same method as applied for extracting a feature point from the taught model pattern, and conducts publicly-known matching between the extracted feature point and a feature point in the model pattern to which the parameter is applied, thereby detecting thetarget mark 5. Thefirst detection unit 103 measures the coordinate position of the detectedtarget mark 5 in the image coordinate system at thecamera 2. - As described above, in this embodiment, the
target mark 5 is moved on a plane designated in advance. This plane may include multiple parallel planes or may be formed by coupling multiple partial planes. To determine an operation range for the robot 4 (calibration range), the first imagerange setting unit 105 sets an image range in the image coordinate system at thecamera 2 corresponding to this calibration range. More specifically, in response to an instruction from an operator, the first imagerange setting unit 105 sets an image range in the image captured by thecamera 2 based on the image coordinate system at thecamera 2. In this case, the first imagerange setting unit 105 can designate the image range as a rectangular range, for example. The first imagerange setting unit 105 may set the image entirely as the image range. If space for move of thetarget mark 5 includes an obstacle, for example, the first imagerange setting unit 105 may limit the image range in response to an instruction from the operator so as to avoid the obstacle existing in the view of thecamera 2. In this case, the first imagerange setting unit 105 may designate the image range as a closed graphic drawn with multiple line segments. - The first calibration
range measurement unit 106 measures a calibration range (a coordinate position in the robot coordinate system showing a boundary of the calibration range, for example) as an operation range for therobot 4 corresponding to the image range set by the first imagerange setting unit 105 before implementation of calibration by controlling therobot 4 so as to move thetarget mark 5 and making thefirst detection unit 103 detect thetarget mark 5. More specifically, the first calibrationrange measurement unit 106 determines whether or not thetarget mark 5 is detectable in the image range designated in the image coordinate system at thecamera 2 by the first imagerange setting unit 105 by moving thetarget mark 5 and detecting thetarget mark 5 repeatedly, thereby measuring a calibration range (a coordinate position in the robot coordinate system showing a boundary of the calibration range, for example) as an operation range for the robot 4 (specifically, target mark 5) corresponding to the designated image range. -
FIGS. 8A and 8B show flowcharts showing a flow followed by the first calibrationrange measurement unit 106 for measuring a coordinate position in the robot coordinate system showing a boundary of a calibration range. The function of the first calibrationrange measurement unit 106 will be described by referring toFIGS. 8A and 8B . A plane on which thetarget mark 5 is to be moved during implementation of calibration is defined as an XY plane in a three-dimensional coordinate system designated in advance. However, this plane is not limited to the XY plane in the three-dimensional coordinate system. - In step S11, the first calibration
range measurement unit 106 acquires the three-dimensional coordinate position of thetarget mark 5 in the robot coordinate system arranged at an initial position on a plane in the view of thecamera 2 and the coordinate position of thistarget mark 5 in the image coordinate system at thecamera 2 through therobot controller 3 and thefirst detection unit 103 respectively. Thetarget mark 5 may be moved to the initial position in response to jog operation by an operator, for example. Alternatively, thetarget mark 5 may be moved to an initial position stored in therobot controller 3. - In step S12, the first calibration
range measurement unit 106 moves the robot 4 (specifically, target mark 5) through therobot controller 3 on the plane from the initial position by a unit stroke in an X direction set in advance in an X-axis direction. Then, the first calibrationrange measurement unit 106 acquires the three-dimensional coordinate position of thetarget mark 5 in the robot coordinate system and the coordinate position of thetarget mark 5 in the image coordinate system at thecamera 2 through therobot controller 3 and thefirst detection unit 103 respectively. - In step S13, the first calibration
range measurement unit 106 moves the robot 4 (specifically, target mark 5) through therobot controller 3 on the plane from the initial position by a unit stroke in a Y direction set in advance in a Y-axis direction. Then, the first calibrationrange measurement unit 106 acquires the three-dimensional coordinate position of thetarget mark 5 in the robot coordinate system and the coordinate position of thetarget mark 5 in the image coordinate system at thecamera 2 through therobot controller 3 and thefirst detection unit 103 respectively. - In step S14, the first calibration
range measurement unit 106 calculates a three-dimensional vector VX (magnitude and direction) in the robot coordinate system corresponding to the X-direction unit stroke in the X-axis direction on the plane, and a two-dimensional vector vx (magnitude and direction) in the image coordinate system at thecamera 2 showing the stroke of thetarget mark 5. The first calibrationrange measurement unit 106 further calculates a three-dimensional vector VY (magnitude and direction) in the robot coordinate system corresponding to the Y-direction unit stroke in the Y-axis direction on the plane, and a two-dimensional vector vy (magnitude and direction) in the image coordinate system at thecamera 2 showing the stroke of thetarget mark 5. By doing so, the first calibrationrange measurement unit 106 can acquire approximate association between the stroke in the image coordinate system at thecamera 2 and the stroke in the robot coordinate system. For example, the first calibrationrange measurement unit 106 can convert a position information in the image coordinate system to an approximate position information in the robot coordinate system by using a transformation matrix R calculated based on the following formula (1): -
- Regarding move by the Y-direction unit stroke set in advance in the Y-axis direction in step S13, a start point for the vector is not limited to the initial position. The vector may be started from a point to which the
robot 4 has been moved from the initial position by the X-direction unit stroke in step S12. - In step S15, based on the association calculated in step S14, the first calibration
range measurement unit 106 calculates and sets the magnitude and the direction (V) of the stroke in the robot coordinate system from the initial position on the plane corresponding to the magnitude and the direction (v) of the stroke in the image coordinate system at thecamera 2 from the initial position to a boundary position (corner, for example) in the image coordinate system at thecamera 2. At this time, the magnitude and the direction (V) are desirably set so as to increase the probability of detection of thetarget mark 5 in a range in the image coordinate system at thecamera 2 by being multiplied by a factor less than 1. Coordinate values in the image range in the image coordinate system at the camera 2 (boundary positions including four corners, for example) are calculated in advance. - In step S16, the first calibration
range measurement unit 106 moves thetarget mark 5. More specifically, based on the set magnitude and the set direction (V) of the stroke, the first calibrationrange measurement unit 106 moves the robot 4 (specifically, target mark 5) through therobot controller 3 from the initial position on the plane. - In step S17, the first calibration
range measurement unit 106 determines through thefirst detection unit 103 whether or not thetarget mark 5 is detectable. If thetarget mark 5 has been detected successfully (Yes), the flow goes to step S18. If thetarget mark 5 has not been detected successfully (No), the flow goes to step S19. - In step S18, the first calibration
range measurement unit 106 determines through thefirst detection unit 103 whether or not a distance from the coordinate position of thetarget mark 5 in the image coordinate system at thecamera 2 to the boundary position (corner, for example) in the image coordinate system at thecamera 2 is small (specifically, whether or not the distance is a predetermined threshold or less). If the distance is determined to be small (Yes), the flow goes to step S20. If the distance is determined not to be small (No), the flow goes to step S22. - In step S19, the first calibration
range measurement unit 106 sets the magnitude and the direction of the stroke again. More specifically, if the first calibrationrange measurement unit 106 determines that thetarget mark 5 has not been detected successfully during move from the initial position, the first calibrationrange measurement unit 106 multiplies the set magnitude and the set direction (V) of the stroke by a factor less than 1, thereby setting the magnitude and the direction (V) of the stroke again. The factor used in this step should be lower than the factor having been applied to the move from the initial position. Then, the flow goes to step S16. - In step S20, the first calibration
range measurement unit 106 stores a current position information of thetarget mark 5 in the robot coordinate system and a current position information of thetarget mark 5 in the image coordinate system at thecamera 2. - In step S21, the first calibration
range measurement unit 106 determines whether or not measurement has been made for all the boundary positions. If measurement has been made for all the boundary positions (Yes), the flow ends. If there is a boundary position to be subject to next measurement (if No), the flow goes to step S15. - In step S22, the first calibration
range measurement unit 106 sets the magnitude and the direction (V′) of the stroke from the current position on the plane in the robot coordinate system corresponding to the magnitude and the direction (v′) of the stroke from the current position to the boundary position. - In step S23, based on the set magnitude and the set direction (V′) of the stroke from the current position, the first calibration
range measurement unit 106 moves the robot 4 (specifically, target mark 5). - In step S24, the first calibration
range measurement unit 106 determines through thefirst detection unit 103 whether or not thetarget mark 5 is detectable. If thetarget mark 5 has been detected successfully (Yes), the flow goes to step S18. If thetarget mark 5 has not been detected successfully (No), the flow goes to step S25. - In step S25, the first calibration
range measurement unit 106 multiplies the set magnitude and the set direction (V′) of the stroke by a factor less than 1, thereby setting the magnitude and the direction (V′) of the stroke again from a position where therobot 4 existed before moving. The factor used in this step should be lower than the factor having been applied to the move from the position where therobot 4 existed before moving. Then, the flow goes to step S23. As described above, the first calibrationrange measurement unit 106 can measure a coordinate position in the robot coordinate system showing a boundary of a calibration range in which thetarget mark 5 is to be detected. The above-described flowchart is not restrictive but has been described as an example. During implementation of calibration, thevisual sensor controller 1 functioning as the calibration device controls therobot 4 so as to move thetarget mark 5 in the calibration range. At this time, thearm 41 desirably moves through the set calibration range uniformly. For example, thevisual sensor controller 1 may control therobot 4 in such a manner that thetarget mark 5 moves by tracing a path shown inFIG. 9A or 9B . - The image
capture control unit 107 makes thecamera 2 capture an image of thetarget mark 5 attached to the end of thearm 41 of therobot 4 and moved by therobot 4 in a calibration range at each of multiple destination positions during implementation of calibration. The number of the destinations is desirably set to be larger than a number minimum for allowing calibration (a desired number is 20 or more, for example). By doing so, calibration can be performed more precisely. - The
calibration unit 108 stores the following coordinate positions into thedetection result storage 152 at each of multiple destinations of thetarget mark 5 attached to the end of thearm 41 of therobot 4 and moved by therobot controller 3 during implementation of calibration: the coordinate position of thetarget mark 5 in the image coordinate system at thecamera 2 appearing in data about an image captured by thecamera 2; and the coordinate position of thetarget mark 5 in the robot coordinate system while the image of thetarget mark 5 is captured by thecamera 2. Then, thecalibration unit 108 calibrates thecamera 2 based on the coordinate position of thetarget mark 5 in the image coordinate system at thecamera 2 stored in thedetection result storage 152, and the coordinate position of thetarget mark 5 in the robot coordinate system while the image of thetarget mark 5 is captured by thecamera 2 stored in thedetection result storage 152. -
FIG. 10 is a flowchart showing calibration process on thecamera 2 executed by the visual sensor controller 1 (CPU 10) according to this embodiment. - In step S31, a model pattern is generated and a parameter is set.
- In step S32, the CPU 10 (first image range setting unit 105) sets an image range (corresponding to a calibration range) for the
camera 2 in the image coordinate system in response to designation by an operator. The operator can designate the image range (corresponding to the calibration range) in an image captured by thecamera 2. - In step S33, the first calibration
range measurement unit 106 measures a coordinate position in the robot coordinate system showing a boundary of the calibration range. More specifically, as described above, the first calibrationrange measurement unit 106 determines whether or not thetarget mark 5 is detectable in the image range by moving thetarget mark 5 and detecting thetarget mark 5 repeatedly, thereby measuring the calibration range (a coordinate position in the robot coordinate system showing a boundary of an operation range for therobot 4, for example). (The process in step S33 is performed based on the above-described process flow followed by the first calibrationrange measurement unit 106.) - In step S34, the CPU 10 (calibration unit 108) sets a measurement counter for counting the number of times measurement is made at 1.
- In step S35, the CPU 10 (calibration unit 108) acquires the three-dimensional coordinate position of the
target mark 5 in the robot coordinate system measured by therobot controller 3. - In step S36, the CPU 10 (first detection unit 103) detects the
target mark 5 from data about the image captured by thecamera 2, and measures the coordinate position of the detectedtarget mark 5 in the image coordinate system at thecamera 2. - In step S37, the CPU 10 (calibration unit 108) stores the three-dimensional coordinate position of the
target mark 5 in the robot coordinate system and the coordinate position of thetarget mark 5 in the image coordinate system at thecamera 2 in association with each other while thetarget mark 5 is at a current position. - In step S38, the CPU 10 (calibration unit 108) increments the measurement counter for counting the number of times measurement is made by 1.
- In step S39, it is determined whether or not the measurement counter exceeds a predetermined value. If the measurement counter exceeds the predetermined value (Yes), the flow goes to step S41. If the measurement counter does not exceed the predetermined value (No), the flow goes to step S40.
- In step S40, the
robot controller 3 moves thetarget mark 5 attached to the end of thearm 41 of therobot 4 to a place in the calibration range where thetarget mark 5 can be measured from thecamera 2. Then, the flow goes to step S35. - In step S41, the CPU 10 (calibration unit 108) calibrates the
camera 2 based on the association stored in step S37 between the three-dimensional coordinate position of thetarget mark 5 in the robot coordinate system and the coordinate position of thetarget mark 5 in the image coordinate system at thecamera 2. - This process flow is not restrictive but has been described as an example.
- As described above, the
visual sensor controller 1 of the first embodiment includes: the first imagerange setting unit 105 that sets an image range in the image coordinate system at thecamera 2; and the first calibrationrange measurement unit 106 that measures a calibration range as an operation range for therobot 4 corresponding to the set image range before implementation of calibration by controlling therobot 4 to move thetarget mark 5 and detecting thetarget mark 5. In this way, an operation range for the robot 4 (calibration range) corresponding to a range of move of thetarget mark 5 is measured in advance before implementation of calibration based on a view range for a single visual sensor, so that the calibration can be performed with an increased degree of freedom. Further, thetarget mark 5 is allowed to move in a range covering the view of thecamera 2 entirely, so that the calibration can be performed more precisely. If a plane on which thetarget mark 5 is to move includes an obstacle, for example, an image range is designated in response to an instruction from an operator so as to avoid the obstacle existing in the view of thecamera 2. By doing so, the calibration range can be set efficiently. - A calibration range can be set on a plane. The plane on which the
target mark 5 is to be moved during implementation of calibration can be a plane vertical to the optical axis of thecamera 2 or a plane tilted from the optical axis of thecamera 2 as shown inFIG. 11A . Further, as shown inFIG. 11B , a calibration range can be set on each of at least two planes. A calibration range can be extended by making such settings. This increases the degree of freedom further in calibration, so that the calibration is performed more precisely. - A calibration method of the first embodiment and a program of the first embodiment achieve effects comparable to those achieved by the
visual sensor controller 1 of the first embodiment. - The first embodiment of the present invention has been described, but the present invention is not limited to the above-described first embodiment. The effects described in the first embodiment are merely a list of most preferred effects resulting from the present invention. Effects achieved by the present invention are not limited to those described in the first embodiment.
- In the first embodiment, the model
pattern generation unit 101 is configured to generate a model pattern by using thecamera 2 and store the generated model pattern. Instead of being configured to generate a model pattern by using thecamera 2, the modelpattern generation unit 101 may be configured to use an existing shape (a shape such as a circle, for example) as thetarget mark 5. - In the first embodiment, the
target mark 5 is moved on a plane designated in advance and a calibration range is measured on this plane. Alternatively, the calibration range may be defined in any three-dimensional space. This is expected to increase the degree of freedom further in calibration. For example, the plane to be designated in advance may include multiple parallel planes or may be formed by coupling multiple partial planes. As shown inFIG. 11B , for example, the plane to be designated may include at least two planes. If the plane includes multiple planes, the first calibrationrange measurement unit 106 measures a coordinate position in the robot coordinate system showing a boundary of a calibration range on each of the planes. As another example, two planes may be defined in a three-dimensional coordinate system. A calibration range may be set as space inside a hexahedron defined by a point of intersection between each plane and a line of sight pointing to a corner in an image captured by thecamera 2. In this case, the first calibrationrange measurement unit 106 measures a coordinate position in the robot coordinate system showing a boundary of a calibration range in this hexahedron. - A second embodiment will be described next. In the second embodiment, a visual sensor controller 1A functioning as a calibration device sets a calibration range as an operation range for a
robot 4 for allowing image capture of atarget mark 5 in an image range for astereo camera 2A including at least afirst camera 21 and asecond camera 22 after thetarget mark 5 is moved. The description of the second embodiment does not cover structures and functions common to those of the first embodiment but is directed to issues peculiar to the second embodiment. -
FIG. 12 shows the configuration of arobot system 1000 entirely for calibration on a stereo camera. Therobot system 1000 is for performing calibration using thestereo camera 2A with thefirst camera 21 and thesecond camera 22. The number of cameras forming thestereo camera 2A is not limited to two but can be any number of two or more. Each of the cameras forming thestereo camera 2A is certainly applicable as a single camera. Thestereo camera 2A is fixed to a pedestal (not shown in the drawings). As shown inFIG. 13A , thefirst camera 21 and thesecond camera 22 may be arranged parallel to each other. As shown inFIG. 13B , each of thefirst camera 21 and thesecond camera 22 may be arranged at a tilt. Tilting each of thefirst camera 21 and thesecond camera 22 makes it possible to increase an area of an overlap between an area of image captured by thefirst camera 21 and an area of image captured by thesecond camera 22, compared to arranging thefirst camera 21 and thesecond camera 22 parallel to each other. Specifically, tilting each of thefirst camera 21 and thesecond camera 22 makes it possible to increase an area where three-dimensional measurement is allowed by thestereo camera 2A, compared to arranging thefirst camera 21 and thesecond camera 22 parallel to each other. Thefirst camera 21 and thesecond camera 22 desirably have the same configuration in terms of a view range, a lens, etc. By doing so, thefirst camera 21 and thesecond camera 22 are expected to catch thetarget mark 5 in the same way. - The
stereo camera 2A is connected to the visual sensor controller 1A. The visual sensor controller 1A makes thefirst camera 21 and thesecond camera 22 capture images of thetarget mark 5, and calibrates each of thefirst camera 21 and thesecond camera 22. - The functional configuration of the visual sensor controller 1A functioning as the calibration device is the same as the functional configuration of the
visual sensor controller 1 of the first embodiment (FIG. 3 ), except that thestereo camera 2A (first camera 21 and second camera 22) is connected to thebus 11 through thecamera interface 16. The functional configuration of the visual sensor controller 1A can be understood by referring toFIG. 3 mentioned above. -
FIG. 14 is a block diagram showing the functional configuration of aCPU 10 in the visual sensor controller 1A. As shown inFIG. 14 , theCPU 10 in the visual sensor controller 1A includes a modelpattern generation unit 101, a firstparameter setting unit 102, afirst detection unit 103, a first imagerange setting unit 105, a first calibrationrange measurement unit 106, a secondparameter setting unit 1022, asecond detection unit 1032, a second imagerange setting unit 1052, a second calibrationrange measurement unit 1062, an imagecapture control unit 107, and acalibration unit 108. - The model
pattern generation unit 101 generates a model pattern by making thefirst camera 21 capture an image of thetarget mark 5 arranged in the view of thefirst camera 21. The model pattern generated by using the image captured by thefirst camera 21 is also used as a model pattern for thesecond camera 22. A model pattern for thesecond camera 22 may be generated individually by making thesecond camera 22 capture an image of thetarget mark 5 arranged in the view of thesecond camera 22. - The first
parameter setting unit 102 sets a first parameter for detecting a model pattern about thetarget mark 5 from data about an image captured by thefirst camera 21. The function of the firstparameter setting unit 102 is comparable to that of the firstparameter setting unit 102 in the first embodiment. - The following description is for the second
parameter setting unit 1022 that uses the model pattern about thetarget mark 5 generated by using thefirst camera 21 to set a second parameter for detecting this model pattern from data about an image captured by thesecond camera 22. The secondparameter setting unit 1022 sets the second parameter for detecting the model pattern about thetarget mark 5 from the data about the image captured by thesecond camera 22 based on the first parameter. More specifically, the secondparameter setting unit 1022 uses the first parameter as it is for initially setting a value of the second parameter. Meanwhile, if a value of the second parameter is set to have a predetermined range during initial setting of the second parameter, for example, the secondparameter setting unit 1022 may employ the same range as the first parameter. Alternatively, the secondparameter setting unit 1022 may set a wide range for the second parameter covering a predetermined range for the first parameter set by the firstparameter setting unit 102. In such cases, if thesecond detection unit 1032 described later has detected the model pattern about thetarget mark 5 successfully from the data about the image captured by thesecond camera 22 by applying a given value of the second parameter, the secondparameter setting unit 1022 can set a range for a value of the second parameter again with respect to this value of the second parameter as a center based on a deviation from a center value in the predetermined range for the first parameter. For example, if the first parameter has a size range from 0.9 to 1.1, a center value is 1.0 and a deviation from the center value in the predetermined range for the first parameter is 0.1. If a subject (target mark 5) of thesecond camera 22 has been detected successfully by setting a size of the second parameter for thesecond camera 22 at 0.95, a center value of the second parameter is set at 0.95 and the deviation in the first parameter is applied to the second parameter. Specifically, a value of the second parameter is set in a range [0.85 to 1.05] with respect to 0.95 as a center value. In this way, the range for a value of the second parameter during initial setting of the second parameter can be readjusted, so that the model pattern can be detected more efficiently from the data about the image captured by thesecond camera 22. The second parameter is not limited to these examples. - The
first detection unit 103 detects the model pattern about thetarget mark 5 from the data about the image captured by thefirst camera 21, and measures the coordinate position of the detectedtarget mark 5 in an image coordinate system at thefirst camera 21. The function of thefirst detection unit 103 is comparable to that of thefirst detection unit 103 in the first embodiment. - The
second detection unit 1032 detects the model pattern about thetarget mark 5 from the data about the image captured by thesecond camera 22, and measures the coordinate position of the detectedtarget mark 5 in an image coordinate system at thesecond camera 22. Detection process by thesecond detection unit 1032 can be understood by replacing thecamera 2, the firstparameter setting unit 102, and the parameter in the description given above in the first embodiment relating to thefirst detection unit 103 by thesecond camera 22, the secondparameter setting unit 1022, and the second parameter respectively. - In response to an instruction from an operator, the first image
range setting unit 105 sets a first image range in the image captured by thefirst camera 21 based on the image coordinate system at thefirst camera 21. The first image range may cover the captured image entirely. The first imagerange setting unit 105 can designate the first image range as a rectangular range, for example. If a plane on which thetarget mark 5 attached to an end portion of anarm 41 is to move includes an obstacle, for example, the first imagerange setting unit 105 may limit the first image range in response to an instruction from the operator so as to avoid the obstacle existing in the view of thefirst camera 21. In this case, the first imagerange setting unit 105 may designate the first image range as a closed graphic drawn with multiple line segments. The function of the first imagerange setting unit 105 is comparable to that of the first imagerange setting unit 105 in the first embodiment. - Like the first image
range setting unit 105, the second imagerange setting unit 1052 sets a second image range in the image captured by thesecond camera 22 based on the image coordinate system at thesecond camera 22. Process by the second imagerange setting unit 1052 can be understood by replacing thecamera 2 in the description given above in the first embodiment relating to the first imagerange setting unit 105 by thesecond camera 22. - The first calibration
range measurement unit 106 measures a first calibration range as an operation range for therobot 4 corresponding to the first image range set by the first imagerange setting unit 105 before implementation of calibration by controlling therobot 4 so as to move thetarget mark 5 and making thefirst detection unit 103 detect thetarget mark 5. More specifically, the first calibrationrange measurement unit 106 determines whether or not thetarget mark 5 is detectable in the first image range by thefirst camera 21 by moving thetarget mark 5 and detecting thetarget mark 5 repeatedly, thereby measuring the first calibration range as an operation range for therobot 4 corresponding to the first image range. The function of the first calibrationrange measurement unit 106 is comparable to that of the first calibrationrange measurement unit 106 in the first embodiment. - Likewise, the second calibration
range measurement unit 1062 determines whether or not thetarget mark 5 is detectable in the second image range by thesecond camera 22 by moving thetarget mark 5 and detecting thetarget mark 5 repeatedly, thereby measuring a second calibration range as an operation range for therobot 4 corresponding to the second image range. Process by the second calibrationrange measurement unit 1062 can be understood by replacing thecamera 2, the firstparameter setting unit 102, the parameter, and the image range in the description given above in the first embodiment relating to the first calibrationrange measurement unit 106 by thesecond camera 22, the secondparameter setting unit 1022, the second parameter, and the second image range respectively. Like in the first embodiment, thetarget mark 5 is moved on a plane designated in advance in this embodiment. The first calibration range and the second calibration range are set on the same plane. A flowchart about process of measuring the first calibration range followed by the first calibrationrange measurement unit 106, and a flowchart about process of measuring the second calibration range followed by the second calibrationrange measurement unit 1062, can be understood by the replacing thecamera 2, the firstparameter setting unit 102, the parameter, and the image range referred to in the description in the first embodiment for the flowchart relating to the calibration range measurement unit 106 (FIGS. 8A and 8B ) by thefirst camera 21, the firstparameter setting unit 102, the first parameter, and the first image range respectively, and by thesecond camera 22, the secondparameter setting unit 1022, the second parameter, and the second image range respectively. While thefirst camera 21 and thesecond camera 22 are calibrated, the visual sensor controller 1A functioning as the calibration device controls therobot 4 so as to move thetarget mark 5 in at least one of the first calibration range and the second calibration range, or in a range covered by both the first calibration range and the second calibration range. Specifically, thetarget mark 5 may be moved only in a range covered by both the first calibration range and the second calibration range, only in a range covered by either the first calibration range or the second calibration range, or in a range including the first calibration range and the second calibration range. - The image
capture control unit 107 makes thefirst camera 21 and thesecond camera 22 capture images of thetarget mark 5 at each of multiple destination positions attached to the end of thearm 41 of therobot 4 and to be moved by therobot 4 in at least one of the first calibration range and the second calibration range, or in a range covered by both the first calibration range and the second calibration range during implementation of calibration. The number of the destinations is desirably set to be larger than a number minimum for allowing calibration (a desired number is 20 or more, for example) on each of thefirst camera 21 and thesecond camera 22. By doing so, calibration can be performed more precisely. - The
calibration unit 108 stores the following coordinate positions into thedetection result storage 152 at each of multiple destinations of thetarget mark 5 moved in at least one of the first calibration range and the second calibration range, or in a range covered by both the first calibration range and the second calibration range during implementation of calibration: the coordinate position of thetarget mark 5 in the image coordinate system at thefirst camera 21; the coordinate position of thetarget mark 5 in the image coordinate system at thesecond camera 22; and the coordinate position of thetarget mark 5 in the robot coordinate system while the images of thetarget mark 5 are captured by thefirst camera 21 and thesecond camera 22. Then, thecalibration unit 108 calibrates thefirst camera 21 and thesecond camera 22 based on the coordinate position of thetarget mark 5 in the image coordinate system at thefirst camera 21 stored in thedetection result storage 152, the coordinate position of thetarget mark 5 in the image coordinate system at thesecond camera 22 stored in thedetection result storage 152, and the coordinate position of thetarget mark 5 in the robot coordinate system while the images of thetarget mark 5 are captured by thefirst camera 21 and thesecond camera 22 stored in thedetection result storage 152. Thefirst camera 21 and thesecond camera 22 may be calibrated individually. -
FIG. 15 is an example of a flowchart showing calibration process on thefirst camera 21 and thesecond camera 22 performed by the visual sensor controller 1A (CPU 10) according to this embodiment. This flowchart shows process performed if thetarget mark 5 is to be moved in a range including the first calibration range and the second calibration range. - In step S51, a model pattern is generated. Further, the first parameter and the second parameter are set.
- In step S52, in response to designation by an operator, the CPU 10 (first image
range setting unit 105 and second image range setting unit 1052) sets the first image range in an image captured by thefirst camera 21 in the image coordinate system at thefirst camera 21, and sets the second image range in an image captured by thesecond camera 22 in the image coordinate system at thesecond camera 22. - In step S53, the CPU 10 (first calibration
range measurement unit 106 and second calibration range measurement unit 1062) measures a coordinate position in the robot coordinate system showing a boundary of the first calibration range corresponding to the first image range for thefirst camera 21, and measures a coordinate position in the robot coordinate system showing a boundary of the second calibration range corresponding to the second image range for thesecond camera 22. In this way, the first calibration range and the second calibration range are set to allow calibration process on thefirst camera 21 and thesecond camera 22 to be started. - In step S54, the CPU 10 (calibration unit 108) sets a measurement counter for counting the number of times measurement is made at 1.
- In step S55, the CPU 10 (calibration unit 108) acquires the three-dimensional coordinate position of the
target mark 5 in the robot coordinate system measured by therobot controller 3. - In step S56, the CPU 10 (
first detection unit 103 and second detection unit 1032) detects thetarget mark 5 from data about the image captured by thefirst camera 21 and from data about the image captured by thesecond camera 22, and measures the coordinate position of the detectedtarget mark 5 in each of the image coordinate system at thefirst camera 21 and the image coordinate system at thesecond camera 22. Thetarget mark 5 may be detected only from thefirst camera 21 and thesecond camera 22. In this case, only a coordinate position in an image coordinate system at a camera having detected thetarget mark 5 successfully is stored. - In step S57, the CPU 10 (calibration unit 108) stores the three-dimensional coordinate position of the
target mark 5 in the robot coordinate system, the coordinate position of thetarget mark 5 in the image coordinate system at thefirst camera 21, and the coordinate position of thetarget mark 5 in the image coordinate system at thesecond camera 22 in association with each other while thetarget mark 5 is at a current position. - In step S58, the CPU 10 (calibration unit 108) increments the measurement counter for counting the number of times measurement is made by 1.
- In step S59, if the measurement counter does not exceed a predetermined value (No), the flow goes to step S60. If the measurement counter exceeds the predetermined value (Yes), the flow goes to step S61.
- In step S60, the
robot controller 3 moves thetarget mark 5 attached to the end of thearm 41 of therobot 4 to a place set in advance in the first calibration range or the second calibration range where thetarget mark 5 can be measured from at least one of thefirst camera 21 and thesecond camera 22. Then, the flow goes to step S55. - In step S61, the CPU 10 (calibration unit 108) calibrates the
first camera 21 and thesecond camera 22 based on the association stored in step S57 between the three-dimensional coordinate position of thetarget mark 5 in the robot coordinate system, the coordinate position of thetarget mark 5 in the image coordinate system at thefirst camera 21, and the coordinate position of thetarget mark 5 in the image coordinate system at thesecond camera 22. This process flow is not restrictive but has been described as an example. For example, if thetarget mark 5 is to be moved only in a range covered by both the first calibration range and the second calibration range, steps S56 and S60 can be replaced as follows. In step S56, only if thetarget mark 5 is detected from both thefirst camera 21 and thesecond camera 22, the coordinate position of thetarget mark 5 in the image coordinate system at thefirst camera 21 and the coordinate position of thetarget mark 5 in the image coordinate system at thesecond camera 22 may be stored. In step S60, therobot controller 3 moves thetarget mark 5 attached to the end of thearm 41 of therobot 4 to a place set in advance in the first calibration range and the second calibration range where thetarget mark 5 can be measured from both thefirst camera 21 and thesecond camera 22. - The visual sensor controller 1A of the second embodiment includes: the first image
range setting unit 105 that sets the first image range in the image coordinate system at thefirst camera 21; the first calibrationrange measurement unit 106 that measures the first calibration range as an operation range for therobot 4 corresponding to the first image range before implementation of calibration by controlling therobot 4 to move thetarget mark 5 and detecting thetarget mark 5; the second imagerange setting unit 1052 that sets the second image range in the image coordinate system at thesecond camera 22; and the second calibrationrange measurement unit 1062 that measures the second calibration range as an operation range for therobot 4 corresponding to the second image range before implementation of the calibration by controlling therobot 4 to move thetarget mark 5 and detecting thetarget mark 5. In this way, an operation range for the robot 4 (first calibration range and second calibration range) corresponding to a range of move of thetarget mark 5 is measured in advance before implementation of the calibration based on view ranges for multiple cameras (visual sensors), so that the calibration can be performed with an increased degree of freedom. Further, thetarget mark 5 is allowed to move in a maximum range for each camera, so that the calibration can be performed more precisely. If space in which thetarget mark 5 is to move includes an obstacle, for example, the first image range and the second image range are designated in response to an instruction from an operator so as to avoid the obstacle. By doing so, the operation range for the robot 4 (first calibration and second calibration range) can be set efficiently. - Each of the first calibration range and the second calibration range can be configured to be set on a plane. A calibration method of the second embodiment and a program of the second embodiment achieve effects comparable to those achieved by the visual sensor controller 1A of the second embodiment.
- The present invention is not limited to the above-described second embodiment. The effects described in the second embodiment are merely a list of most preferred effects resulting from the present invention. Effects achieved by the present invention are not limited to those described in the second embodiment.
- In the second embodiment, the
stereo camera 2A includes two cameras. Alternatively, thestereo camera 2A may be configured to include three or more cameras. - In the second embodiment, the model
pattern generation unit 101 is configured to generate a model pattern by using thefirst camera 21 and store the generated model pattern. Alternatively, the modelpattern generation unit 101 may be configured to generate a model pattern by using thesecond camera 22 and store the generated model pattern. In the second embodiment, the modelpattern generation unit 101 is configured to generate and store a model pattern. Instead of being configured to generate a model pattern by using thefirst camera 21 or thesecond camera 22, the modelpattern generation unit 101 may be configured to use an existing shape (a shape such as a circle, for example) as thetarget mark 5. - In the second embodiment, the
target mark 5 is moved on a plane designated in advance. Further, the first calibration range and the second calibration range are measured on this plane. Alternatively, the first calibration range and the second calibration range may be defined in any three-dimensional space. For example, the plane to be designated in advance may include multiple parallel planes or may be formed by coupling multiple partial planes. - In these embodiments, the
visual sensor controller 1 or 1A functions as the calibration device. However, the calibration device is not limited to thevisual sensor controller 1 or 1A. The calibration device may be a controller including thevisual sensor controller 1 or 1A and therobot controller 3 integrated with each other. Alternatively, the calibration device may cover information processing devices (computers) in general. For example, the calibration device may be a server, a PC, various types of controllers, etc. - The calibration method implemented by the
visual sensor controller 1 or 1A is realized by software. To realize the calibration method by software, programs constituting the software are installed on a computer (visual sensor controller 1). These programs may be stored in a removable medium and then distributed to a user. Alternatively, these programs may be distributed by being downloaded onto a computer of the user through a network. -
- 1000 Robot system
- 1 Visual sensor controller (calibration device)
- 1A Visual sensor controller (calibration device)
- 10 CPU
- 101 Model pattern generation unit
- 102 First parameter setting unit
- 1022 Second parameter setting unit
- 103 First detection unit
- 1032 Second detection unit
- 105 First image range setting unit
- 1052 Second image range setting unit
- 106 First calibration range measurement unit
- 1062 Second calibration range measurement unit
- 107 Image capture control unit
- 108 Calibration unit
- 11 Bus
- 12 Frame memory
- 13 ROM
- 14 RAM
- 15 Nonvolatile memory
- 151 Reference information storage
- 152 Detection result storage
- 16 Camera interface
- 17 Monitor interface
- 18 External equipment interface
- 19 Monitor
- 2 Camera
- 2A Stereo camera
- 21 First camera
- 22 Second camera
- 3 Robot controller
- 4 Robot
- 41 Arm
- 5 Target mark
Claims (11)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2017-003667 | 2017-01-12 | ||
JP2017003667A JP6396516B2 (en) | 2017-01-12 | 2017-01-12 | Visual sensor calibration apparatus, method and program |
Publications (2)
Publication Number | Publication Date |
---|---|
US20180194008A1 true US20180194008A1 (en) | 2018-07-12 |
US10647001B2 US10647001B2 (en) | 2020-05-12 |
Family
ID=62636676
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/862,103 Active US10647001B2 (en) | 2017-01-12 | 2018-01-04 | Calibration device, calibration method, and computer readable medium for visual sensor |
Country Status (4)
Country | Link |
---|---|
US (1) | US10647001B2 (en) |
JP (1) | JP6396516B2 (en) |
CN (1) | CN108297095B (en) |
DE (1) | DE102018200154A1 (en) |
Cited By (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180272535A1 (en) * | 2017-03-22 | 2018-09-27 | Kabushiki Kaisha Toshiba | Object handling device and calibration method thereof |
US10137574B2 (en) * | 2014-12-25 | 2018-11-27 | Keyence Corporation | Image processing apparatus, image processing system, image processing method, and computer program |
CN109062138A (en) * | 2018-09-11 | 2018-12-21 | 苏州伟信奥图智能科技有限公司 | A kind of five shaft platform system calibrating schemes based on stereo calibration block |
US10369698B1 (en) * | 2019-03-07 | 2019-08-06 | Mujin, Inc. | Method and system for performing automatic camera calibration for robot control |
US10565737B1 (en) * | 2019-07-09 | 2020-02-18 | Mujin, Inc. | Method and system for performing automatic camera calibration for a scanning system |
US10647003B2 (en) * | 2017-05-18 | 2020-05-12 | Fanuc Corporation | Programming device and robot control method |
CN111625001A (en) * | 2020-05-28 | 2020-09-04 | 珠海格力智能装备有限公司 | Robot control method and device and industrial robot |
US20200391385A1 (en) * | 2019-06-17 | 2020-12-17 | Kabushiki Kaisha Toshiba | Object handling control device, object handling device, object handling method, and computer program product |
US10940591B2 (en) * | 2017-08-09 | 2021-03-09 | Omron Corporation | Calibration method, calibration system, and program |
US10984292B2 (en) * | 2018-08-20 | 2021-04-20 | Rubbish Co. | Object information collection, classification and analytics via a smart grabber tool |
US20210114222A1 (en) * | 2019-03-29 | 2021-04-22 | Mujin, Inc. | Method and control system for verifying and updating camera calibration for robot control |
US20210146942A1 (en) * | 2018-07-06 | 2021-05-20 | Brain Corporation | Systems, methods and apparatuses for calibrating sensors mounted on a device |
CN112917510A (en) * | 2019-12-06 | 2021-06-08 | 中国科学院沈阳自动化研究所 | Industrial robot space position appearance precision test system |
US11043008B2 (en) * | 2017-03-31 | 2021-06-22 | Panasonic Intellectual Property Management Co., Ltd. | Imaging system, calibration method, and calibrator |
US20210229290A1 (en) * | 2018-06-05 | 2021-07-29 | Hitachi, Ltd. | Camera Position/Attitude Calibration Device, Camera Position/Attitude Calibration Method, and Robot |
US11173609B2 (en) * | 2019-01-22 | 2021-11-16 | Samsung Electronics Co., Ltd | Hand-eye calibration method and system |
US20220044444A1 (en) * | 2018-09-28 | 2022-02-10 | Shanghai Eyevolution Technology Co., Ltd | Stereo calibration method for movable vision system |
US11254019B2 (en) * | 2019-03-05 | 2022-02-22 | The Boeing Company | Automatic calibration for a robot optical sensor |
US11273554B2 (en) * | 2018-03-30 | 2022-03-15 | Nidec Corporation | Method and device for evaluating calibration precision |
US20220358680A1 (en) * | 2019-11-15 | 2022-11-10 | Wuyi University | Full-automatic calibration method and apparatus oriented to structured light 3d vision system |
US11577400B2 (en) * | 2018-09-03 | 2023-02-14 | Abb Schweiz Ag | Method and apparatus for managing robot system |
US11584013B2 (en) * | 2020-03-31 | 2023-02-21 | Wipro Limited | System, device and method for determining error in robotic manipulator-to-camera calibration |
US11721040B2 (en) | 2017-03-31 | 2023-08-08 | Panasonic Intellectual Property Management Co., Ltd. | Imaging system, calibration method, and calibrator |
Families Citing this family (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP7070127B2 (en) * | 2018-06-15 | 2022-05-18 | オムロン株式会社 | Robot control system |
JP6871220B2 (en) * | 2018-11-08 | 2021-05-12 | ファナック株式会社 | Control system |
US10399227B1 (en) * | 2019-03-29 | 2019-09-03 | Mujin, Inc. | Method and control system for verifying and updating camera calibration for robot control |
US10576636B1 (en) * | 2019-04-12 | 2020-03-03 | Mujin, Inc. | Method and control system for and updating camera calibration for robot control |
CN112188187B (en) * | 2019-07-01 | 2024-01-02 | 北京小米移动软件有限公司 | Position detection method and device of image acquisition component and storage medium |
CN114174006B (en) * | 2019-07-19 | 2024-03-05 | 西门子(中国)有限公司 | Robot hand-eye calibration method, device, computing equipment, medium and product |
CN110332930B (en) * | 2019-07-31 | 2021-09-17 | 小狗电器互联网科技(北京)股份有限公司 | Position determination method, device and equipment |
CN110640745B (en) * | 2019-11-01 | 2021-06-22 | 苏州大学 | Vision-based robot automatic calibration method, equipment and storage medium |
JP7468288B2 (en) | 2020-10-16 | 2024-04-16 | オムロン株式会社 | CALIBRATION DEVICE AND METHOD FOR AUTOMATIC CALIBRATION SETTING |
CN112936315B (en) * | 2021-03-18 | 2022-12-20 | 深圳市梯易易智能科技有限公司 | Mechanical arm calibration method and device based on imaging mode |
DE112021007102T5 (en) | 2021-04-19 | 2024-02-29 | Fanuc Corporation | Program generating device and robot control device |
WO2023013740A1 (en) | 2021-08-04 | 2023-02-09 | 京セラ株式会社 | Robot control device, robot control system, and robot control method |
CN114523471B (en) * | 2022-01-07 | 2023-04-25 | 中国人民解放军海军军医大学第一附属医院 | Error detection method based on association identification and robot system |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080188983A1 (en) * | 2007-02-05 | 2008-08-07 | Fanuc Ltd | Calibration device and method for robot mechanism |
US20160039094A1 (en) * | 2013-04-05 | 2016-02-11 | Abb Technology Ltd. | Robot System And Method For Calibration |
Family Cites Families (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2690603B2 (en) | 1990-05-30 | 1997-12-10 | ファナック株式会社 | Vision sensor calibration method |
US6101455A (en) | 1998-05-14 | 2000-08-08 | Davis; Michael S. | Automatic calibration of cameras and structured light sources |
JP4267005B2 (en) | 2006-07-03 | 2009-05-27 | ファナック株式会社 | Measuring apparatus and calibration method |
JP5365218B2 (en) | 2009-01-28 | 2013-12-11 | 富士電機株式会社 | Robot vision system and automatic calibration method |
JP2010188439A (en) * | 2009-02-16 | 2010-09-02 | Mitsubishi Electric Corp | Method and apparatus for calculating parameter |
US9393694B2 (en) | 2010-05-14 | 2016-07-19 | Cognex Corporation | System and method for robust calibration between a machine vision system and a robot |
EP2729850A4 (en) | 2011-08-11 | 2015-07-08 | Siemens Healthcare Diagnostics | Methods and apparatus to calibrate an orientation between a robot gripper and a camera |
JP2013215867A (en) * | 2012-04-12 | 2013-10-24 | Seiko Epson Corp | Robot control device, robot control method, robot control program, and robot |
JP5928114B2 (en) | 2012-04-12 | 2016-06-01 | セイコーエプソン株式会社 | Robot system, robot system calibration method, robot |
JP6468741B2 (en) | 2013-07-22 | 2019-02-13 | キヤノン株式会社 | Robot system and robot system calibration method |
JP6335460B2 (en) | 2013-09-26 | 2018-05-30 | キヤノン株式会社 | Robot system control apparatus, command value generation method, and robot system control method |
DE102014100538B4 (en) | 2014-01-17 | 2022-09-08 | Pi4_Robotics Gmbh | Method for calibrating a robot and a camera and system for performing the method |
JP2015174191A (en) | 2014-03-17 | 2015-10-05 | 株式会社安川電機 | Robot system, calibration method of robot system and position correction method of robot system |
JP6429473B2 (en) * | 2014-03-20 | 2018-11-28 | キヤノン株式会社 | Robot system, robot system calibration method, program, and computer-readable recording medium |
JP6415190B2 (en) | 2014-09-03 | 2018-10-31 | キヤノン株式会社 | ROBOT DEVICE, ROBOT CONTROL PROGRAM, RECORDING MEDIUM, AND ROBOT DEVICE CONTROL METHOD |
DE102015104582A1 (en) | 2015-03-26 | 2016-09-29 | Pi4_Robotics Gmbh | Method for calibrating a robot at a work area and system for carrying out the method |
-
2017
- 2017-01-12 JP JP2017003667A patent/JP6396516B2/en active Active
-
2018
- 2018-01-04 US US15/862,103 patent/US10647001B2/en active Active
- 2018-01-08 DE DE102018200154.5A patent/DE102018200154A1/en active Pending
- 2018-01-10 CN CN201810023805.6A patent/CN108297095B/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080188983A1 (en) * | 2007-02-05 | 2008-08-07 | Fanuc Ltd | Calibration device and method for robot mechanism |
US20160039094A1 (en) * | 2013-04-05 | 2016-02-11 | Abb Technology Ltd. | Robot System And Method For Calibration |
Cited By (38)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10137574B2 (en) * | 2014-12-25 | 2018-11-27 | Keyence Corporation | Image processing apparatus, image processing system, image processing method, and computer program |
US10576635B2 (en) * | 2017-03-22 | 2020-03-03 | Kabushiki Kaisha Toshiba | Object handling device and calibration method thereof |
US11318619B2 (en) | 2017-03-22 | 2022-05-03 | Kabushiki Kaisha Toshiba | Object handling device and calibration method thereof |
US20180272535A1 (en) * | 2017-03-22 | 2018-09-27 | Kabushiki Kaisha Toshiba | Object handling device and calibration method thereof |
US11721040B2 (en) | 2017-03-31 | 2023-08-08 | Panasonic Intellectual Property Management Co., Ltd. | Imaging system, calibration method, and calibrator |
US11043008B2 (en) * | 2017-03-31 | 2021-06-22 | Panasonic Intellectual Property Management Co., Ltd. | Imaging system, calibration method, and calibrator |
US10647003B2 (en) * | 2017-05-18 | 2020-05-12 | Fanuc Corporation | Programming device and robot control method |
US11364636B2 (en) | 2017-05-18 | 2022-06-21 | Fanuc Corporation | Programming device and robot control method |
US10940591B2 (en) * | 2017-08-09 | 2021-03-09 | Omron Corporation | Calibration method, calibration system, and program |
US11273554B2 (en) * | 2018-03-30 | 2022-03-15 | Nidec Corporation | Method and device for evaluating calibration precision |
US20210229290A1 (en) * | 2018-06-05 | 2021-07-29 | Hitachi, Ltd. | Camera Position/Attitude Calibration Device, Camera Position/Attitude Calibration Method, and Robot |
US11865730B2 (en) * | 2018-06-05 | 2024-01-09 | Hitachi, Ltd. | Camera position/attitude calibration device, camera position/attitude calibration method, and robot |
US20210146942A1 (en) * | 2018-07-06 | 2021-05-20 | Brain Corporation | Systems, methods and apparatuses for calibrating sensors mounted on a device |
US11586859B2 (en) | 2018-08-20 | 2023-02-21 | Rubbish Co. | Object information collection, classification and analytics via a smart grabber tool |
US10984292B2 (en) * | 2018-08-20 | 2021-04-20 | Rubbish Co. | Object information collection, classification and analytics via a smart grabber tool |
US11577400B2 (en) * | 2018-09-03 | 2023-02-14 | Abb Schweiz Ag | Method and apparatus for managing robot system |
CN109062138A (en) * | 2018-09-11 | 2018-12-21 | 苏州伟信奥图智能科技有限公司 | A kind of five shaft platform system calibrating schemes based on stereo calibration block |
US11663741B2 (en) * | 2018-09-28 | 2023-05-30 | Anhui Eyevolution Technology Co., Ltd. | Stereo calibration method for movable vision system |
US20220044444A1 (en) * | 2018-09-28 | 2022-02-10 | Shanghai Eyevolution Technology Co., Ltd | Stereo calibration method for movable vision system |
US11173609B2 (en) * | 2019-01-22 | 2021-11-16 | Samsung Electronics Co., Ltd | Hand-eye calibration method and system |
US11254019B2 (en) * | 2019-03-05 | 2022-02-22 | The Boeing Company | Automatic calibration for a robot optical sensor |
WO2020180342A1 (en) * | 2019-03-07 | 2020-09-10 | Mujin, Inc. | Method and system for performing automatic camera calibration for robot control |
US10628966B1 (en) | 2019-03-07 | 2020-04-21 | Mujin, Inc. | Method and system for performing automatic camera calibration for robot control |
US10373336B1 (en) * | 2019-03-07 | 2019-08-06 | Mujin, Inc. | Method and system for performing automatic camera calibration for robot control |
US10369698B1 (en) * | 2019-03-07 | 2019-08-06 | Mujin, Inc. | Method and system for performing automatic camera calibration for robot control |
US10636172B1 (en) | 2019-03-07 | 2020-04-28 | Mujin, Inc. | Method and system for performing automatic camera calibration for robot control |
US11883964B2 (en) | 2019-03-29 | 2024-01-30 | Mujin, Inc. | Method and control system for verifying and updating camera calibration for robot control |
US20210114222A1 (en) * | 2019-03-29 | 2021-04-22 | Mujin, Inc. | Method and control system for verifying and updating camera calibration for robot control |
US11590656B2 (en) * | 2019-03-29 | 2023-02-28 | Mujin, Inc. | Method and control system for verifying and updating camera calibration for robot control |
US11559894B2 (en) * | 2019-06-17 | 2023-01-24 | Kabushiki Kaisha Toshiba | Object handling control device, object handling device, object handling method, and computer program product |
US20200391385A1 (en) * | 2019-06-17 | 2020-12-17 | Kabushiki Kaisha Toshiba | Object handling control device, object handling device, object handling method, and computer program product |
US11074722B2 (en) | 2019-07-09 | 2021-07-27 | Mujin, Inc. | Method and system for performing automatic camera calibration for a scanning system |
US10565737B1 (en) * | 2019-07-09 | 2020-02-18 | Mujin, Inc. | Method and system for performing automatic camera calibration for a scanning system |
US11967113B2 (en) | 2019-07-09 | 2024-04-23 | Mujin, Inc. | Method and system for performing automatic camera calibration for a scanning system |
US20220358680A1 (en) * | 2019-11-15 | 2022-11-10 | Wuyi University | Full-automatic calibration method and apparatus oriented to structured light 3d vision system |
CN112917510A (en) * | 2019-12-06 | 2021-06-08 | 中国科学院沈阳自动化研究所 | Industrial robot space position appearance precision test system |
US11584013B2 (en) * | 2020-03-31 | 2023-02-21 | Wipro Limited | System, device and method for determining error in robotic manipulator-to-camera calibration |
CN111625001A (en) * | 2020-05-28 | 2020-09-04 | 珠海格力智能装备有限公司 | Robot control method and device and industrial robot |
Also Published As
Publication number | Publication date |
---|---|
DE102018200154A1 (en) | 2018-07-12 |
CN108297095A (en) | 2018-07-20 |
JP2018111166A (en) | 2018-07-19 |
CN108297095B (en) | 2019-04-09 |
JP6396516B2 (en) | 2018-09-26 |
US10647001B2 (en) | 2020-05-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10647001B2 (en) | Calibration device, calibration method, and computer readable medium for visual sensor | |
US10434654B2 (en) | Calibration device, calibration method, and computer readable medium for visual sensor | |
CN111482959B (en) | Automatic hand-eye calibration system and method of robot motion vision system | |
JP5850962B2 (en) | Robot system using visual feedback | |
US9495750B2 (en) | Image processing apparatus, image processing method, and storage medium for position and orientation measurement of a measurement target object | |
EP3421930B1 (en) | Three-dimensional shape data and texture information generation system, photographing control program, and three-dimensional shape data and texture information generation method | |
JP6587489B2 (en) | Image processing apparatus, image processing method, and image processing system | |
US20060088203A1 (en) | Method and apparatus for machine-vision | |
EP3309505A1 (en) | Dimension measurement device and dimension measurement method | |
JP2016103230A (en) | Image processor, image processing method and program | |
EP3446838A1 (en) | Calibration method, calibration system, and program | |
JP5383836B2 (en) | An image processing apparatus having a function of automatically adjusting a search window | |
KR20140008262A (en) | Robot system, robot, robot control device, robot control method, and robot control program | |
KR20160003776A (en) | Posture estimation method and robot | |
EP3577629B1 (en) | Calibration article for a 3d vision robotic system | |
JP7167418B2 (en) | Position control system, position detector, and control program | |
JP6199000B2 (en) | Information processing device | |
US20210042576A1 (en) | Image processing system | |
US20220176560A1 (en) | Control system, control method, and control unit | |
JP2015007639A (en) | Information processing apparatus, information processing method and program | |
JP6770826B2 (en) | Automatic collimation method and automatic collimation device for measuring the placement position of structures | |
JP7414850B2 (en) | robot system | |
WO2021145304A1 (en) | Image processing system | |
CN111742349B (en) | Information processing apparatus, information processing method, and information processing storage medium | |
CN113858214A (en) | Positioning method and control system for robot operation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: FANUC CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NAMIKI, YUUTA;WARASHINA, FUMIKAZU;TAKIZAWA, SHOUTA;REEL/FRAME:044537/0482 Effective date: 20171219 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING RESPONSE FOR INFORMALITY, FEE DEFICIENCY OR CRF ACTION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |