JPWO2018043524A1 - Robot system, robot system control apparatus, and robot system control method - Google Patents

Abstract

An articulated robot system, controller and control method that are easy to implement and are not affected by lens distortion and the like. The robot system (10) has an articulated robot (20), one or more cameras (30, 31) for imaging the work space of the robot, and a control device (40). Then, at the time of calibration, the control device obtains posture parameters of the robot, obtains an image from the camera, recognizes a sign (50) fixed to the robot in the image, and calculates position information of the sign Create and store a reference table that associates the position information of the sign with the posture parameter of the robot. Then, at the time of work, the control device acquires an image from the camera, recognizes an object in the image, calculates the position information of the object, and refers to the reference table based on the position information of the object to obtain a hand. The posture parameter for moving (24) to the object is acquired, and the acquired posture parameter is transmitted to the robot. .

Description

  The present invention relates to a robot system for moving an articulated robot's hand to a target object. The present invention also relates to a control device and control method of the robot system.

  A system has been studied in which the robot itself recognizes the surrounding environment and performs work based on information obtained from a visual sensor such as a camera using an articulated robot. For example, it is an operation such as recognizing a bulk-stacked object with a camera, moving a hand at the tip of a robot arm to the work object and grasping it.

  In a general robot system, in order to move the robot's hand to a target object, after obtaining position information of the object with a camera or the like, the position information is converted from the camera coordinate system to the robot coordinate system. Tell the robot. Here, in the conversion from the camera coordinate system to the robot coordinate system, it is necessary to obtain in advance the relative positional relationship between the camera and the robot. Therefore, a conversion matrix for performing coordinate conversion between the camera coordinate system and the world coordinate system and between the robot coordinate system and the world coordinate system is obtained by calibration. Thus, the position coordinates of the object in the camera coordinate system can be converted to position coordinates in the robot coordinate system via the world coordinate system.

  However, to determine the transformation matrix between the camera coordinate system and the world coordinate system, it is necessary to use the camera's focal length, image center, image size (pixel size), internal parameters such as distortion aberration coefficient, and external parameters that represent the camera's position and orientation. You need to know the parameters. For this purpose, for example, a plane pattern such as a checker board and the position of a camera are to be determined using a high-precision three-dimensional measuring device, which requires extremely time-consuming and expensive work.

  Even when determining the coordinate conversion between the robot coordinate system and the world coordinate system, it is also extremely laborious to measure the posture parameter by manually moving the hand to the teaching point whose position coordinate in the world coordinate system is accurately known. And expensive work was required.

  On the other hand, in Patent Document 1, the robot arm device is operated so that the tip of the pointer stick held by the tip of the robot arm device draws a rectangle, and stereo image measurement of the position of each vertex of the rectangle is performed. There is described a method of calculating coordinate conversion data based on the data, and determining a relative position and orientation relationship between a robot arm device and a stereo measuring device through an intermediate coordinate system based on a rectangle. According to Patent Document 1, not the design data, but the data actually measured by the robot system is used. Therefore, the robot arm device and the stereo image measuring device are not affected by the error due to the processing and installation occurring at the time of system manufacture. The relative position and posture relationship can be determined, and the time required for the calibration operation can be shortened.

JP-A-9-128549

  However, in the method described in Patent Document 1, since the transformation matrix is calculated based on the measurement of the four rectangular vertices, the distortion aberration etc. of the lens can not be accurately reproduced. Even if there is no distortion, a transformation matrix is used for coordinate transformation between the camera coordinate system and the intermediate coordinate system and between the intermediate coordinate system and the robot coordinate system, so if there is an error due to irregular lens distortion etc. It can not be completely excluded.

  The present invention has been made in consideration of the above, and it is not necessary to measure the position of the installed camera or robot, and is not influenced by lens distortion or the like, and the hand of the articulated robot is obtained based on the camera image. An object of the present invention is to provide a robot system movable to a target position, a robot system controller, and a robot system control method.

  For the above purpose, in the present invention, the robot is controlled using a reference table in which positional information obtained from an image of a camera is directly associated with posture parameters of the articulated robot.

  Specifically, the robot system of the present invention has an articulated robot, one or more cameras for imaging the work space of the robot, and a control device. Then, at the time of calibration, the control device acquires posture parameters of the robot, acquires an image from the camera, recognizes a sign fixed to the robot in the image, and detects position information of the sign. A reference table is created which is calculated and associated with the position information of the sign and the posture parameter of the robot, and is stored. Then, at the time of work, the control device acquires an image from the camera, recognizes an object in the image, calculates position information of the object, and obtains the reference table from the position information of the object. To acquire the posture parameter for moving the hand to the object, and transmitting the acquired posture parameter to the robot.

  Here, the posture parameter of the robot refers to a parameter related to the posture of the robot and at least the position of the hand can be determined. Further, the sign fixed to the robot means a sign such as a mark held by the hand, whose positional relationship with the hand or arm of the robot is fixed.

  Preferably, the robot system has a first camera and a second camera, and the positional information of the marker and the positional information of the object indicate the pixels in the image by the first camera of the marker or the object, respectively. It is a set of the position and the pixel position in the image by the second camera.

  Alternatively, the robot system has a first camera and a second camera, and the positional information of the marker and the positional information of the object indicate the pixel position in the image of the marker or the object by the first camera, respectively. And a set of distance information calculated based on the principle of triangulation from the image by the first camera and the image by the second camera.

  Alternatively, the robot system has a first camera and a second camera, and the position information of the marker and the position information of the object are respectively an image of the marker or the object by the first camera and the second It may be three-dimensional position coordinates in the camera coordinate system calculated based on the principle of triangulation from the image by the camera.

  Alternatively, the robot system further includes a mirror which is located within the angle of view of the camera and in which the work space is reflected when viewed from the camera; The position information of the object may be a set of the pixel position of the direct image in the image by the camera and the pixel position of the reflection image by the mirror of the marker or the object, respectively.

  Alternatively, the camera is a distance image camera, and the position information of the marker and the position information of the object are respectively the pixel position of the marker or the object in the distance image by the distance image camera and the distance information It may be a set of

  Alternatively, the position information of the sign and the position information of the object may be respectively the pixel position in the image of the sign or the object by the camera and the area in the image when the sign is at the position. It may be a pair.

  In any of the above robot systems, preferably, the posture parameter is a set of joint variables of the robot.

  Alternatively, in any of the above robot systems, the posture parameter may be three-dimensional position coordinates of the marker or the robot coordinate system of the object.

  The robot system control apparatus according to the present invention acquires a posture parameter of the robot at the time of calibration of the articulated robot, and a sign fixed to the hand of the robot from one or more cameras for imaging the work space of the robot Acquire an image in which the image is captured, recognize the sign in the image, calculate position information of the sign, and create and store a reference table in which the position information of the sign and the posture parameter of the robot are associated .

  Preferably, the robot system control apparatus according to the present invention acquires an image from the camera during work of the robot, recognizes an object in the image, and calculates position information of the object, the object The position parameter for moving the hand to the object is acquired from the position information of the position table, and the acquired position parameter is transmitted to the robot.

  Another robot system control apparatus according to the present invention includes a first communication unit that transmits and / or receives posture parameters of an articulated robot to the robot, and an image showing a marker fixed on the robot from a camera A second communication unit to receive, an operation unit that recognizes the sign from the image, calculates position information of the sign, and creates a reference table in which position information of the hand is associated with the posture parameter; And a storage unit for storing a table.

  Preferably, in another robot system control apparatus according to the present invention, the second communication unit receives an image in which an object is photographed from the camera, and the operation unit recognizes the object from the image, and the object is received. The position information of the object is calculated, and the posture parameter for moving the hand of the robot to the object is acquired from the position information of the object with reference to the reference table, and the first communication unit Posture parameters are sent to the robot.

  The robot system control method of the present invention has a calibration process and a work process. In the calibration step, while operating the articulated robot on which the marker is fixed to move the marker, the working space of the robot is imaged by one or more cameras, and the marker is displayed in the image captured by the camera. The position information of the mark is calculated, the position information of the mark is associated with the posture parameter of the robot at the time of imaging, and sequentially registered in the reference table. In the work process, the camera captures an image of a work space in which an object is present, recognizes the object in an image captured by the camera, calculates position information of the object, and calculates position information of the object The posture parameter for moving the hand of the robot to the object is acquired with reference to the reference table, and the acquired posture parameter is transmitted to the robot.

  According to the robot system, robot system control apparatus, or robot system control method of the present invention, using an actual camera and robot, a reference table is created in which positional information by the camera is directly associated with posture parameters of the robot, and reference is made Operate the robot based on the table. Since the reference table is created by using an actual camera and robot, the hand can be moved to the target position without being affected by errors in system manufacture or irregular lens distortion. In addition, since other coordinate systems such as world coordinates are not used, it is not necessary to measure the internal and external parameters of the camera, the installation position and orientation of the robot, etc., and the labor of calibration can be saved.

BRIEF DESCRIPTION OF THE DRAWINGS It is a whole block diagram of the robot system of 1st Embodiment of this invention. It is a functional block diagram of a robot system control device of a 1st embodiment of the present invention. It is a flowchart of the robot control method of 1st Embodiment of this invention. It is a flowchart of the calibration process of the robot control method of 1st Embodiment of this invention. It is a flowchart of the hand movement in the operation | work process of the robot control method of 1st Embodiment of this invention. It is an image at the time of calibration by a 1st embodiment of the present invention. It is a figure which shows the example of the reference table of 1st Embodiment of this invention. It is a figure which shows the other example of the reference table of 1st Embodiment of this invention. It is a figure which shows the other example of the reference table of 1st Embodiment of this invention. It is a whole block diagram of the robot system of 2nd Embodiment of this invention. It is an image at the time of calibration by a 2nd embodiment of the present invention. It is a figure which shows the example of the reference table of 2nd Embodiment of this invention. It is a whole block diagram of the robot system of 3rd Embodiment of this invention. It is a figure which shows the example of the reference table of 3rd Embodiment of this invention. It is a whole block diagram of the robot system of 4th Embodiment of this invention. It is an image at the time of calibration by a 4th embodiment of the present invention. It is a figure which shows the example of the reference table of 4th Embodiment of this invention. It is a whole block diagram of the robot system of 5th Embodiment of this invention.

  A first embodiment of the present invention will be described based on FIGS. 1 to 9.

  In FIG. 1, a robot system 10 according to the present embodiment includes a multi-joint robot 20, a first camera 30 and a second camera 31 that capture a working space of the robot, and a robot system control apparatus 40. Hereinafter, the “robot system control device” is simply referred to as a “control device”.

  The robot 20 is fixed to the work place. The robot 20 has a base 25 for fixing the robot to the work place, an arm 21 having a plurality of links 22 connected by joints 23, a hand 24 connected to the tip of the arm, and a drive unit 26. Each joint of the robot includes an actuator (not shown) such as a servomotor and a sensor (not shown) that detects joint variables of the joint. Here, the joint variable refers to the displacement of the joint, and specifically refers to the joint angle (interlink angle) of the rotary joint and the link distance of the linear joint. The hand 24 is also called a hand or end effector. FIG. 1 shows the state at the time of calibration, in which the robot holds the mark 50 on the hand.

  The drive unit 26 of the robot 20 drives an arm and a hand. The drive unit of the simplest configuration comprises a drive device such as a servo amplifier, and drives an actuator of each joint based on a joint variable received from the outside. Alternatively, the drive unit further has a memory for storing various operations, such as a CPU, a program, etc., can receive the three-dimensional position coordinates of the hand in the robot coordinate system from the outside, and can calculate joint variables. It may be. Here, the robot coordinate system is a coordinate system based on the fixed point of the robot, preferably a coordinate system based on the base 25. The robot coordinate system may be an orthogonal coordinate system or various polar coordinate systems. Hereinafter, the term “three-dimensional position coordinates” simply means three-dimensional position coordinates in the robot coordinate system.

  The robot 20 has the base 25 fixed to the work area and does not move through calibration and work. Therefore, the position of the hand 24 can be represented by a combination of joint variables and three-dimensional position coordinates. For example, when the degree of freedom of the arm 21 is 3, the three joint variables of the arm determine the position of the hand and the posture of the arm. Alternatively, when the arm 21 has 3 degrees of freedom and the hand 24 has 3 degrees of freedom, the six joint variables of the arm and the hand determine the position and posture of the hand and the posture of the arm. The three-dimensional position coordinates of the hand indicate the position of the hand. As described above, a parameter that can determine at least the position of the hand in relation to the posture of the robot is referred to herein as “the posture parameter of the robot”.

  The type of robot is not particularly limited, and in addition to the serial link type vertical articulated robot illustrated in FIG. 1, various articulated robots such as horizontal articulated type and parallel link type can be used.

  The first camera 30 and the second camera 31 are fixed to the work place. The two cameras are installed to image the work space of the robot 20. Here, the work space of the robot refers to an area of the space where the hand 24 of the robot can reach, in which the hand may move during work. The two cameras 30 and 31 are cameras capable of capturing a two-dimensional image.

  Referring to FIG. 2, control device 40 includes a first communication unit 41, a second communication unit 42, a calculation unit 43, a storage unit 44, and a third communication unit 45.

  The first communication unit 41 is connected to the drive unit 26 of the robot 20 by a signal line, and performs communication with the robot. The first communication unit transmits the posture parameter corresponding to the target position to the robot, and instructs to move the hand 24 to the target position. The first communication unit also receives posture parameters such as joint variables from the robot. The second communication unit 42 is connected to the first camera 30 and the second camera 31 by signal lines, and communicates with the cameras. The second communication unit transmits an imaging instruction to the camera and receives an image from the camera.

  The calculation unit 43 processes the image received by the second communication unit 42 from the cameras 30 and 31. At the time of calibration, the arithmetic unit recognizes the marker 50 in the image and calculates its position information, and the position information and the attitude parameter transmitted by the first communication unit 41 to the robot or the first communication unit received from the robot Create a reference table that associates posture parameters. More precisely, the operation unit creates one record that associates the position information of the sign and the posture parameter of the robot, and additionally registers it in the existing reference table. At the time of operation, the operation unit recognizes an object in an image and calculates its position information. The calculation unit refers to the reference table and acquires posture parameters for moving the hand of the robot to the object from the position information of the object.

  The storage unit 44 stores a reference table. As the storage unit, a rewritable and randomly accessible auxiliary storage device such as a hard disk drive can be suitably used. The third communication unit 45 is connected to the input device 46 and the output device 47 by signal lines, and communicates with the operator. The third communication unit receives an instruction from the operator from the input device 46, and transmits the status and the like of the robot system to the output device 47.

  The control device 40 does not necessarily have to be physically one device, and a plurality of devices may share processing. Further, the control device may be integrally formed with the drive unit 26 of the robot.

  Next, a robot system control method of this embodiment will be described. The reference numerals of the respective parts are the ones shown in FIG. 1 and FIG.

  In FIG. 3, the robot system control method of the present embodiment includes installation of a robot and a camera on a work place, a calibration process, and a work process using the robot.

  First, the robot 20, the first camera 30, and the second camera 31 are installed in the work place. At this time, the two cameras are installed at positions and orientations capable of imaging the work space of the robot. The angle formed by the optical axes of the two cameras is preferably 30 to 150 degrees, more preferably 60 to 120 degrees, and particularly preferably 70 to 110 degrees. This is because the spatial resolution in the optical axis direction of the camera decreases as the orientations of the two cameras are closer to parallel or opposite to each other. However, as the orientations of the two cameras differ, the possibility of occurrence of an occlusion in which a part of the sign is a blind spot increases. The orientations of the two cameras may be determined appropriately according to the shape of the sign and the work space so that occlusion does not easily occur and the necessary spatial resolution can be secured. The robot and camera are fixed to the work place through the following calibration and work steps.

  FIG. 4 shows the flow of the calibration process. In the calibration step, a reference table is created in which positional information obtained from the camera is associated with posture parameters of the robot.

  First, the marker 50 is fixed to the hand 24. To fix the mark to the hand, hold the mark with the hand, hold the mark with the hand by suction or other method, fix the object to be the mark to the hand, or mark the part of the hand It includes drawing a mark and marking a characteristic part that is a part of the hand. The number of labels may be more than one. When the marker is held by a hand, the marker itself may be the object itself or an object shaped like the object, a spherical object, a pointer rod or the like. The marker may be fixed to an arm which is a movable part of the robot.

  The marker is preferably fixed so that the position of the marker corresponds to the gripping position of the hand. This is because the marker position registered in the reference table matches the target position of the hand movement at the time of operation. In FIG. 1, the spherical marker 50 is fixed by being held by the hand, and the position of the marker corresponds to the holding position of the hand. In the present embodiment, the description will be continued on the assumption that the mark position and the grip position of the hand coincide with each other.

  Next, the robot is operated according to an instruction from the first communication unit 41 of the control device 40 to move the hand 24 and the marker 50 in the work space. At this time, it is not necessary to accurately determine the movement path and the stop position of the sign in advance, and it may be determined to go around the work space approximately. The extent to which the marker is moved can be communicated by the operator from the input device 46 to the control device. At this time, the operator may directly input a joint variable corresponding to a point on the moving path, or the operator may input a three-dimensional position coordinate of a point on the moving path to drive the computing unit 43 or the robot. Unit 26 may calculate joint variables.

  Then, while moving the label within the above range, a lookup table is created with the label stopped and / or in motion.

  For example, the method of creating a lookup table at the stop of the label is as follows. The posture parameter registered in the reference table may be a posture parameter transmitted from the first communication unit 41 to the robot 20, or may be a posture parameter received from the robot by the first communication unit. For example, when the sign stops at the position instructed from the control device, the control device 40 already knows the posture parameter transmitted from the first communication unit 41 to the robot, so even if this posture parameter is registered in the reference table Good. Also, for example, even when the first communication unit transmits the three-dimensional position coordinates of the stop position to the robot, the first communication unit receives the joint variable from the robot when the marker is stopped, and registers the joint variable in the reference table. May be

In parallel with the acquisition of the posture parameter, the second communication unit 42 transmits an imaging instruction to the cameras 30 and 31 and receives an image from the cameras 30 and 31 when the sign is stopped. The image which the control apparatus 40 received from the camera 30 and 31 in FIG. 6 is shown. A marker 50 is recorded in the two images. The computing unit 43 of the control device processes the image received from the camera to recognize the sign in the image. As a method of recognizing a label, known methods such as extraction of feature points and template matching can be used. Computing section of the label, the pixel positions in the two images set _{(u 1i, v 1 i,} u 2i, v 2i) , and calculates the position information of the label.

  The method of creating a reference table in the state in which the sign is moving is, for example, as follows. The first communication unit 41 receives the posture parameters to be registered in the reference table from the robot. When the imaging timing of the camera and the acquisition timing of the posture parameter of the robot can be synchronized, the first communication unit 41 and the second communication unit 42 designate the robot and the cameras 30 and 31 at the same time, and the posture parameter or image respectively You can request to send Further, based on one synchronization signal, the first communication unit 41 and the second communication unit 42 may receive posture parameters or images from the robot or the camera, respectively. The synchronization signal may be generated by a synchronization signal generation unit (not shown) in the control device 40, or may be generated by either the first or second communication unit. When the cameras 30 and 31 can generate an image with imaging time, the second communication unit receives images from the two cameras every moment, and the first communication unit acquires the posture parameter acquired from the robot Images of the same time can be selected. Arithmetic unit 43 recognizes marker 50 in the image and calculates its positional information.

  As described above, position information by the camera and posture parameters of the robot corresponding to the position can be obtained for one marker position. The calculation unit 43 of the control device associates the position information and the posture parameter to create one record, and additionally registers the reference table stored in the storage unit 44.

  The marker 50 is circulated in the work space, and the calibration is ended when the marker reaches the scheduled end position. Preferably, at the end of the calibration, the records in the reference table are sorted using position information as a key. This is to streamline search during work.

FIG. 7 shows an example of the reference table. Each record of this reference table includes coordinates (u _1i , v _1i ) as pixel positions in the first camera image as position information, coordinates (u _2i , v _2i ) as pixel positions in the second camera image, and The posture parameters include six joint angles (θ _1i , θ _2i , θ _3i , θ _4i , θ _5i , θ _6i ) in a robot having six degrees of freedom.

  FIG. 5 shows a flow of hand movement in the work process. In the work process, the hand of the robot is moved to the work object based on the position information calculated from the camera image.

  First, the control device 40 transmits an imaging instruction to the cameras 30 and 31 from the second communication unit 42, and the camera captures an image of the work space. There are objects in the work space. The second communication unit receives an image from the camera. The computing unit 43 processes the two images to recognize an object in the images. The method of recognizing an object can be performed in the same manner as the method of recognizing a label. The calculation unit calculates position coordinates of the object in each camera coordinate system as position information of the object which is the target of the hand movement.

  The calculation unit 43 searches the reference table stored in the storage unit 44 from the position information of the object. If the record including the position information of the object is found in the reference table, the operation unit acquires the posture parameter corresponding to the position information. If the record including the position information of the object is not found in the reference table, the operation unit may estimate and acquire the posture parameter corresponding to the position information of the object. For example, posture parameters can be estimated by interpolating between posture parameters in records of a plurality of points having close position information. The first communication unit 41 transmits the posture parameter to the robot and instructs to move the hand to the target.

  If the posture parameter is estimated, the position of the hand may be finely adjusted after the robot operates according to the posture parameter. For fine adjustment of the hand position, known methods such as the method described in WO 2013/176212 can be used.

  Also, when the hand position is finely adjusted, it is preferable to feed back the result to the reference table. Specifically, after the hand reaches the object after fine adjustment, the first communication unit 41 of the control device transmits a posture parameter request to the robot, receives the posture parameter from the robot, and the calculation unit 43 targets The position information of the object and the posture parameter may be associated and additionally registered in the reference table of the storage unit 44.

  Here, the reference table will be described in more detail.

  In the reference table, for a point in the work space of the robot, position information obtained from a camera image of the point and a posture parameter of the robot corresponding to the point are registered in association with each other.

The position information registered in the reference table is not limited to the set of two pixel coordinates as shown in FIG. For example, as shown in FIG. 8, the position information is calculated based on the pixel position (u _1i , v _1i ) in the image by the first camera, the image by the first camera and the image by the second camera based on the principle of triangulation. It may be a set with the calculated distance information (d _i ). Alternatively, the position information may be three-dimensional position coordinates in the camera coordinate system calculated based on the triangulation principle from the image by the first camera and the image by the second camera. In these cases, it is necessary to know the center-to-center distance (baseline length) of the first camera and the second camera and the epipolar line in the image, for example, a commercially available stereo camera etc. in which the first camera and the second camera are integrated. In the case of using the camera, geometrical parameters of such a camera are adjusted in advance, and an internal calibration operation is unnecessary.

  An example of a posture parameter registered in the reference table is a joint variable of a robot. The joint variables are not limited to the set of joint angles as shown in FIG. When using a joint variable as the posture parameter, a joint angle (angle between links) in the rotary joint and a distance between the links in the linear motion joint are combined as joint variables, and the joint variables according to the types of joints constituting the robot are combined.

  When joint variables are registered in the reference table, as shown in FIG. 1, it is preferable that the position of the marker at the time of calibration and the grip position of the hand coincide. This is because the position information of the mark and the joint variable in a state in which the mark is actually gripped can be associated and registered in the reference table. In this case, the hand can be moved to a target position for grasping the object by pointing the robot to a joint variable obtained by searching the reference table from the position information of the object at work. Furthermore, even if there is a deflection of the arm due to gravity, the positional information and joint variables registered in the reference table have already incorporated effects of the deflection and so on, and no special consideration is necessary at the time of operation.

  On the other hand, when the position of the marker at the time of calibration and the grip position of the hand do not match, for example, when the marker is fixed to a part different from the grip position such as the side of the hand, the position information of the marker is obtained Joint variables can not be registered in the reference table as they are. The joint variable does not correspond to the position of the marker. In other words, even if the robot is instructed to perform joint variables when position information of the marker is obtained, the hand can not grip the object at the position of the marker. In this case, the three-dimensional position coordinates of the sign are once obtained from the joint variables when the position information of the sign is obtained, converted to the joint variable corresponding to the sign position based on the design data of the robot, and registered in the reference table. be able to. At this time, joint variables can be determined in consideration of geometric constraints and the like for avoiding obstacles in the work place.

Another example of posture parameters to be registered in the reference table is the three-dimensional position coordinates of the marker in the robot coordinate system. The robot coordinate system may be any coordinate system that does not change throughout calibration and work processes. Preferably, a coordinate system based on the base 25 of the robot can be used. The robot coordinate system may be an orthogonal coordinate system or a polar coordinate system. As an example, FIG. 9 shows a reference table in which orthogonal coordinates (Rx _i , Ry _i , Rz _i ) of a sign position in the robot coordinate system are used as posture parameters. Representing the posture parameters in the three-dimensional position coordinates of the marker has an advantage that the number of parameters can be reduced.

  The three-dimensional position coordinates of the marker can be calculated from joint variables at that time as long as the marker is fixed to the robot, regardless of whether the marker position matches or does not match the grip position of the hand. The computing unit can calculate the three-dimensional position coordinates of the marker, for example, based on the joint variable received by the first communication unit of the control device from the robot and the fixed position information of the marker. The fixed position information of the marker may be, for example, relative coordinates of the marker represented in the local coordinate system of the arm on which the marker is fixed. At the time of work, the reference table is searched from the position information of the object to obtain the three-dimensional position coordinates of the position, and the operation unit of the control device or the drive unit of the robot having the operation function takes the hand at the three-dimensional coordinate position. It is possible to calculate joint variables for moving.

Preferably, as many points as possible in the work space are registered in the reference table. This is because the probability of finding a record including an object in the reference table in the work process is increased and time is shortened. For example, if the size of the work space is 1 m 3 and the spatial resolution in the work space of the camera image is 1 mm, 10 ⁹ records can cover grid points at 1 mm intervals included in the work space. The density of the points registered in the reference table is preferably at least 512 times, more preferably at least 64 times, the density of the grid points of the cubic grid of the same pitch as the spatial resolution of the image, and particularly preferably 8 It is more than a fraction. The points registered in the lookup table most preferably include all grid points of a cubic grid of the same pitch as the spatial resolution of the image.

  Further, the distribution of points registered in the reference table may be dense in an area where the movement frequency of the hand is high in the work space, and sparse in an area where the movement frequency of the hand is low. Thereby, while shortening the time required for the calibration process, the probability of finding the position information of the object in the reference table in the work process is not excessively reduced.

  Also, in the distribution of points registered in the reference table, if the working space on the image is distorted due to a wide angle of view of the camera lens, etc., regions with large distortion may be dense and regions other than this may be sparse. . Specifically, it is preferable that the points registered in the reference table be registered at a high density such that linear interpolation can be performed from two adjacent points when estimating the posture parameters of points not in the reference table. This is because even if the work space on the image is distorted, the posture parameter can be estimated by simple calculation.

  In this embodiment, since calibration is performed using an actual camera and robot, it is not affected by an error due to processing, mounting, etc. at the time of system manufacture. In addition, since it is not necessary to measure the internal and external parameters of the camera and the installation position / orientation of the robot without passing through other coordinate systems such as world coordinates, it is possible to save on calibration work. There is a high degree of freedom in selecting and installing devices. For example, a camera with a large angle of view can be used to cover a wider range.

  In the present embodiment, since the hand is moved using the reference table, it is possible to reliably reach the point where the hand has visited once even after the second time. In addition, even if there are irregular error factors such as lens distortion, they are not affected. For example, when performing coordinate conversion using a transformation matrix etc. for lens distortion, even if distortion coefficients etc. are used, it is only possible to correct for modeled barrel distortion, pincushion distortion etc. Distortion effects can not be completely eliminated. By using the reference table, a camera with a cheaper lens system can be adopted.

  Next, a second embodiment of the present invention will be described based on FIGS. In the present embodiment, a mirror is used, and a single camera captures a direct image of the work space of the robot and a reflection image of the mirror.

  In FIG. 10, a robot system 11 according to this embodiment includes an articulated robot 20, a mirror 38, a single camera 32 for capturing a direct image of a working space of the robot and a reflected image by the mirror, and a control device 40. Have.

  The robot 20 is the same as in the first embodiment. The camera 32 can be the same as the first camera 30 or the second camera 31 of the first embodiment. The camera 32 and the mirror 38 are provided so as to be able to capture a direct image of the robot's work space and a reflected image from the mirror. FIG. 10 shows a state at the time of calibration, in which the robot holds the mark 50 on the hand 24.

  The configuration of the control device 40 is the same as that of FIG. 2 except that the function of the arithmetic unit is different. As shown in FIG. 11, the direct image of the marker 50 and the reflection image by the mirror 38 are reflected in the image which the second communication unit 42 receives from the camera 32 at the time of calibration. The arithmetic unit 43 processes one image, recognizes the marker 50 during calibration, recognizes the direct image and the reflection image of the object during the operation in the image, and calculates the position information of the marker or the object. Others are the same as in the first embodiment.

  FIG. 12 shows an example of the reference table. In each record of this reference table, coordinates (u_di, V_di), Coordinates as pixel position in the image of the reflection image (u_{m i}, V_mi), 6 joint variables (θ _1i, Θ_2i, Θ_3i, Θ_4i, Θ_5i, Θ_6i)It is included.

  Next, a third embodiment of the present invention will be described based on FIG. 13 and FIG. In this embodiment, the working space of the robot is imaged by one TOF camera.

  In FIG. 13, a robot system 12 according to the present embodiment includes an articulated robot 20, one TOF camera 33 that images a working space of the robot, and a control device 40. FIG. 13 shows the state at the time of calibration, in which the robot holds the mark 50 on the hand 24.

  The robot 20 is the same as in the first embodiment. The TOF camera 33 is a distance image generating camera based on a time of flight (TOF) method, and captures a two-dimensional image and a distance image in which the distance from the camera to a point captured in each pixel is recorded.

  The configuration of the control device 40 is the same as that of FIG. 2 except that the function of the arithmetic unit is different. The calculation unit 43 processes the two-dimensional image and the distance image received by the second communication unit 42 from the TOF camera 33, recognizes the marker 50 during calibration, and recognizes an object in the image at work, Get the location information of the object. Others are the same as in the first embodiment.

FIG. 14 shows an example of the reference table. Each record of this reference table is the coordinates (Cx _i , Cy _i , C d _i ) in the two-dimensional image of the marker and the distance image, and six joint variables (θ _1i , θ _2i ) as posture parameters in a robot with six degrees of freedom , Θ _3i , θ _4i , θ _5i , θ _6i ).

  Next, a fourth embodiment of the present invention will be described based on FIGS. In the present embodiment, the working space of the robot is imaged by one camera, and a set of coordinates of the marker in the image and a feature that changes according to the distance from the camera to the marker is used as position information. Specifically, the area of the sign in the image is used as the feature amount.

  In FIG. 15, a robot system 13 according to the present embodiment includes an articulated robot 20, a single camera 34 for imaging a work space of the robot, and a control device 40. The robot 20 is the same as in the first embodiment. The camera 34 can be the same as the first camera 30 or the second camera 31 of the first embodiment.

The configuration of the control device 40 is the same as that of FIG. 2 except that the function of the arithmetic unit is different. The computing unit 43 processes one image (FIG. 16) received by the second communication unit from the camera 34 at the time of calibration, and recognizes the marker 50 in the image at the time of calibration, and the area of the marker in the image Calculate (S _mi ). In addition, the ratio between the area of the object (S _wi ) in the image and the area of the marker (S _mi ) is separately calculated at at least one position. At the time of operation, the computing unit 43 recognizes an object in an image, calculates the area (S _wi ) of the object in the image, and further converts the area (S _mi ) of the marker 50. Then, the calculation unit searches the reference table using, as position information of the object, a combination of the pixel position in the image of the object and the area of the marker in the image when it is assumed that the mark is present at the position.

FIG. 17 shows an example of the reference table. Each record in this reference table has coordinates (u _i , v _i ) as pixel positions in the image, an area (S _mi ) of the marker in the image, and six joint variables as posture parameters in the robot with six degrees of freedom ( θ _1i , θ _2i , θ _3i , θ _4i , θ _5i , θ _6i ).

  Next, a fifth embodiment of the present invention will be described based on FIG. In this embodiment, two articulated robots and six cameras are used. The present embodiment can be considered as a combination of a plurality of robot systems of the first embodiment.

  In the robot system 14 shown in FIG. 18, two articulated robots 60 and 61 and six cameras 70 to 75 are connected to one control device 40. The camera 70 and the camera 71 are provided with a wide angle of view lens, and can capture work spaces of both the robot 60 and the robot 61. The camera 72 and the camera 73 are provided with a lens with a narrow angle of view, and can capture the working space of one robot 60 with high spatial resolution. The camera 74 and the camera 75 are provided with a lens with a narrow angle of view, and can capture the working space of the other robot 61 with high spatial resolution.

The control device 40 creates and stores four reference tables composed of records consisting of the following combinations.
Reference table A: Position information by camera 70, 71-Posture parameter of robot 60 Reference table B: Position information by camera 70, 71-Posture parameter of robot 61 Reference table C: Position information by camera 72, 73-Robot 60 posture parameters and reference table D: position information by the cameras 74 and 75-posture parameters of the robot 61

  As a result, the hand movement of the robot 60 can be roughly performed using the reference table A and using the reference table C in an area where precise position control is required. Similarly, the hand movement of the robot 61 can be roughly performed using the lookup table B and using the lookup table D in an area where precise position control is required.

  The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the technical idea thereof.

  For example, although the fifth embodiment combines a plurality of robot systems according to the first embodiment, the present invention is not limited to this, and the first to fourth embodiments can be freely combined.

10 to 14 Robot System 20 Articulated Robot 21 Arm 22 Link 23 Joint 24 Hand 25 Base 26 Drive 30 First Camera 31 Second Camera 32 34 Camera 33 TOF Camera 38 Mirror 40 Robot System Controller (Control Device)
41 1st communication unit (interface with robot)
42 Second communication unit (interface with camera)
43 operation unit 44 storage unit 45 third communication unit (interface with input / output device)
46 input device 47 output device 50 signs 60, 61 articulated robot 70-75 camera

Claims (28)

With articulated robots,
One or more cameras for imaging the work space of the robot;
And a control device,
At the time of calibration, the control device
Get the robot's attitude parameter,
Obtaining an image from the camera, recognizing a sign fixed to the robot in the image, and calculating position information of the sign;
Creating and storing a reference table in which the position information of the sign and the posture parameter of the robot are associated;
At the time of work, the control device
Obtaining an image from the camera, recognizing an object in the image, and calculating position information of the object;
The posture parameter for moving the hand of the robot to the object is acquired from the position information of the object with reference to the reference table,
Transmitting the acquired posture parameter to the robot;
Robot system. The robot system has a first camera and a second camera,
The position information of the sign and the position information of the object are a set of the position of the sign or the object of the pixel in the image by the first camera and the position of the pixel in the image by the second camera, respectively.
The robot system according to claim 1. The robot system has a first camera and a second camera,
The position information of the sign and the position information of the object are respectively based on the pixel position of the sign or the object in the image of the first camera, the image of the first camera, and the image of the second camera. It is a pair with distance information calculated based on the principle of surveying,
The robot system according to claim 1. The robot system has a first camera and a second camera,
A camera coordinate system in which position information of the sign and position information of the object are calculated based on an image of the sign or the object by the first camera and an image of the second camera based on the principle of triangulation. Three-dimensional position coordinates in
The robot system according to claim 1. The robot system further comprises a mirror within the angle of view of the camera and positioned such that the work space is visible when viewed from the camera;
The position information of the sign and the position information of the object are respectively a set of the position of the sign or the object of the direct image in the image by the camera and the position of the reflection image by the mirror.
The robot system according to claim 1. The camera is a distance image camera,
The position information of the sign and the position information of the object are a set of pixel positions and distance information of the sign or the object in the distance image by the distance image camera, respectively.
The robot system according to claim 1. The position information of the sign and the position information of the object are respectively a set of the position of the sign or the pixel position in the image of the object by the camera and the area in the image when the sign is at the position. is there,
The robot system according to claim 1. The posture parameter is a set of joint variables of the robot,
The robot system according to any one of claims 1 to 7. The posture parameter is three-dimensional position coordinates of the marker or the robot coordinate system of the object,
The robot system according to any one of claims 1 to 7. At the time of calibration of the articulated robot,
Get the robot's attitude parameter,
An image with a sign fixed to the robot is acquired from one or more cameras for imaging the work space of the robot, the sign is recognized in the image, and position information of the sign is calculated;
Create and store a reference table that associates the position information of the sign with the posture parameter of the robot.
Robot system controller. At the time of work of the robot
Obtaining an image from the camera, recognizing an object in the image, and calculating position information of the object;
The posture parameter for moving the hand of the robot to the object is acquired from the position information of the object with reference to the reference table,
Transmitting the acquired posture parameter to the robot;
The robot system control device according to claim 10. The robot system controller acquires an image from the first camera and the second camera,
The position information of the sign and the position information of the object are respectively a set of the position of the sign or the object of the pixel in the image by the first camera and the position of the pixel in the image by the second camera.
The robot system control device according to claim 11. The robot system controller acquires an image from the first camera and the second camera,
The position information of the sign and the position information of the object are respectively the pixel position of the sign or the object in the image by the first camera, and the image by the first camera and the image by the second camera. A pair with distance information calculated based on
The robot system control device according to claim 11. The robot system controller acquires an image from the first camera and the second camera,
A third order in a camera coordinate system in which position information of the sign and position information of the object are calculated based on an image by the first camera and an image by the second camera of the sign or the object respectively based on the principle of triangulation The original position coordinates,
The robot system control device according to claim 11. The camera captures a direct image of the work space and a reflected image from a mirror into a single image;
The position information of the sign and the position information of the object are respectively a set of the position of the sign or the object of the direct image in the image by the camera and the position of the reflection image by the mirror.
The robot system control device according to claim 11. The camera is a distance image camera,
The position information of the sign and the position information of the object are a set of the pixel position of the sign or the object in the image by the distance image camera and the distance information, respectively.
The robot system control device according to claim 11. The position information of the sign and the position information of the object are respectively a set of the position of the sign or the pixel position in the image of the object by the camera and the area in the image when the sign is at the position. is there,
The robot system control device according to claim 11. The posture parameter is a set of joint variables of the robot,
The robot system control device according to any one of claims 11 to 17. The posture parameter is three-dimensional position coordinates of the marker or the robot coordinate system of the object,
The robot system control device according to any one of claims 11 to 17. While operating the articulated robot on which the sign is fixed and moving the sign,
The working space of the robot is imaged by one or more cameras, the marker is recognized in an image photographed by the camera, position information of the marker is calculated, and position information of the marker and the robot at the time of the imaging Calibration step of sequentially registering in the reference table in association with the posture parameters of
The camera captures an image of a work space in which an object exists,
The object is recognized in the image captured by the camera, position information of the object is calculated, and from the position information of the object, the hand of the robot is referred to the object with reference to the reference table. Obtain the posture parameter for moving
A work process of transmitting the acquired posture parameter to the robot;
A robot system control method comprising: The one or more cameras are a first camera and a second camera,
The position information of the sign and the position information of the object are respectively a set of the position of the sign or the object of the pixel in the image by the first camera and the position of the pixel in the image by the second camera.
A robot system control method according to claim 20. The one or more cameras are a first camera and a second camera,
The position information of the sign and the position information of the object are respectively the pixel position of the sign or the object in the image by the first camera, and the image by the first camera and the image by the second camera. A pair with distance information calculated based on
A robot system control method according to claim 20. The one or more cameras are a first camera and a second camera,
A third order in a camera coordinate system in which position information of the sign and position information of the object are calculated based on an image by the first camera and an image by the second camera of the sign or the object respectively based on the principle of triangulation The original position coordinates,
A robot system control method according to claim 20. The camera captures the work space and a mirror arranged to reflect the work space in one image.
The position information of the sign and the position information of the object are respectively a set of the position of the sign or the object of the direct image in the image by the camera and the position of the reflection image by the mirror.
A robot system control method according to claim 20. The camera is a distance image camera,
The position information of the sign and the position information of the object are a set of pixel positions and distance information of the sign or the object in the distance image by the distance image camera, respectively.
A robot system control method according to claim 20. The position information of the sign and the position information of the object are respectively a set of the position of the sign or the pixel position in the image of the object by the camera and the area in the image when the sign is at the position. is there,
A robot system control method according to claim 20. The posture parameter is a set of joint variables of the robot,
The robot system control method according to any one of claims 20 to 26. The posture parameter is three-dimensional position coordinates of the marker or the robot coordinate system of the object,
The robot system control method according to any one of claims 20 to 26.

Images