JP6855491B2 - Robot system, robot system control device, and robot system control method - Google Patents

Description

The present invention relates to a robot system for moving the hand of an articulated robot to a target object. The present invention also relates to a control device and a control method for such a robot system.

Research is being conducted on a system in which an articulated robot recognizes the surrounding environment and performs work based on information obtained from a visual sensor such as a camera. For example, the work is such that the camera recognizes the objects piled up separately, and the hand at the tip of the arm of the robot is moved to the work object and grasped.

In a general robot system, in order to move the robot's hand to a target object, after obtaining the position information of the object with a camera or the like, this position information is converted from the camera coordinate system to the robot coordinate system. Tell the robot. Here, in order to convert from the camera coordinate system to the robot coordinate system, it is necessary to obtain the relative positional relationship between the camera and the robot in advance. Therefore, by calibration, a transformation matrix for performing coordinate transformation between the camera coordinate system and the world coordinate system and between the robot coordinate system and the world coordinate system is obtained. As a result, the position coordinates of the object in the camera coordinate system can be converted into the position coordinates in the robot coordinate system via the world coordinate system.

However, in order to determine the transformation matrix between the camera coordinate system and the world coordinate system, internal parameters such as the focal length of the camera, the image center, the image size (pixel size), and the distortion coefficient, and the external that represents the position and orientation of the camera You need to know the parameters. Therefore, for example, the plane pattern of a checkerboard or the like and the position of the camera have to be obtained by using a high-precision coordinate measuring device, which requires extremely laborious and costly work.

In order to determine the coordinate transformation between the robot coordinate system and the world coordinate system, it is also extremely troublesome to manually move the hand to the teaching point where the position coordinates in the world coordinate system are known and measure the posture parameters. And costly work was required.

On the other hand, in Patent Document 1, the robot arm device is operated so that the tip of the pointing rod gripped by the tip of the robot arm device draws a rectangle, and the position of each coordinate of the rectangle is measured in a stereo image. A method of calculating coordinate conversion data based on the data and obtaining a relative position-orientation relationship between the robot arm device and the stereo measuring device via an intermediate coordinate system based on a rectangle is described. According to Patent Document 1, since the data actually measured by the robot system is used instead of the design data, the robot arm device and the stereo image measurement device are accurate without being affected by errors due to processing and mounting that occur during system production. It is said that the relative position-orientation relationship can be obtained, and the time required for the calibration work can be shortened.

Japanese Unexamined Patent Publication No. 9-128549

However, in the method described in Patent Document 1, since the transformation matrix is calculated based on the measurement of the four vertices of the rectangle, the distortion aberration of the lens and the like cannot be accurately reproduced. Even if there is no distortion, the transformation matrix is used for coordinate conversion between the camera coordinate system and the intermediate coordinate system, and between the intermediate coordinate system and the robot coordinate system. It cannot be completely excluded.

The present invention has been made in consideration of the above, and it is not necessary to measure the position of the installed camera or robot, and the hand of the articulated robot is based on the camera image without being affected by lens distortion or the like. An object of the present invention is to provide a robot system, a robot system control device, and a robot system control method capable of moving to a target position.

For the above purpose, in the present invention, the robot is controlled by using a reference table in which the position information obtained from the image of the camera and the posture parameter of the articulated robot are directly associated with each other.

Specifically, the robot system of the present invention includes an articulated robot, one or more cameras that image the working space of the robot, and a control device. Then, at the time of calibration, the control device acquires the posture parameter of the robot, acquires an image from the camera, recognizes a sign fixed to the robot in the image, and obtains position information of the sign. Calculate and create and store a reference table that associates the position information of the sign with the posture parameter of the robot. Then, at the time of work, the control device acquires an image from the camera, recognizes the object in the image, calculates the position information of the object, and uses the position information of the object as the reference table. With reference to, the posture parameter for moving the hand to the object is acquired, and the acquired posture parameter is transmitted to the robot.

Here, the posture parameter of the robot is a parameter related to the posture of the robot and can at least determine the position of the hand. Further, the sign fixed to the robot means a sign having a fixed positional relationship with the robot's hand or arm, such as a sign held by the hand.

Preferably, the robot system has a first camera and a second camera, and the position information of the sign and the position information of the object are pixels in the image of the sign or the object by the first camera, respectively. It is a set of a position and a pixel position in an image taken by the second camera.

Alternatively, the robot system has a first camera and a second camera, and the position information of the sign and the position information of the object are the pixel positions of the sign or the object in the image by the first camera, respectively. And the distance information calculated from the image taken by the first camera and the image taken by the second camera based on the principle of triangulation.

Alternatively, the robot system has a first camera and a second camera, and the position information of the sign and the position information of the object are an image of the sign or the object by the first camera and the second camera, respectively. It may be the three-dimensional position coordinates in the camera coordinate system calculated from the image by the camera based on the principle of triangulation.

Alternatively, the robot system further includes a mirror that is within the angle of view of the camera and is arranged so that the work space is reflected when viewed from the camera, and the position information of the sign and the above. The position information of the object may be a set of the pixel position of the direct image of the marker or the object in the image taken by the camera and the pixel position of the reflected image taken by the mirror, respectively.

Alternatively, the camera is a distance image camera, and the position information of the sign and the position information of the object are the pixel position and the distance information of the sign or the object in the distance image by the distance image camera, respectively. It may be a set of.

Alternatively, the position information of the sign and the position information of the object are the pixel position in the image of the sign or the object by the camera and the area in the image when the sign is in the position, respectively. It may be a pair.

In any of the robot systems, the posture parameter is preferably a set of joint variables of the robot.

Alternatively, in any of the robot systems, the posture parameter may be three-dimensional position coordinates in the robot coordinate system of the sign or the object.

The robot system control device of the present invention acquires a posture parameter of the robot at the time of calibrating the articulated robot, and a marker fixed to the hand of the robot from one or more cameras that image the work space of the robot. Is acquired, the sign is recognized in the image, the position information of the sign is calculated, and a reference table in which the position information of the sign is associated with the posture parameter of the robot is created and stored. ..

Preferably, the robot system control device of the present invention acquires an image from the camera during the working of the robot, recognizes the object in the image, calculates the position information of the object, and calculates the position information of the object. With reference to the reference table, the posture parameter for moving the hand to the object is acquired from the position information of the robot, and the acquired posture parameter is transmitted to the robot.

In another robot system control device of the present invention, a first communication unit that transmits and / or receives a posture parameter of an articulated robot to the robot and an image of a sign fixed to the robot are captured from a camera. A second communication unit to receive, a calculation unit that recognizes the sign from the image, calculates the position information of the sign, and creates a reference table that associates the position information of the hand with the posture parameter, and the reference. It has a storage unit for storing a table.

Preferably, in another robot system control device of the present invention, the second communication unit receives an image of the object from the camera, the calculation unit recognizes the object from the image, and the object is displayed. The position information of the object is calculated, the reference table is referred to, the posture parameter for moving the hand of the robot to the object is acquired from the position information of the object, and the first communication unit performs the above. The attitude parameter is transmitted to the robot.

The robot system control method of the present invention includes a calibration step and a working step. In the calibration step, an articulated robot having a fixed sign is operated to move the sign, the work space of the robot is imaged by one or more cameras, and the sign is included in the image captured by the camera. It recognizes and calculates the position information of the sign, associates the position information of the sign with the posture parameter of the robot at the time of the imaging, and sequentially registers it in the reference table. In the work process, the work space in which the object exists is imaged by the camera, the object is recognized in the image captured by the camera, the position information of the object is calculated, and the position information of the object is calculated from the position information of the object. , The posture parameter for moving the hand of the robot to the object is acquired with reference to the reference table, and the acquired posture parameter is transmitted to the robot.

According to the robot system, the robot system control device, or the robot system control method of the present invention, a reference table in which the position information by the camera and the posture parameter of the robot are directly related is created by using an actual camera and the robot, and the reference table is created. Operate the robot based on the table. Since the reference table is created using an actual camera and robot, the hand can be moved to the target position without being affected by errors during system fabrication, irregular lens distortion, and the like. In addition, since it does not go through other coordinate systems such as world coordinates, it is not necessary to measure the internal / external parameters of the camera, the installation position / orientation of the robot, etc., and the calibration work can be saved.

It is an overall block diagram of the robot system of 1st Embodiment of this invention. It is a functional block diagram of the robot system control device of 1st Embodiment of this invention. It is a flow chart of the robot control method of 1st Embodiment of this invention. It is a flow chart of the calibration process of the robot control method of 1st Embodiment of this invention. It is a flow chart of the hand movement in the work process of the robot control method of 1st Embodiment of this invention. It is an image at the time of calibration by 1st Embodiment of this invention. It is a figure which shows the example of the reference table of the 1st Embodiment of this invention. It is a figure which shows another example of the reference table of 1st Embodiment of this invention. It is a figure which shows another example of the reference table of 1st Embodiment of this invention. It is an overall block diagram of the robot system of 2nd Embodiment of this invention. It is an image at the time of calibration by the 2nd Embodiment of this invention. It is a figure which shows the example of the reference table of the 2nd Embodiment of this invention. It is an overall block diagram of the robot system of 3rd Embodiment of this invention. It is a figure which shows the example of the reference table of the 3rd Embodiment of this invention. It is an overall block diagram of the robot system of 4th Embodiment of this invention. It is an image at the time of calibration by the 4th Embodiment of this invention. It is a figure which shows the example of the reference table of the 4th Embodiment of this invention. It is an overall block diagram of the robot system of 5th Embodiment of this invention.

The first embodiment of the present invention will be described with reference to FIGS. 1 to 9.

In FIG. 1, the robot system 10 of the present embodiment includes an articulated robot 20, a first camera 30 and a second camera 31 that capture an image of the robot's work space, and a robot system control device 40. Hereinafter, the "robot system control device" is simply referred to as a "control device".

The robot 20 is fixed to the work place. The robot 20 has a base 25 for fixing the robot to the workplace, an arm 21 to which a plurality of links 22 are connected by joints 23, a hand 24 connected to the tip of the arm, and a drive unit 26. Each joint of the robot is provided with an actuator (not shown) such as a servomotor and a sensor (not shown) that detects joint variables of the joint. Here, the joint variable means the displacement of the joint, specifically, the joint angle (inter-link angle) of the rotary joint and the inter-link distance of the linear motion joint. The hand 24 is also called a hand or an end effector. FIG. 1 shows a state at the time of calibration, and the robot holds the sign 50 in the hand.

The drive unit 26 of the robot 20 drives the arm and the hand. The drive unit having the simplest configuration consists of a drive device such as a servo amplifier, and drives the actuator of each joint based on the joint variables received from the outside. Alternatively, the drive unit further has a memory for storing a CPU, a program, etc. that perform various arithmetic processes, and can receive the three-dimensional position coordinates of the hand in the robot coordinate system from the outside and calculate joint variables. There may be. Here, the robot coordinate system is a coordinate system based on the fixed point of the robot, preferably a coordinate system based on the base 25. The robot coordinate system may be an orthogonal coordinate system or various polar coordinate systems. Hereinafter, the term "three-dimensional position coordinates" simply means three-dimensional position coordinates in the robot coordinate system.

In the robot 20, the base 25 is fixed to the work place and does not move during calibration and work. Therefore, the position of the hand 24 can be represented by a combination of joint variables and three-dimensional position coordinates. For example, when the degree of freedom of the arm 21 is 3, the position of the hand and the posture of the arm are determined by the three joint variables of the arm. Alternatively, when the degree of freedom of the arm 21 is 3 and the degree of freedom of the hand 24 is 3, the position and posture of the hand and the posture of the arm are determined by the six joint variables of the arm and the hand. The three-dimensional position coordinates of the hand represent the position of the hand. Such a parameter related to the posture of the robot and capable of determining at least the position of the hand is referred to as a "robot posture parameter" in the present specification.

The type of the robot is not particularly limited, and in addition to the serial link type vertical articulated robot illustrated in FIG. 1, various articulated robots such as the horizontal articulated type and the parallel link type can be used.

The first camera 30 and the second camera 31 are fixed to the work place. The two cameras are installed so as to image the work space of the robot 20. Here, the robot work space refers to an area in which the hand 24 of the robot can reach, in which the hand may move during work. The two cameras 30 and 31 are cameras capable of capturing a two-dimensional image.

With reference to FIG. 2, the control device 40 includes a first communication unit 41, a second communication unit 42, a calculation unit 43, a storage unit 44, and a third communication unit 45.

The first communication unit 41 is connected to the drive unit 26 of the robot 20 by a signal line and communicates with the robot. The first communication unit transmits the posture parameter corresponding to the target position to the robot, and instructs the robot to move the hand 24 to the target position. In addition, the first communication unit receives posture parameters such as joint variables from the robot. The second communication unit 42 is connected to the first camera 30 and the second camera 31 by a signal line, and communicates with the cameras. The second communication unit transmits an imaging instruction to the camera and receives an image from the camera.

The calculation unit 43 processes the images received from the cameras 30 and 31 by the second communication unit 42. At the time of calibration, the calculation unit recognizes the sign 50 in the image, calculates the position information thereof, and receives the position information and the posture parameter transmitted by the first communication unit 41 to the robot or the first communication unit receives from the robot. Create a reference table associated with the posture parameters. More precisely, the calculation unit creates one record that associates the position information of the sign with the posture parameter of the robot, and additionally registers it in the existing reference table. At the time of work, the calculation unit recognizes the object in the image and calculates the position information thereof. The calculation unit refers to the reference table and acquires the posture parameter for moving the robot's hand to the object from the position information of the object.

The storage unit 44 stores the reference table. As the storage unit, a rewritable and randomly accessible auxiliary storage device such as a hard disk device can be preferably used. The third communication unit 45 is connected to the input device 46 and the output device 47 by a signal line, and communicates with the operator. The third communication unit receives an instruction from the operator from the input device 46, and transmits the status of the robot system and the like to the output device 47 to the output device 47.

The control device 40 does not necessarily have to be one physical device, and a plurality of control devices 40 may share the processing. Further, the control device may be integrally formed with the drive unit 26 of the robot.

Next, the robot system control method of the present embodiment will be described. The reference reference numerals of the respective parts are the reference numerals shown in FIGS. 1 and 2.

In FIG. 3, the robot system control method of the present embodiment includes installation of a robot and a camera in a workplace, a calibration process, and a work process using the robot.

First, the robot 20, the first camera 30, and the second camera 31 are installed in the workplace. At this time, the two cameras are installed at a position and orientation in which the robot's work space can be imaged. The angle formed by the optical axes of the two cameras is preferably 30 to 150 degrees, more preferably 60 to 120 degrees, and particularly preferably 70 to 110 degrees. This is because the closer the directions of the two cameras are to parallel or opposite, the lower the spatial resolution of the cameras in the optical axis direction. However, the more different the orientations of the two cameras, the more likely it is that occlusion will occur where part of the sign will be a blind spot. The orientation of the two cameras may be appropriately determined according to the shape of the sign and the work space so that occlusion is unlikely to occur and the required spatial resolution can be secured. The robot and camera are fixed to the workplace through the following calibration and work steps.

FIG. 4 shows the flow of the calibration process. In the calibration process, a reference table that associates the position information obtained from the camera with the posture parameters of the robot is created.

First, the sign 50 is fixed to the hand 24. To fix the sign to the hand, grasp the sign with the hand, hold the sign with the hand by suction or other methods, fix the object to be the sign to the hand, and become a sign on a part of the hand. It includes drawing a mark and using a characteristic part of the hand as a sign. The number of signs may be plural. When the sign is gripped by a hand, the sign can be an object itself or an object having a shape similar to the object, a spherical object, a pointer, or the like. The sign may be fixed to an arm which is a movable part of the robot.

The sign is preferably fixed so that the position of the sign coincides with the gripping position of the hand. This is because the sign position registered in the reference table and the target position for hand movement during work match. In FIG. 1, the spherical sign 50 is fixed by being gripped by the hand, and the position of the sign coincides with the gripping position of the hand. In the present embodiment, the description will be continued below assuming that the marking position and the gripping position of the hand match.

Next, the robot is operated according to the instruction from the first communication unit 41 of the control device 40 to move the hand 24 and the sign 50 in the work space. At this time, it is not necessary to accurately determine the movement route and the stop position of the sign in advance, and it is sufficient to determine so that the sign makes a round in the work space. The range in which the sign is moved can be input by the operator from the input device 46 and transmitted to the control device. At that time, the operator may directly input the joint variable corresponding to the point on the movement path, or the operator inputs the three-dimensional position coordinates of the point on the movement path to drive the calculation unit 43 or the robot. Part 26 may calculate joint variables.

Then, while moving the sign within the above range, a reference table is created with the sign stopped and / or moving.

For example, the method of creating a reference table when the sign is stopped is as follows. The posture parameter registered in the reference table may be a posture parameter transmitted from the first communication unit 41 to the robot 20, or may be a posture parameter received from the robot by the first communication unit. For example, when the sign stops at the position instructed by the control device, the control device 40 already knows the posture parameter transmitted from the first communication unit 41 to the robot, so even if this posture parameter is registered in the reference table. Good. Further, for example, even when the first communication unit transmits the three-dimensional position coordinates of the stop position to the robot, the first communication unit receives the joint variable from the robot when the sign is stopped, and registers this joint variable in the reference table. You may.

In parallel with the acquisition of the attitude parameter, the second communication unit 42 transmits an imaging instruction to the cameras 30 and 31, and receives an image when the sign is stopped from the cameras 30 and 31. FIG. 6 shows images received by the control device 40 from the cameras 30 and 31. A sign 50 is recorded on the two images. The calculation unit 43 of the control device processes the image received from the camera and recognizes the sign in the image. As a method for recognizing a marker, a known method such as feature point extraction or template matching can be used. The calculation unit calculates the set of pixel positions (u _1i , v _{1 i} , u _2i , v _2i ) of the sign in the two images as the position information of the sign.

For example, the method of creating a reference table while the sign is moving is as follows. The posture parameters registered in the reference table are received by the first communication unit 41 from the robot. When the imaging timing of the camera and the acquisition timing of the robot attitude parameter can be synchronized, the first communication unit 41 and the second communication unit 42 specify the same time for the robot and the cameras 30 and 31, respectively, and the attitude parameter or the image is obtained. Can be requested to be sent. Further, the first communication unit 41 and the second communication unit 42 may receive posture parameters or images from the robot or the camera, respectively, based on one synchronization signal. The synchronization signal may be generated by a synchronization signal generation unit (not shown) in the control device 40, or may be generated by either the first or second communication unit. When the cameras 30 and 31 can generate an image with an imaging time, the second communication unit receives the image from the two cameras every moment, and the first communication unit obtains the posture parameter from the robot. Images at the same time can be selected. The calculation unit 43 recognizes the sign 50 in the image and calculates the position information thereof.

As described above, for one sign position, the position information by the camera and the posture parameter of the robot corresponding to the position can be obtained. The calculation unit 43 of the control device creates one record by associating the position information with the posture parameter, and additionally registers the record in the reference table stored in the storage unit 44.

The sign 50 is circulated in the work space, and when the sign reaches the scheduled end position, the calibration is finished. It is preferable that the records in the reference table are sorted by using the position information as a key at the end of calibration. This is to streamline the search during work.

FIG. 7 shows an example of a reference table. Each record in this reference table has coordinates as pixel positions in the first camera image (u _1i , v _1i ) as position information, coordinates as pixel positions in the second camera image (u _2i , v _2i ), and The posture parameter consists of six joint angles (θ _1i , θ _2i , θ _3i , θ _4i , θ _5i , θ _6i ) in a robot with six degrees of freedom.

FIG. 5 shows the flow of hand movement in the work process. In the work process, the robot's hand is moved to the work object based on the position information calculated from the camera image.

First, the control device 40 transmits an imaging instruction from the second communication unit 42 to the cameras 30 and 31, and the camera images the work space. There is an object in the work space. The second communication unit receives the image from the camera. The calculation unit 43 processes the two images and recognizes the object in the images. The method of recognizing an object can be performed in the same manner as the method of recognizing a sign. The calculation unit calculates the position coordinates of the object in each camera coordinate system as the position information of the object that is the target of the hand movement.

The calculation unit 43 searches the reference table stored in the storage unit 44 from the position information of the object. If a record containing the position information of the object is found in the reference table, the calculation unit acquires the posture parameter corresponding to the position information. If the record containing the position information of the object is not found in the reference table, the calculation unit may estimate and acquire the attitude parameter corresponding to the position information of the object. For example, the posture parameters can be estimated by interpolating between the posture parameters in a plurality of records having close position information. The first communication unit 41 transmits a posture parameter to the robot and instructs the robot to move the hand to the object.

If the posture parameter is estimated, the position of the hand may be fine-tuned after the robot operates according to the posture parameter. For fine-tuning the hand position, known methods such as those described in International Publication No. 2013/176212 can be used.

When the hand position is finely adjusted, it is preferable to feed back the result to the reference table. Specifically, after the hand reaches the object through fine adjustment, the first communication unit 41 of the control device sends a posture parameter request to the robot, receives the posture parameter from the robot, and the calculation unit 43 targets the target. The position information of the object and the posture parameter may be additionally registered in the reference table of the storage unit 44 in association with each other.

Here, the reference table will be described in more detail.

In the reference table, the position information obtained from the camera image of the point in the work space of the robot and the posture parameter of the robot corresponding to the point are registered in association with each other.

The position information registered in the reference table is not limited to the set of two pixel coordinates as shown in FIG. 7. As shown in FIG. 8, for example, the position information is _{calculated from the pixel positions (u 1i} , v _1i ) in the image taken by the first camera, the image taken by the first camera, and the image taken by the second camera based on the principle of triangulation. distance information may be a set of a (d _i) that is. Alternatively, the position information may be three-dimensional position coordinates in the camera coordinate system calculated based on the principle of triangulation from the image taken by the first camera and the image taken by the second camera. In these cases, it is necessary to know the distance between the centers of the first camera and the second camera (base line length) and the epipolar line in the image. For example, a commercially available stereo camera in which the first camera and the second camera are integrated, etc. When using, the geometric parameters of such a camera are adjusted in advance, and no internal calibration work is required.

An example of posture parameters registered in the reference table is a robot joint variable. The joint variables are not limited to the set of joint angles as shown in FIG. When joint variables are used as posture parameters, the joint angles (inter-link angles) for rotary joints and the inter-link distances for linear motion joints are used as joint variables, and joint variables according to the types of joints that make up the robot are combined.

When registering the joint variables in the reference table, it is preferable that the position of the marker at the time of calibration and the gripping position of the hand match as shown in FIG. This is because the position information of the sign and the joint variable in the state where the sign is actually grasped can be associated and registered in the reference table. In this case, if the robot is instructed at the joint variable obtained by searching the reference table from the position information of the object during the work, the hand can be moved to the target position for grasping the object. Furthermore, even if there is bending of the arm due to gravity, the influence of bending etc. is already incorporated in the position information and joint variables registered in the reference table, so no special consideration is required during work.

On the other hand, when the position of the sign at the time of calibration and the gripping position of the hand do not match, for example, when the sign is fixed to a part different from the gripping position such as the side surface of the hand, the position information of the sign is obtained. The joint variable of the time cannot be registered in the reference table as it is. This is because the joint variable does not correspond to the position of the marker. In other words, even if the robot is instructed on the joint variable when the position information of the sign is obtained, the hand cannot grasp the object at the position of the sign. In this case, the three-dimensional position coordinates of the sign are once obtained from the joint variable when the position information of the sign is obtained, converted to the joint variable corresponding to the sign position based on the design data of the robot, and registered in the reference table. be able to. At that time, the joint variables can be obtained in consideration of geometrical restraint conditions for avoiding obstacles in the workplace.

Another example of the attitude parameter registered in the reference table is the three-dimensional position coordinates in the robot coordinate system of the marker. The robot coordinate system may be a coordinate system that does not fluctuate throughout the calibration and work process. Preferably, a coordinate system with reference to the base 25 of the robot can be used. The robot coordinate system may be a Cartesian coordinate system or a polar coordinate system. As an example, FIG. 9 shows a reference table in which _{the Cartesian coordinates (Rx i} , Ry _i , Rz _i ) of the marker position in the robot coordinate system are used as posture parameters. Representing the attitude parameters with the three-dimensional position coordinates of the sign has the advantage that the number of parameters can be reduced.

The three-dimensional position coordinates of the sign can be calculated from the joint variables at that time as long as the sign is fixed to the robot, regardless of whether the sign position matches or does not match the gripping position of the hand. The three-dimensional position coordinates of the sign can be calculated by the calculation unit based on, for example, the joint variable received from the robot by the first communication unit of the control device and the fixed position information of the sign. The fixed position information of the sign may be, for example, the relative coordinates of the sign represented by the local coordinate system of the arm on which the sign is fixed. At the time of work, the reference table is searched from the position information of the object, the three-dimensional position coordinates of the position are acquired, and the calculation unit of the control device or the drive unit of the robot having the calculation function moves the hand to the three-dimensional coordinate position. The joint variables to be moved can be calculated.

It is preferable that as many points in the workspace as possible are registered in the reference table. This is because the probability of finding a record containing an object in the reference table in the work process increases, and the time is shortened. For example, the size is 1m cubic work space, if the spatial resolution in the working space of the camera image of 1mm, can be at 10 ^nine records, covering the lattice points of 1mm intervals included in the working space. The density of the points registered in the reference table is preferably 1/512 or more, more preferably 1/64 or more, and particularly preferably 1/64 or more of the density of the lattice points of the cubic lattice having the same pitch as the spatial resolution of the image. It is more than one-third. The points registered in the reference table most preferably include all the lattice points of the cubic lattice having the same pitch as the spatial resolution of the image.

In addition, the distribution of points registered in the reference table may be dense in the areas of the work space where the hand moves frequently, and may be sparse in the areas where the hand moves frequently. This is because the time required for the calibration process is shortened, and the probability of finding the position information of the object in the reference table in the work process is not excessively reduced.

In addition, the distribution of points registered in the reference table may be such that when the work space on the image is distorted due to the wide angle of view of the camera lens, the areas with large distortion may be dense, and the areas where the distortion is not large may be sparse. .. Specifically, it is preferable that the points registered in the reference table are registered at such a high density that linear interpolation can be performed from two adjacent points when estimating the attitude parameters of points not in the reference table. This is because the posture parameters can be estimated by a simple calculation even if the work space on the image is distorted.

In this embodiment, since calibration is performed using an actual camera and robot, it is not affected by errors due to processing, mounting, etc. at the time of system production. In addition, since it is not necessary to measure the internal and external parameters of the camera, the installation position and orientation of the robot, etc. without going through other coordinate systems such as world coordinates, it is possible to save labor in calibration work. Great degree of freedom in device selection and installation. For example, a camera with a large angle of view can be used to cover a wider range.

In the present embodiment, since the hand is moved by using the reference table, the point where the hand has visited once can be surely reached even after the second time. Further, even if there is an irregular error factor due to lens distortion or the like, it is not affected by it. For example, when performing coordinate conversion using a transformation matrix or the like for lens distortion, even if the distortion aberration coefficient or the like is used, it can only be corrected for the modeled barrel distortion, thread winding distortion, etc., and is irregular. The effects of such distortion cannot be completely eliminated. By using the reference table, it is possible to adopt a camera equipped with a cheaper lens system.

Next, the second embodiment of the present invention will be described with reference to FIGS. 10 to 12. In this embodiment, a mirror is used, and one camera captures a direct image of the robot's work space and a reflected image by the mirror.

In FIG. 10, the robot system 11 of the present embodiment includes an articulated robot 20, a mirror 38, a camera 32 that captures a direct image of the robot's work space and a reflection image by the mirror, and a control device 40. Have.

The robot 20 is the same as that of the first embodiment. As the camera 32, the same camera as the first camera 30 or the second camera 31 of the first embodiment can be used. The camera 32 and the mirror 38 are installed so as to be able to capture a direct image of the robot's work space and a reflected image by the mirror. FIG. 10 shows the state at the time of calibration, and the robot holds the sign 50 on the hand 24.

The configuration of the control device 40 is the same as that in FIG. 2, but the function of the calculation unit is different. As shown in FIG. 11, the direct image of the sign 50 and the reflected image by the mirror 38 are reflected in the image received from the camera 32 by the second communication unit 42 at the time of calibration. The calculation unit 43 processes one image, recognizes the sign 50 at the time of calibration, and recognizes the direct image and the reflected image of the object in the image at the time of work, and calculates the position information of the sign or the object. Others are the same as those in the first embodiment.

FIG. 12 shows an example of a reference table. Each record in this reference table has coordinates (u) as pixel positions in the image of the direct image of the marker._di, V_di), Coordinates as pixel positions in the reflected image (u)_{m i}, V_mi), 6 joint variables (θ) as posture parameters in a robot with 6 degrees of freedom _1i， Θ_2i， Θ_3i， Θ_4i， Θ_5i， Θ_6i)It is included.

Next, a third embodiment of the present invention will be described with reference to FIGS. 13 and 14. In this embodiment, one TOF camera captures the working space of the robot.

In FIG. 13, the robot system 12 of the present embodiment includes an articulated robot 20, a TOF camera 33 that captures an image of the robot's work space, and a control device 40. FIG. 13 shows the state at the time of calibration, and the robot holds the sign 50 on the hand 24.

The robot 20 is the same as that of the first embodiment. The TOF camera 33 is a distance image generation camera based on the time-of-flight (TOF) method, and captures a two-dimensional image as well as a distance image in which the distance from the camera to a point captured by each pixel is recorded.

The configuration of the control device 40 is the same as that in FIG. 2, but the function of the calculation unit is different. The calculation unit 43 processes the two-dimensional image and the distance image received from the TOF camera 33 by the second communication unit 42, recognizes the sign 50 at the time of calibration, and recognizes the object in the image at the time of work, and recognizes the sign or the target in the image. Get the position information of an object. Others are the same as those in the first embodiment.

FIG. 14 shows an example of a reference table. Each record in this lookup table, the coordinates in the labeling of the two-dimensional image and the distance image _{(Cx i, Cy i, Cd} i), 6 as orientation parameter in the six degrees of freedom robotic one joint variables (θ _1i, θ _2i , Θ _3i , θ _4i , θ _5i , θ _6i ).

Next, a fourth embodiment of the present invention will be described with reference to FIGS. 15 to 17. In this embodiment, the work space of the robot is imaged by one camera, and the set of the coordinates of the sign in the image and the feature amount that changes depending on the distance from the camera to the sign is used as the position information. Specifically, the area of the sign in the image is used as the feature amount.

In FIG. 15, the robot system 13 of the present embodiment includes an articulated robot 20, a camera 34 that captures a working space of the robot, and a control device 40. The robot 20 is the same as that of the first embodiment. As the camera 34, the same camera as the first camera 30 or the second camera 31 of the first embodiment can be used.

The configuration of the control device 40 is the same as that in FIG. 2, but the function of the calculation unit is different. At the time of calibration, the calculation unit 43 processes one image (FIG. 16) received from the camera 34 by the second communication unit, recognizes the sign 50 in the image at the time of calibration, and determines the area of the sign in the image. (S _mi ) is calculated. In addition, the ratio of the area of the object (S _wi ) and the area of the sign (S _mi ) in the image is separately calculated at at least one position. At the time of work, the calculation unit 43 recognizes the object in the image _{, calculates the area (S wi} ) of the object in the image, and further converts it into the area (S _{mi) of the sign 50.} Then, the calculation unit searches the reference table using the set of the pixel position in the image of the object and the area of the sign in the image when it is assumed that the mark is present at the position as the position information of the object.

FIG. 17 shows an example of a reference table. Each record in this lookup table, the coordinates of the pixel position in the image (u _{i, v i),} labeling of the area in the image (S _mi), 6 6 one joint variables as orientation parameter in the degree of freedom of the robot ( It consists of _{θ 1i} , θ _2i , θ _3i , θ _4i , θ _5i , and θ _6i).

Next, a fifth embodiment of the present invention will be described with reference to FIG. In this embodiment, two articulated robots and six cameras are used. The present embodiment can be considered as a combination of a plurality of robot systems of the first embodiment.

In the robot system 14 shown in FIG. 18, two articulated robots 60 and 61 and six cameras 70 to 75 are connected to one control device 40. The camera 70 and the camera 71 are provided with a lens having a wide angle of view, and can image the work spaces of both the robot 60 and the robot 61. The camera 72 and the camera 73 are provided with a lens having a narrow angle of view, and can image the work space of one of the robots 60 with high spatial resolution. The camera 74 and the camera 75 are provided with a lens having a narrow angle of view, and can image the work space of the other robot 61 with high spatial resolution.

The control device 40 creates and stores four reference tables composed of records composed of the following combinations.
-Reference table A: Position information by cameras 70 and 71-Attitude parameters of robot 60-Reference table B: Position information by cameras 70 and 71-Attitude parameters of robot 61-Reference table C: Position information by cameras 72 and 73-Robot Attitude parameters of 60 ・ Reference table D: Position information by cameras 74 and 75-Attitude parameters of robot 61

As a result, the hand movement of the robot 60 can be performed by using the reference table A for rough movement and by using the reference table C in the area where precise position control is required. Similarly, the hand movement of the robot 61 can be performed by using the reference table B for rough movement and by using the reference table D in the area where precise position control is required.

The present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of its technical idea.

For example, the fifth embodiment combines a plurality of robot systems of the first embodiment, but the present invention is not limited to this, and the first to fourth embodiments can be freely combined.

10-14 Robot system 20 Articulated robot 21 Arm 22 Link 23 Joint 24 Hand 25 Base 26 Drive unit 30 1st camera 31 2nd camera 32, 34 Camera 33 TOF camera 38 Mirror 40 Robot system control device (control device)
41 First communication unit (interface with robot)
42 Second communication unit (interface with camera)
43 Calculation unit 44 Storage unit 45 Third communication unit (interface with input / output device)
46 Input device 47 Output device 50 Sign 60, 61 Articulated robot 70-75 Camera

Claims (28)

With articulated robots
One or more cameras that image the work space of the robot,
Has a control device
The control device is used at the time of calibration.
Acquire the posture parameters of the robot and
An image is acquired from the camera, a sign fixed to the robot is recognized in the image, and the position information of the sign is calculated.
A reference table in which the position information of the sign and the posture parameter of the robot are associated with each other is created and stored, and stored.
The control device is used during work.
An image is acquired from the camera, an object is recognized in the image, and the position information of the object is calculated.
From the position information of the object, the posture parameter for moving the hand of the robot to the object is acquired by referring to the reference table.
The acquired posture parameter is transmitted to the robot.
Robot system. The robot system has a first camera and a second camera.
The position information of the sign and the position information of the object are a set of the pixel position of the sign or the object in the image taken by the first camera and the pixel position in the image taken by the second camera, respectively.
The robot system according to claim 1. The robot system has a first camera and a second camera.
The position information of the sign and the position information of the object are triangular from the pixel positions of the sign or the object in the image taken by the first camera, the image taken by the first camera, and the image taken by the second camera, respectively. It is a set with distance information calculated based on the principle of surveying,
The robot system according to claim 1. The robot system has a first camera and a second camera.
A camera coordinate system in which the position information of the sign and the position information of the object are calculated from the image of the sign or the object by the first camera and the image by the second camera, respectively, based on the principle of triangulation. Is the three-dimensional position coordinates in
The robot system according to claim 1. The robot system further includes a mirror that is within the angle of view of the camera and is arranged so that the work space is reflected when viewed from the camera.
The position information of the sign and the position information of the object are a set of a pixel position of a direct image of the sign or the object in an image taken by the camera and a pixel position of a reflected image taken by the mirror, respectively.
The robot system according to claim 1. The camera is a distance image camera
The position information of the sign and the position information of the object are a set of the pixel position and the distance information of the sign or the object in the distance image by the distance image camera, respectively.
The robot system according to claim 1. The position information of the sign and the position information of the object are a set of a pixel position in the image of the sign or the object by the camera and an area in the image when the sign is in the position, respectively. is there,
The robot system according to claim 1. The posture parameter is a set of joint variables of the robot.
The robot system according to any one of claims 1 to 7. The attitude parameter is a three-dimensional position coordinate in the robot coordinate system of the sign or the object.
The robot system according to any one of claims 1 to 7. When calibrating an articulated robot
Acquire the posture parameters of the robot and
An image showing a sign fixed to the robot is acquired from one or more cameras that image the work space of the robot, the sign is recognized in the image, and the position information of the sign is calculated.
Create and store a reference table that associates the position information of the sign with the posture parameter of the robot.
Robot system control device. During the work of the robot
An image is acquired from the camera, an object is recognized in the image, and the position information of the object is calculated.
From the position information of the object, the posture parameter for moving the hand of the robot to the object is acquired by referring to the reference table.
The acquired posture parameter is transmitted to the robot.
The robot system control device according to claim 10. The robot system control device acquires images from the first camera and the second camera, and obtains images.
The position information of the sign and the position information of the object are a set of the pixel position in the image taken by the first camera and the pixel position in the image taken by the second camera of the sign or the object, respectively.
The robot system control device according to claim 11. The robot system control device acquires images from the first camera and the second camera, and obtains images.
The position information of the sign and the position information of the object are the principle of triangulation from the pixel position in the image of the sign or the object, the image of the first camera, and the image of the second camera, respectively. It is a set with the distance information calculated based on,
The robot system control device according to claim 11. The robot system control device acquires images from the first camera and the second camera, and obtains images.
The position information of the sign and the position information of the object are cubic in the camera coordinate system calculated based on the principle of triangulation from the image taken by the first camera and the image taken by the second camera of the sign or the object, respectively. Original position coordinates,
The robot system control device according to claim 11. The camera captures a direct image of the work space and a reflected image by a mirror into one image.
The position information of the sign and the position information of the object are a set of a pixel position of a direct image of the sign or the object in an image taken by the camera and a pixel position of a reflected image taken by the mirror, respectively.
The robot system control device according to claim 11. The camera is a distance image camera
The position information of the sign and the position information of the object are a set of the pixel position and the distance information of the sign or the object in the image taken by the distance image camera, respectively.
The robot system control device according to claim 11. The position information of the sign and the position information of the object are a set of a pixel position in the image of the sign or the object by the camera and an area in the image when the sign is in the position, respectively. is there,
The robot system control device according to claim 11. The posture parameter is a set of joint variables of the robot.
The robot system control device according to any one of claims 11 to 17. The attitude parameter is a three-dimensional position coordinate in the robot coordinate system of the sign or the object.
The robot system control device according to any one of claims 11 to 17. While operating an articulated robot with a fixed sign to move the sign,
The work space of the robot is imaged by one or more cameras, the sign is recognized in the image captured by the cameras, the position information of the sign is calculated, and the position information of the sign and the robot at the time of the imaging are calculated. The calibration process, which is associated with the posture parameters of the camera and is sequentially registered in the reference table,
The work space where the object exists is imaged by the camera.
The object is recognized in the image captured by the camera, the position information of the object is calculated, and the robot's hand is moved to the object from the position information of the object by referring to the reference table. Acquire the above-mentioned posture parameter for moving, and
A work process of transmitting the acquired posture parameters to the robot, and
Robot system control method having. The one or more cameras are the first camera and the second camera.
The position information of the sign and the position information of the object are a set of the pixel position in the image taken by the first camera and the pixel position in the image taken by the second camera of the sign or the object, respectively.
The robot system control method according to claim 20. The one or more cameras are the first camera and the second camera.
The position information of the sign and the position information of the object are the principle of triangulation from the pixel position in the image of the sign or the object, the image of the first camera, and the image of the second camera, respectively. It is a set with the distance information calculated based on,
The robot system control method according to claim 20. The one or more cameras are the first camera and the second camera.
The position information of the sign and the position information of the object are cubic in the camera coordinate system calculated based on the principle of triangulation from the image taken by the first camera and the image taken by the second camera of the sign or the object, respectively. Original position coordinates,
The robot system control method according to claim 20. The camera captures the work space and a mirror arranged so as to reflect the work space in one image.
The position information of the sign and the position information of the object are a set of a pixel position of a direct image of the sign or the object in an image taken by the camera and a pixel position of a reflected image taken by the mirror, respectively.
The robot system control method according to claim 20. The camera is a distance image camera
The position information of the sign and the position information of the object are a set of the pixel position and the distance information of the sign or the object in the distance image by the distance image camera, respectively.
The robot system control method according to claim 20. The position information of the sign and the position information of the object are a set of a pixel position in the image of the sign or the object by the camera and an area in the image when the sign is in the position, respectively. is there,
The robot system control method according to claim 20. The posture parameter is a set of joint variables of the robot.
The robot system control method according to any one of claims 20 to 26. The attitude parameter is a three-dimensional position coordinate in the robot coordinate system of the sign or the object.
The robot system control method according to any one of claims 20 to 26.

Images