JP2016078195A - Robot system, robot, control device and control method of robot - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot system capable of performing calibration between a robot and an imaging section even when an operator is inexperienced in operation of the robot, and further to provide the robot, a control device, a control method of the robot and the like.SOLUTION: A robot system includes a robot and a control device 100. The control device 100 causes the robot to perform marker forming operation forming a marker with respect to an object body provided on a work bench and identifies a relative position attitude relation of an imaging section 200 with respect to the robot on the basis of a pickup image obtained by imaging the marker by the imaging section 200.SELECTED DRAWING: Figure 1

Description

The present invention relates to a robot system, a robot, a control device, a robot control method, and the like.

Currently, a robot that grips a workpiece using image information acquired by a camera has been developed. For example, by calculating the three-dimensional position of the workpiece from image information acquired using two cameras (stereo cameras), the robot can grip the workpiece.

In order to detect a workpiece by a camera and perform a gripping operation by a robot, the coordinate system of the camera (hereinafter referred to as the camera coordinate system) and the coordinate system of the robot (hereinafter referred to as the robot coordinate system) are calibrated. It is necessary to obtain parameters for converting between coordinate systems.

As a method of calibrating the camera coordinate system and the robot coordinate system, for example, as disclosed in Patent Document 1, another coordinate system (coordinate system of the calibration jig) is provided between the robot coordinate system and the camera coordinate system. There is a method of calibrating by interposing. This calibration enables conversion from the camera coordinate system to the calibration jig coordinate system and from the calibration jig coordinate system to the robot coordinate system.

JP-A-8-210816

However, in the method of interposing another coordinate system (the coordinate system of the calibration jig) between the robot coordinate system and the camera coordinate system as disclosed in Patent Document 1 and the like described above,
There are the following problems. First, it is necessary to touch up a calibration jig by a robot, and at that time, calibration work by a human hand is essential. Second, since the touch-up of the calibration jig by the robot is performed using a jog tool or the like, the operator needs to be used to operating the robot with the jog tool. Therefore, when the operator is not familiar with the operation of the robot, it is difficult to perform calibration between the robot and the imaging unit.

One aspect of the present invention includes a robot and a control device, and the control device causes the robot to perform a marker forming operation for forming a marker on an object provided on a work table.
The present invention relates to a robot system that specifies a relative position and orientation relationship of the imaging unit with respect to the robot based on a captured image obtained by imaging the marker by an imaging unit.

In one embodiment of the present invention, a relative position of an imaging unit with respect to a robot is determined based on a captured image obtained by causing a robot to perform a marker forming operation on an object provided on a workbench and imaging the marker by an imaging unit. Identify posture relationships. Therefore, even when the operator is not familiar with the operation of the robot, the robot and the imaging unit can be calibrated.

In the aspect of the invention, the marker may be a marker formed by a gripped object gripped by an end effector of the robot.

As a result, the robot can freely form a marker on the object on the work table.

  In one embodiment of the present invention, the grasped object may be a stamp.

  Thereby, it becomes possible for the robot to form markers by a simple procedure.

In one aspect of the present invention, the control device specifies the position of the marker in a camera coordinate system based on the captured image, and the position of the marker in the robot coordinate system specified when the marker is formed; The relative position and orientation relationship of the imaging unit with respect to the robot may be specified based on the position of the marker in the camera coordinate system.

This makes it possible to obtain the marker position in the robot coordinate system from the marker position on the captured image.

In the aspect of the invention, the control device may cause the robot to re-execute the marker forming operation after the relative position and orientation relationship is specified.

This makes it possible to verify whether or not the robot can perform the same marker forming operation as that during calibration.

In one aspect of the present invention, the control device, after the relative position and orientation relationship is specified,
When the accuracy of work by the robot is reduced, the robot performs the marker forming operation again, and the accuracy of the work is reduced based on a captured image obtained from the imaging unit after the re-execution of the marker forming operation. Factors may be specified.

As a result, the cause of the decrease in work accuracy is, for example, that the mechanical calibration information of the robot itself has changed, the relative position and orientation relationship between the camera and the robot has changed, or the relative position and orientation relationship between the cameras has changed. It is possible to determine whether the change has occurred.

Moreover, in 1 aspect of this invention, you may have the 1st imaging part which is the said imaging part, and the 2nd imaging part different from the said 1st imaging part.

As a result, it is possible to determine the reprojection error for the first imaging unit and the second imaging unit, and to identify the cause of the decrease in work accuracy by the robot.

In one aspect of the present invention, the control device causes the robot to re-execute the marker forming operation after the relative position / posture relationship is specified, and after the re-execution of the marker forming operation, the first controller The reprojection error may be obtained based on the first captured image acquired from the imaging unit and the second captured image acquired from the second imaging unit.

Accordingly, it is possible to determine whether or not the relative position and orientation of the first imaging unit and the second imaging unit are changed based on the reprojection error.

In the aspect of the invention, the control device may determine whether a relative position and orientation of the first imaging unit and the second imaging unit are changed based on the reprojection error. .

Thereby, when the reprojection error is larger than a given threshold, it is possible to determine that the relative position and orientation of the first imaging unit and the second imaging unit have changed since the calibration.

In the aspect of the invention, the control device may perform the first imaging from a position of the marker in the first captured image, a camera internal parameter of the first imaging unit, and a position on the first captured image. The position of the marker in the first camera coordinate system based on the position conversion parameter to the position in the first camera coordinate system of the unit, the position of the marker in the first camera coordinate system, and the first Based on the inter-camera position conversion parameter from one camera coordinate system to the second camera coordinate system of the second imaging unit, the estimated position of the marker on the second captured image is estimated, and based on the second captured image The actual position of the marker on the second captured image is specified, and the reprojection error that is the difference between the actual position of the marker on the second captured image and the estimated position is determined. The may be obtained.

Thereby, based on the difference between the position of the marker estimated on the second captured image and the position of the marker actually reflected on the second captured image, the relative position and orientation between the first imaging unit and the second imaging unit However, it is possible to determine whether or not the value has changed since the calibration.

In one aspect of the present invention, the control device may cause the robot to form a plurality of markers capable of extracting an image feature amount from the object on the work table.

As a result, it is possible to improve the accuracy of calibration and the identification accuracy of the cause of the decrease in work accuracy.

According to another aspect of the present invention, a marker forming operation is performed to form a marker on an object provided on a workbench, and the robot is operated based on a captured image obtained by imaging the marker by an imaging unit. The present invention relates to a robot that specifies the relative position and orientation relationship of the imaging unit.

In another aspect of the present invention, an operation of forming a marker on an object provided on a workbench is performed.
A control unit that causes the robot to perform imaging, and an image reception unit that receives a captured image obtained by imaging the marker from the imaging unit, and the control unit captures the image of the robot based on the captured image. The present invention relates to a control device that identifies the relative position and orientation relationship of parts.

In another aspect of the present invention, an operation of forming a marker on an object provided on a workbench is performed.
Based on a captured image obtained by causing the robot to perform imaging of the marker by the imaging unit,
The present invention relates to a robot control method for specifying a relative position and orientation relationship of the imaging unit with respect to the robot.

According to some aspects of the present invention, a robot system, a robot, a control device, and a robot control capable of calibrating a robot and an imaging unit even when an operator is not familiar with the operation of the robot A method or the like can be provided.

The system configuration example of this embodiment. The other system configuration example of this embodiment. FIG. 3A to FIG. 3D are explanatory diagrams of marker formation processing. The flowchart explaining the flow of a calibration process. FIG. 5A to FIG. 5C are explanatory diagrams of calibration processing. The flowchart explaining the flow of the cause specific process of a precision fall. FIGS. 7A and 7B are explanatory diagrams of captured images before and after calibration. 8A and 8B are explanatory diagrams of reprojection errors. 9A and 9B are configuration examples of the robot. 1 is a configuration example of a robot control system that controls a robot via a network.

Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Outline As described above, for example, when the calibration of the robot and the imaging unit is performed by the method of Patent Document 1, it is necessary to touch up the calibration jig by the robot. Touch-up refers to touching any point at the robot endpoint,
In other words, the robot end point is moved over an arbitrary point.

In that case, calibration work by human hands is essential. Furthermore, since the touch-up of the calibration jig by the robot is performed using a jog tool or the like, the operator needs to be used to operating the robot by the jog tool. Therefore, when the worker is not familiar with the operation of the robot, it is difficult to perform calibration between the robot and the imaging unit.

Therefore, in the robot system and the like of the present embodiment described below, it is possible to perform calibration between the robot and the imaging unit even when the operator is not familiar with the operation of the robot.

Specifically, in the present embodiment, as will be described later with reference to FIGS. 3A to 3D, for example, the robot holds a stamp or the like with an end effector. Then, at the time of calibration, the robot presses the gripped stamp against the object provided on the work table to perform an operation of forming a marker. Conventionally, a marker is provided in advance on an object on a workbench, and the position of the marker in the robot coordinate system is detected by the robot touching up the marker (or calibration jig) as described above. Had been calibrated. On the other hand, in this embodiment, since the robot itself performs the marker forming operation, the marker position in the robot coordinate system can be specified simultaneously with the marker formation.

Therefore, it is not necessary for the robot to touch up a calibration jig or a marker provided in advance during calibration. This eliminates the need for the operator to operate the robot using the jog tool. Therefore, even when the operator is not familiar with the operation of the robot, the robot and the imaging unit can be calibrated.

In addition, after the calibration of the robot and the imaging unit, the robot performs various operations based on the captured image obtained from the imaging unit. However, the operation accuracy may decrease as the operation is repeated. For example, immediately after calibration, the workpiece can be accurately placed at a predetermined position but cannot be placed.

In this case, it is necessary to identify the cause of the decrease in work accuracy and take measures to eliminate the cause. For example, the cause of the decrease in work accuracy is firstly that the mechanical calibration information of the robot itself has changed, and secondly that the positional relationship between the camera and the robot has changed. Furthermore, in the case of using a stereo camera, in addition to the above two causes, the positional relationship between the two cameras may be changed.

Conventionally, it has been difficult to specify the cause of the decrease in work accuracy, but in this embodiment, the cause of the decrease in work accuracy can be specified by a method described later.

2. System Configuration Example Next, a configuration example of the robot system, the robot and the control device 100 according to the present embodiment will be described with reference to FIG.
Shown in

The robot system according to the present embodiment includes a robot and a control device 100. The control device 100 includes an image receiving unit 110 and a control unit 120. The robot includes a drive mechanism 300, and the drive mechanism 300 includes an end effector 310 and an arm 320. Further, the robot system may include the imaging unit 200. However, the robot system, the robot, and the control device 100 are not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of these components or adding other components are possible. It is. Further, the robot may be a separate body from the control device 100 as will be described later with reference to FIG. 9A, or may be integrated with the control device 100 as will be described later with reference to FIG. 9B. There may be.

  Next, the operation of each unit will be described.

The control unit 120 causes the robot to perform a marker forming operation for forming a marker on an object provided on the work table. The function of the control unit 120 includes various processors (CPU, etc.), AS
It can be realized by hardware such as an IC (gate array or the like), a program, or the like.

The image receiving unit 110 receives a captured image obtained by imaging the marker from the imaging unit 200. The image receiving unit 110 can be realized by an interface unit that receives a captured image from the imaging unit 200, or a communication unit that receives a captured image from the imaging unit 200 via a network including at least one of wired and wireless. .

Then, the control unit 120 specifies the relative position and orientation relationship of the imaging unit 200 with respect to the robot based on the captured image.

Although the details of the process will be described later, in this embodiment, as described above, the robot does not need to touch up the calibration jig or a marker provided in advance at the time of calibration, and the operator uses a jog tool. There is no need to operate the robot. Therefore, even when the operator is not familiar with the operation of the robot, the robot and the imaging unit can be calibrated.

The marker is a marker formed by a gripped object gripped by the end effector 310 of the robot, for example.

Therefore, it becomes possible for the robot to control the position and orientation of the end effector 310 to press the gripped object against the object on the work table. As a result, the robot can freely form a marker on the object on the work table.

Here, the marker is a tangible object that can be used as a mark, a character, a figure, a symbol, a pattern, a three-dimensional shape, a combination thereof, or a combination of these and a color, and can be fixed to an object. .

The grasped object is, for example, a stamp. When the gripping object is a stamp, the marker is an imprint transferred to the object on the work table by the stamp. When the grasped object is a stamp, the robot can form a marker by a simple procedure. However, this embodiment is not limited thereto, and the marker may be, for example, a seal, a sticker, a label, or the like. It doesn't matter what the shape, color, pattern, etc. of the marker is, but it can be easily distinguished from other areas for stable detection of the marker, for example, black for white areas An image or a sticker is desirable. Furthermore, the marker is QR (Quick Respon
se) code or AR (Augmented Reality) marker.

Further, the control device 100 may cause the robot to form a plurality of markers capable of extracting the image feature amount on the object on the work table.

As a result, it is possible to improve the accuracy of calibration and the identification accuracy of the cause of the decrease in work accuracy.

Further, as shown in FIG. 2, the robot system includes a first imaging unit 200 and a first imaging unit 20.
A second imaging unit 250 different from 0 may be included. That is, the imaging unit 200 described above with reference to FIG. 1 may be a stereo camera.

As a result, it is possible to determine the reprojection error for the first imaging unit 200 and the second imaging unit 250 and to identify the cause of the decrease in work accuracy by the robot. Details of this processing will be described later.

3. Details of Calibration Processing The calibration processing according to the present embodiment is processing for specifying the relative position and orientation relationship between the camera coordinate system of the imaging unit and the robot coordinate system of the robot. In other words, this is processing for accurately specifying a conversion parameter for converting an arbitrary position in the camera coordinate system into a position in the robot coordinate system. Finally, a conversion parameter from the position on the captured image to the position in the robot coordinate system is obtained. Therefore, when the calibration process is normally performed, the position of an arbitrary point in the captured image can be accurately converted to a position in the robot coordinate system.

At this time, in the present embodiment, the control device 100 specifies the position of the marker in the robot coordinate system when the marker is formed, and specifies the position of the marker in the camera coordinate system based on the captured image. Then, the control device 100 specifies the relative position and orientation relationship of the imaging unit 200 with respect to the robot based on the marker position in the robot coordinate system and the marker position in the camera coordinate system.

Here, the relative position and orientation relationship between the camera coordinate system and the robot coordinate system specified in this calibration process specifically refers to a transformation matrix from the camera coordinate system to the robot coordinate system.

Next, the detailed flow of the calibration process of the present embodiment will be described with reference to FIGS.
This will be described with reference to (D), the flowchart of FIG. 4 and FIGS. 5 (A) to 5 (C). FIG.
In the example of FIG. 3D, the robot RB is used by using the first imaging unit CM1 and the second imaging unit CM2.
It is assumed that an image of the work situation of In addition, after calibration, for example, the robot R
It is assumed that a work (not shown) is gripped by B and the gripped work is moved onto the work table WA.

First, as shown in FIG. 3A, paper PP is placed on a work bench WA on which a work is to be placed later. Note that the object placed on the work table WA may be an object other than the paper PP. In addition, distance information to the work table WA is stored in the robot RB in advance. However, if it is possible to detect that the hand of the robot RB touches the work table, for example, the robot hand has a force sensor, the distance information to the work table must not be stored. Also good.

Next, as shown in FIG. 3B, a stamp is provided at or near the hand position of the robot RB and fixed.

Then, as shown in FIG. 3C, the robot RB is pressed with three or more stamps on the paper PP to form three markers MK (S101 in FIG. 4). The position where the stamp is pressed may be any position on the paper PP.

Then, the control device 100 calculates a position (marker position) where the stamp is pressed in the robot coordinate system, and stores it in a storage unit (not shown) (S102). The marker position in the robot coordinate system can be calculated from the joint angle of the arm, the link length, and the stamp length.

Next, after the pressing of the stamp is finished, as shown in FIG. 3D, the first imaging unit CM1 and the second imaging unit CM2 capture three markers MK, and the control device 100 generates two captured images. Obtain (S103). Hereinafter, the captured image captured by the first imaging unit CM1 is the first captured image, the captured image captured by the second imaging unit CM2 is the second captured image, and the first imaging unit C.
The camera coordinate system for M1 is referred to as a first camera coordinate system, and the camera coordinate system for the second imaging unit CM2 is referred to as a second camera coordinate system.

Then, as illustrated in FIG. 5A, the control device 100 determines the marker position on the first captured image IM1 based on the camera internal parameters of the first imaging unit CM1 in the first camera coordinate system C.
The position is converted to the position in C1 (S104). Here, for example, the position of the point B shown in the first captured image IM1 is converted to a position C12B in the first camera coordinate system CC1. The camera internal parameters are parameters such as a focal length and an image center point (in other words, an image sensor center point). Then, the first captured image IM1 shown in FIG.
A conversion parameter D1 from the upper position to the position C12B in the first camera coordinate system CC1 is obtained based on camera internal parameters.

Similarly, the position conversion process is performed for the second imaging unit CM2 (S105). Thereby, the marker position in the first camera coordinate system and the second camera coordinate system can be specified.

Next, as illustrated in FIG. 5B, the control device 100 determines the first imaging unit CM1 based on the marker position R2B of the robot coordinate system RC and the marker position C12B of the first camera coordinate system CC1. Calibration of external parameters is performed (S106). Here, the external parameter is a position conversion parameter used for converting (or inversely converting) the position of the origin of the robot coordinate system RC into the position of the origin of the first camera coordinate system CC1. The external parameter is represented by, for example, a rotation matrix and a parallel progression. That is, in step S106,
The external parameter for converting the marker position R2B of the robot coordinate system RC and the marker position C12B of the first camera coordinate system CC1 as indicated by D2 is calibrated to an accurate value.

Similarly, the control device 100 calibrates external parameters of the second imaging unit CM2 based on the position of the marker in the robot coordinate system and the position of the marker in the second camera coordinate system (S107).

After that, calibration of stereo parameters is performed based on the external parameters for the first imaging unit CM1 and the external parameters for the second imaging unit CM2 (S108), and the process ends. Here, the stereo parameter is a parameter for obtaining a three-dimensional position on the principle of triangulation using two cameras. The stereo parameters include camera internal parameters of the first imaging unit CM1, camera internal parameters of the second imaging unit CM2, and position conversion parameters between the cameras. If a stereo parameter that is accurately calibrated is used, a specific point shown in two captured images can be converted into a position in the robot coordinate system RB. For example, as shown in FIG. 5C, the position R2B in the robot coordinate system RC can be obtained from the position of the point B appearing in the first captured image IM1 and the second captured image IM2, as indicated by D3.

If the stereo parameter is used, as indicated by the arrow D4 in FIG. 5C, where the position of the point (B) reflected in one captured image (IM1) appears in the other captured image (IM2). Can be calculated.

This makes it possible to obtain the marker position in the robot coordinate system from the marker position on the captured image. Further, it is possible to automate the calibration of the camera and the robot without using a special calibration jig or the like.

However, the calibration processing of the present embodiment is not limited to the flow shown in the flowchart of FIG. 4, and various modifications such as changing part of the processing contents or changing the processing order are possible. . For example, in step S108 described above, the stereo parameter is obtained from the two external parameters, but the stereo parameter is obtained based on the marker position on the captured image and each marker position in the robot coordinate system, not on the external parameter. It is also possible. Furthermore, stereo parameters can also be obtained based on the marker position on the captured image and each marker position in the marker coordinate system.

4). Details of the process of identifying the cause of work accuracy degradation As described above, after calibration of the robot and the imaging unit, the robot performs various tasks based on the captured image obtained from the imaging unit. In some cases, work accuracy may be reduced. In the present embodiment, the cause of the decrease in work accuracy after calibration is specified by performing the processing shown in the flowchart of FIG.

First, paper is placed on the work table in the same way as in calibration, the robot is given a stamp, and the stamp is pressed in the same posture as in calibration (S201).

Next, the control device 100 determines whether or not the distance between the pressed stamp is the same as that at the time of calibration (S202).

For example, FIGS. 7A and 7B show an example. FIG. 7A shows the (first) captured image BIM at the time of calibration, and FIG. 7B shows the (first) captured image AIM acquired in step S201. The captured image BIM includes a marker A ₁ and a marker B
_It can be seen that ₁ is formed at intervals of Δ1. In contrast, the captured image AIM, the marker _{A 2} corresponding to the marker _{A 1,} and the marker _{B 2} corresponding to the marker _{B 1,}
It can be seen that they are formed at an interval of Δ2 that is larger than Δ1. That is, FIG.
In the example of FIG. 7B, it can be determined that the distance between the pressed stamp is different from that at the time of calibration. In this determination, a predetermined error range is provided to determine whether the distance between the stamps is within the predetermined error range. If the distance is within the predetermined error range, it is determined that the calibration is the same as at the time of calibration. If it is out of the predetermined error range, it may be determined that it is different from that at the time of calibration.

When the control device 100 determines that the distance between the pressed stamps is significantly different from that at the time of calibration, it determines that the mechanical calibration information has changed (S203).

On the other hand, if the control device 100 determines that the distance between the pressed stamp is the same as that at the time of calibration, the position of the pressed stamp on the captured image is calculated and reprojected in step S201. Find the error. The reprojection error will be described in detail later.

Then, the control device 100 determines whether or not the calculated reprojection error is within a predetermined error range. If the calculated reprojection error is outside the predetermined error range, it is determined that the positional relationship between the two cameras has changed. (S
205).

On the other hand, when determining that the calculated reprojection error is within a predetermined error range, the control device 100 determines that the positional relationship between the camera and the robot has changed (S206).

By performing the above processing, the cause of the decrease in work accuracy is that the mechanical calibration information of the robot itself has changed, the positional relationship between the camera and the robot has changed, or the two cameras have It is possible to determine whether the positional relationship has changed. As a result, it is possible to easily specify a parameter that needs to be recalibrated when the accuracy is lowered in an operation using the vision of the robot.

Here, the cause identifying process for the reduction in work accuracy will be summarized. First, work accuracy refers to the high reproducibility of work corresponding to the same work performed by a robot or work performed earlier. For example, when an operation of repeatedly touching up an arbitrary point on the workbench is performed, it can be determined that the reproducibility of the work accuracy is high when the same point can be touched up. On the other hand, when the deviation between the point where the robot touches up and the original point becomes large, the reproducibility of the work accuracy decreases according to the deviation.

Further, the work performed in the present embodiment may not be the same work as long as it is a work corresponding to the work performed previously. For example, when a robot performs a work of picking up a work, the position where each work is placed is different, but the work content itself is the same. That is, it can be said that this work corresponds to the work performed previously. In this case, if the workpiece can be picked up normally, the work accuracy is high. If the workpiece cannot be picked up normally, it can be determined that the work accuracy is low.

In the present embodiment, the control device 100 causes the robot to re-execute the marker forming operation after the relative position and orientation relationship of the imaging unit with respect to the robot is specified.

This makes it possible to verify whether or not the robot can perform the same marker forming operation as that during calibration.

In addition, the control device 100 causes the robot to re-execute the marker forming operation when the accuracy of the operation by the robot decreases after the relative position and orientation relationship is specified, and obtains from the imaging unit 200 after the re-execution of the marker forming operation. Based on the obtained captured image, a factor that reduces the accuracy of work may be specified.

As a result, as described above, the cause of the decrease in work accuracy is, for example, that the mechanical calibration information of the robot itself has changed, the positional relationship between the camera and the robot has changed, or the two cameras It becomes possible to determine whether or not the positional relationship has changed.

In addition, the control device 100 causes the robot to re-execute the marker formation operation after the relative position and orientation relationship is specified, and the first captured image acquired from the first imaging unit 200 after the re-execution of the marker formation operation The reprojection error may be obtained based on the second captured image acquired from the second imaging unit 250.

Accordingly, the control device 100 can determine whether or not the relative position and orientation of the first imaging unit 200 and the second imaging unit 250 are changed based on the reprojection error. For example, when the reprojection error is larger than a given threshold, it is possible to determine that the relative positions and orientations of the first imaging unit and the second imaging unit have changed since the calibration. And
When it is determined that the relative positions and orientations of the first imaging unit 200 and the second imaging unit 250 have changed, the positions and orientations of the first imaging unit 200 and the second imaging unit 250 are set to the same position and orientation as at the time of calibration. It is possible to adjust so as to be.

Furthermore, based on the reprojection error, it can also be determined whether or not the relative position and orientation of the robot and the imaging unit have changed from those at the time of calibration.

Here, the reprojection error is a marker on the second captured image calculated based on the image position of the marker acquired by the first image capturing unit 200, the camera internal parameter of the first image capturing unit 200, and the inter-camera position conversion parameter. And the image position of the marker actually acquired by the second imaging unit. In short, the reprojection error is a difference between the estimated position of the marker on the second captured image and the position on the actual image. In the description of the reprojection error, the first imaging unit 200 and the second imaging unit 250 may be interchanged.

That is, the control device 100 determines the position of the marker in the first captured image and the first imaging unit 2.
The position of the marker in the first camera coordinate system is specified based on the camera internal parameter of 00 and the position conversion parameter from the position on the first captured image to the position in the first camera coordinate system of the first imaging unit 200 To do. Then, the control device 100 determines the second captured image based on the marker position in the first camera coordinate system and the inter-camera position conversion parameter from the first camera coordinate system to the second camera coordinate system of the second imaging unit 250. Estimate the estimated position of the upper marker. Finally, the control device 100 identifies the actual position of the marker on the second captured image based on the second captured image, and calculates the difference between the actual position of the marker on the second captured image and the estimated position. Find some reprojection error.

For example, FIGS. 8A and 8B show an example. FIG. 8A shows the first captured image I.
FIG. 8B illustrates M1 as a second captured image IM2 actually captured by the second imaging unit 250 and acquired. The first image IM1 and the second image IM2, is reflected how the markers _{A 1} and marker _{B 1} is formed. However, the first captured image IM1 and the second captured image
In the captured image IM2, for the position and orientation of the two imaging unit is different, the position is reflected marker A ₁ and marker B ₁ is are offset from each other. Furthermore, in the second captured image IM2 in FIG. 8B, the marker A ₂ estimated based on the positions of the markers A ₁ and B ₁ in the first captured image IM1, the camera internal parameters, and the inter-camera position conversion parameters. and it is shown superimposed marker B _2. In FIG. 8B, the positions of the marker A ₁ and the marker A _{2 and} the positions of the marker B ₁ and the marker B ₂ indicate the same marker position, respectively, but the positions of the marker A ₂ and the marker B ₂ are It is estimated based on various parameters and may be different from the actual marker position. Originally, it is only necessary that the estimated marker position and the actually reflected marker position match on the second captured image. However, in the example of FIG. 8B, there is a reprojection error because they do not match. become. In this case, the first imaging unit 2
It can be determined that the relative position and orientation between 00 and the second imaging unit 250 have changed since calibration.

Thus, based on the difference (reprojection error) between the position of the marker estimated on the second captured image and the position of the marker actually reflected on the second captured image, the first imaging unit 200 and the second Imaging unit 2
It is possible to determine whether or not the relative position and orientation with respect to 50 has changed since calibration. Even if the image position of the marker shown in step S201 is different from that at the time of calibration, the same value can be obtained for the reprojection error if the distance between the two cameras is the same.

5. Next, FIGS. 9A and 9B show a configuration example of the robot according to the present embodiment. FIG.
) And FIG. 9B, the robot mechanism (driving mechanism) 300 includes an end effector (hand) 310 and an arm 320. Note that the robot shown in FIG. 9A is a single-arm type, but the robot may be a multi-arm type robot such as a double-arm type as shown in FIG. 9B.

Here, the end effector 310 is a part attached to the end point of the arm in order to grip, lift, lift, adsorb, or process the work (work object). Say. The end effector 310 may be, for example, a hand (gripping unit), a hook, a suction cup, a driver, or the like. Further, a plurality of end effectors may be provided for one arm.

The arm 320 is a part of the robot mechanism 300 and refers to a movable part including one or more joints.

In the present embodiment, the first imaging unit 200 and the second imaging unit 250 are provided outside the robot.

In the robot of this embodiment, as shown in FIG. 9A, the robot mechanism 300 and the control device 100 are configured separately. In this case, some or all of the functions of the control device 100 are realized by, for example, a PC (Personal Computer).

Further, the robot of the present embodiment is not limited to the configuration shown in FIG. 9A, and the robot mechanism 300 and the control device 100 may be integrated as shown in FIG. 9B. Specifically, FIG.
As shown in B), the robot may include a robot mechanism 300 and a base unit portion 350 that supports the robot mechanism 300, and the control device 100 may be stored in the base unit portion 350. The base unit 350 of the robot shown in FIG. 9B is provided with wheels 370 and the like so that the entire robot can move. The robot may be moved manually, or may be moved by providing a motor for driving the wheel 370 and controlling the motor by the control device 100. Moreover, the control apparatus 100 is not necessarily provided in the base unit part 350 provided under the robot mechanism 300 as shown in FIG.

Further, as shown in FIG. 10, the function of the control device 100 is that the server 50 is connected to the robot mechanism 300 via a network 400 including at least one of wired and wireless.
It may be realized by 0.

Alternatively, in the present embodiment, a part of the processing of the control device of the present invention may be performed by the control device of the server 500. In this case, the processing is realized by distributed processing with a control device provided in the robot mechanism 300. Note that the control device of the robot mechanism 300 is realized by, for example, a terminal device 330 (control unit) installed in the robot mechanism 300.

In this case, the control device of the server 500 performs a process assigned to the control device of the server 500 among the processes in the control device of the present invention. On the other hand, the robot mechanism 3
The control device provided in 00 performs processing assigned to the control device of the robot mechanism 300 among the processing of the control device of the present invention. Each process of the control device of the present invention is performed by the server 5.
The process assigned to 00 or the process assigned to the robot mechanism 300 may be used.

Thereby, for example, the server 500 having a higher processing capability than the terminal device 330 can perform processing with a large processing amount. Further, for example, the server 500 can control the operation of each robot in a lump, and it becomes easy to cause a plurality of robots to perform a cooperative operation.

In recent years, the production of a small number of various types of parts has been increasing. And when changing the kind of components to manufacture, it is necessary to change the operation | movement which a robot performs. With the configuration as shown in FIG. 10, it is possible for the server 500 to collectively change the operation performed by each robot without re-instructing each robot of the plurality of robots.

Further, in the configuration as shown in FIG. 10, one control device 100 is provided for each robot.
Compared with the case where the software is provided, it is possible to significantly reduce the time and effort required to update the software of the control device 100.

Note that the robot system, the robot, the control device, and the like of the present embodiment may realize part or most of the processing by a program. In this case, the robot system, the robot, the control device, and the like of the present embodiment are realized by a processor such as a CPU executing a program. Specifically, a program stored in a non-temporary information storage device is read, and a processor such as a CPU executes the read program. Here, an information storage device (device readable by a computer) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), or memory (card type). It can be realized by memory, ROM, etc. A processor such as a CPU performs various processes according to the present embodiment based on a program (data) stored in the information storage device. That is, in the information storage device, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing a computer to execute the processing of each unit). Is memorized.

In addition, the robot system, the robot, the control device, and the like according to the present embodiment may include a processor and a memory. The processor here is, for example, a CPU (Central Processing U).
nit). However, the processor is not limited to the CPU,
Various processors such as U (Graphics Processing Unit) or DSP (Digital Signal Processor) can be used. The processor is ASIC (Application
A hardware circuit using a specific integrated circuit) may be used. The memory stores instructions that can be read by a computer. When the instructions are executed by the processor, each unit of the robot system, the robot, the control device, and the like according to the present embodiment is realized. . The memory here is SRAM (Static Random Access).
It may be a semiconductor memory such as a memory (RAM) or a DRAM (dynamic random access memory), or may be a register or a hard disk. In addition, the instruction here may be an instruction of an instruction set constituting the program, or an instruction for instructing an operation to the hardware circuit of the processor.

Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configuration and operation of the robot system, the robot, the control device, and the robot control method are not limited to those described in the present embodiment, and various modifications can be made.

100 control device, 110 image receiving unit, 120 control unit, 200 first imaging unit,
250 second imaging unit, 300 drive mechanism (robot mechanism), 310 end effector,
320 arm, 330 terminal device, 350 base unit, 370 wheels,
400 networks, 500 servers

Claims (14)

With robots,
A control device;
Including
The controller is
Causing the robot to perform a marker forming operation for forming a marker on an object provided on a workbench;
A robot system, wherein a relative position and orientation relation of the imaging unit with respect to the robot is specified based on a captured image obtained by imaging the marker by an imaging unit. In claim 1,
The marker is
A robot system comprising a marker formed by a gripped object gripped by an end effector of the robot. In claim 2,
The gripping object is
A robot system characterized by being a stamp. In any one of Claims 1 thru | or 3,
The controller is
Based on the captured image, identify the position of the marker in the camera coordinate system,
The relative position and orientation relationship of the imaging unit with respect to the robot is specified based on the position of the marker in the robot coordinate system specified at the time of forming the marker and the position of the marker in the camera coordinate system. Robot system to do. In any one of Claims 1 thru | or 4,
The controller is
A robot system that causes the robot to re-execute the marker forming operation after the relative position / posture relationship is specified. In claim 5,
The controller is
After the relative position and orientation relationship is specified, when the accuracy of work by the robot is lowered, the robot is caused to re-execute the marker forming operation, and obtained from the imaging unit after the re-execution of the marker forming operation A robot system, characterized in that a factor that reduces the accuracy of the work is specified based on a captured image. In any one of Claims 1 thru | or 6.
A first imaging unit which is the imaging unit;
A second imaging unit different from the first imaging unit;
A robot system characterized by comprising: In claim 7,
The controller is
After the relative position and orientation relationship is specified, the robot is caused to re-execute the marker forming operation, and the first acquired from the first imaging unit after the re-execution of the marker forming operation.
A robot system characterized in that a reprojection error is obtained based on a captured image and a second captured image acquired from the second imaging unit. In claim 8,
The controller is
A robot system that determines whether or not relative positions and orientations of the first imaging unit and the second imaging unit have changed based on the reprojection error. In claim 8 or 9,
The controller is
The position of the marker in the first captured image, the camera internal parameter of the first image capturing unit, and the position conversion parameter from the position on the first captured image to the position in the first camera coordinate system of the first image capturing unit And specifying the position of the marker in the first camera coordinate system based on
Based on the position of the marker in the first camera coordinate system and the inter-camera position conversion parameter from the first camera coordinate system to the second camera coordinate system of the second imaging unit, the second
Estimating the estimated position of the marker on the captured image,
Based on the second captured image, the actual position of the marker on the second captured image is specified,
The robot system according to claim 1, wherein the reprojection error, which is a difference between an actual position of the marker on the second captured image and the estimated position, is obtained. In any one of Claims 1 thru | or 10.
The controller is
A robot system, wherein the robot is formed with a plurality of markers capable of extracting image feature amounts on the object on the workbench. A marker forming operation is performed to form a marker on an object provided on a workbench, and the relative position and orientation relationship of the imaging unit with respect to the robot is specified based on a captured image obtained by imaging the marker by the imaging unit. Robot characterized by doing. A control unit that causes the robot to perform an operation of forming a marker on an object provided on the workbench;
An image receiving unit that receives a captured image obtained by imaging the marker from the imaging unit;
Including
The controller is
A control apparatus that identifies a relative position and orientation relationship of the imaging unit with respect to the robot based on the captured image. Let the robot perform an action to form a marker on the object provided on the workbench,
A robot control method comprising: specifying a relative position and orientation relationship of the imaging unit with respect to the robot based on a captured image obtained by imaging the marker by an imaging unit.