JP7323993B2 - Control device, robot system, operating method and program for control device - Google Patents

Description

The present invention relates to a control device, a robot system, an operation method and program for the control device, and more particularly to visual servo technology for controlling a robot based on images.

Conventionally, visual servoing is known as one method of controlling a robot based on camera images. Visual servoing is a technology that feedback-controls a robot so that a target object appears at a desired position. Using visual servos eliminates the need to accurately determine the position and orientation relationships between the robot and the camera and between the robot and the end effector, reducing the burden of calibration work to determine these positional relationships. can. Among them, the method of extracting the characteristics of the object to be guided to the target position and the object at the target position from the image and controlling the robot based on the difference in those positions is a method that can be used to control the position of the target to be guided and the target position. There are advantages that can be accommodated.

In Japanese Patent Laid-Open No. 2002-100000, feature amounts are detected from images of an object to be assembled and an object to be assembled, and the assembly work is performed based on the detection results. It is disclosed that the assembling work can be realized even when there is a

JP 2015-85450 A

In order to realize visual servoing using image features, the target to be guided to the target position and the target at the target position are imaged in the same image, and the position and orientation can be specified by extracting multiple features for each. It is necessary to However, depending on the shape of the object, only a few features can be extracted. Also, depending on the arrangement of the camera, only a part of the object can be seen, and the features may not be sufficiently extracted.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems, and aims to provide a technique for realizing visual servoing and enabling tasks to be executed with high accuracy even when sufficient features cannot be extracted from an image. aim.

A control device according to the present invention that achieves the above objects includes:
An imaging device attached to a robot acquires an image of a controlled object whose rotation can be controlled by the robot and a target position, which is a destination of movement of the controlled object, and is a target position in a concave portion. an image acquisition means;
A progression for acquiring a two-dimensional direction in the image representing the traveling direction of the rotational axis of the controlled object and along the rotational axis of the controlled object, based on the edges of the controlled object in the image. a direction acquisition means;
target position detection means for detecting a two-dimensional position in the image representing the target position from the image ;
The target position exists on the extension of the traveling direction of the rotating shaft, and the controlled object performs an operation of changing the attitude of the controlled object so that the traveling direction of the rotating shaft and the depth direction of the recess match. a motion generation means for generating a motion of moving the controlled object closer to the target position along the traveling direction of the controlled object whose attitude has been changed, which is generated before reaching the target position;
a control means for controlling the controlled object according to the operation;
with
The motion generating means obtains a control amount of the posture of the control object that brings the line segment along the two-dimensional direction closer to the two-dimensional position, and updates the posture of the control object based on the control amount. characterized by

According to the present invention, it is possible to implement visual servoing and execute tasks with high accuracy even when sufficient features cannot be extracted from an image.

1 is a configuration diagram showing a configuration example of a robot system according to a first embodiment; FIG. 1 is a block diagram showing a configuration example of a robot system according to a first embodiment; FIG. 4 is a flowchart for explaining processing by the robot system of the first embodiment; FIG. 2 is a diagram for explaining a coordinate system of the robot system of the first embodiment; FIG. 4A and 4B are diagrams for explaining a motion generator according to the first embodiment; FIG. 9 is a flowchart for explaining processing by the robot system of Modification 1 of the first embodiment; FIG. 11 is a diagram for explaining a motion generation unit according to Modification 2 of the first embodiment; FIG. 11 is a diagram for explaining a motion generation unit according to Modification 2 of the first embodiment; FIG. 11 is a diagram for explaining an operation display section of Modification 3 of the first embodiment; The block diagram which shows the structural example of the robot system of 2nd Embodiment. 8 is a flowchart for explaining processing by the robot system of the second embodiment; The block diagram which shows the structural example of the robot system of 3rd Embodiment. 10 is a flowchart for explaining processing by the robot system of the third embodiment;

Hereinafter, embodiments will be described with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

(First embodiment)
In the first embodiment, an imaging device attached to a robot captures an image of a controlled object and a target position, and the robot guides the controlled object to the target position. In this embodiment, the controlled object is a driver, which is an end effector attached to the robot. The target position is the position at which the end effector executes the task, and in this embodiment, the screw center, which is the position of one image feature point, is used. Note that the target position may be two or more image features. A robot system that automatically unscrews screws by using a screwdriver, which is an end effector, to remove screws after being guided as a task by the robot will also be described. By generating the motion of the robot from the positional relationship between the driver and the screw in the image, it is possible to accurately control even if there is an error in the calibration between the robot and imaging device or between the imaging device and end effector.

[Device configuration]
The configuration diagram of FIG. 1 and the block diagram of FIG. 2 show a configuration example of a robot system 1000 including the control device 100 of the first embodiment. A robot system 1000 includes an imaging device 1 , a robot arm 2 , an end effector 3 , and a control device 100 .

The imaging device 1 is a device that is attached to the end effector 3 and captures an image of a scene including the end effector 3 and a target position. For example, the imaging apparatus 1 is composed of two grayscale cameras, performs imaging by an imaging trigger from the control unit 106 , and sends an image signal to the image acquisition unit 101 .

The robot arm 2 is a device that moves an object to be controlled. For example, the robot arm 2 is composed of a 6-axis robot and operates by receiving control values from the control device 100 . The end effector 3 is a device attached to the tip of the robot arm that manipulates a target. For example, the end effector 3 is composed of a driver and a proximity sensor, and inputs the contact state between the driver and the target to the control device 100 . Also, the end effector 3 operates upon input of a control value from the control device 100 .

The control device 100 includes an image acquisition unit 101 , an axial direction acquisition unit 102 , a target position detection unit 103 , a motion generation unit 104 , an end determination unit 105 and a control unit 106 .

The image acquisition unit 101 receives image signals from the imaging device 1 and sends image data to the axial direction acquisition unit 102 and the target position detection unit 103 . For example, the image acquisition unit 101 is composed of a memory.

The axial direction acquisition unit 102 receives the image data from the image acquisition unit 101, acquires the axial direction of the control target set by the axial direction setting unit (not shown), and transmits the axial direction data to the motion generation unit 104 and the end determination unit 105. send. Here, the axis is, for example, a rotation axis, and the axial direction is a line segment in an image associated with the controlled object for measuring the position/orientation relationship between the controlled object and the target position. is represented by a two-dimensional point on the axis and a two-dimensional vector extending from the root to the tip. In addition, the vector may conversely be directed from the tip to the root. By setting the axial direction, even in a situation where the imaging device 1 attached to the end effector 3 can only partially image the end effector 3 and the characteristics of the end effector 3 cannot be sufficiently detected, the motion generation unit 104 can calculate the position and orientation that define the motion of the tip of the robot arm 2 .

The target position detection unit 103 receives image data from the image acquisition unit 101 , detects a target position from the image data, and sends the target position data to the action generation unit 104 and the end determination unit 105 . Based on the axial direction data and the target position data received from the axial direction acquisition unit 102 and the target position detection unit 103, the motion generation unit 104 calculates the position and orientation that define the motion of the distal end of the robot arm 2, and calculates the position and orientation. Send the data to the control unit 106 . First, the tip of the robot arm 2 is moved so that the target position exists on the extension of the rotation axis, and then the tip of the robot arm 2 is made to reach the target position along the axial direction.

The end determination unit 105 determines whether the controlled object has reached the target position based on the axial direction data and the target position data received from the axial direction acquisition unit 102 and the target position detection unit 103, and transmits the end determination data to the control unit. Send to 106.

The control unit 106 controls the robot arm 2 based on the position/orientation data received from the motion generation unit 104 . Based on the end determination data received from the end determination unit 105, the control unit 106 sends an imaging trigger to the imaging device 1 when continuing the control loop, and the robot arm based on the task operation when ending the control loop. 2 and end effector 3.

[Control processing]
A control processing method by the control device 100 and the robot system 1000 of the first embodiment will be described with reference to the flow chart of FIG. 3 and the definition of the coordinate system of FIG. In FIG. 4, 41 is the robot coordinate system that is the reference of the robot arm 2, 42 is the flange coordinate system that is the tip of the robot arm 2, 43 is the imaging device coordinate system that is the reference of the imaging device 1, and 44 is the end effector 3. It is an end effector coordinate system that is the tip. Q is the flange coordinate system 42 (flange position/orientation) in the robot coordinate system 41, G is the transformation matrix from the flange coordinate system 42 to the imaging device coordinate system 43, and H is the transformation from the imaging device coordinate system 43 to the end effector coordinate system 44. represents a matrix.

(Step S101)
In step S101, the control unit 106 controls the robot arm 2 to move to the initial position and orientation. In this embodiment, the control unit 106 acquires the position and orientation set by the teaching operation using the teaching pendant as the initial position and orientation. Also, the motion generation unit 104 acquires the transformation matrix G and the transformation matrix H. FIG. Note that the transformation matrix G and the transformation matrix H are obtained in advance by the user through design values and calibration, but may contain errors. After the robot arm 2 moves to the initial position and orientation, the control unit 106 sends an imaging trigger to the imaging device 1 .

(Step S102)
In step S<b>102 , the imaging device 1 receives an imaging trigger from the control unit 106 , performs imaging, and sends an image signal of the imaged image to the image acquiring unit 101 .

(Step S103)
In step S<b>103 , the image acquisition unit 101 receives an image signal from the imaging device 1 and sends image data to the axial direction acquisition unit 102 and the target position detection unit 103 .

(Step S104)
In step S<b>104 , the axial direction acquisition unit 102 receives the image data from the image acquisition unit 101 , acquires the axial direction of the controlled object, and sends the acquired axial direction data to the motion generation unit 104 and the end determination unit 105 .

In this embodiment, two long line segments are detected from straight edges of the driver, which is the end effector 3, captured in the image. Get directions. The axial direction may be detected from an image as in the present embodiment, or data set in advance by an axial direction setting unit (not shown) may be acquired. The axial direction setting unit may set, for example, a line segment connecting the two points specified by the user in the image as the axial direction. Alternatively, the axial direction may be set using a value obtained by projecting the axial direction of the end effector 3 onto an image based on the result of measuring the three-dimensional positional relationship between the imaging device 1 and the end effector 3 .

(Step S105)
In step S<b>105 , the target position detection unit 103 receives the image data from the image acquisition unit 101 , detects the target position based on the image data, and sends the detected target position data to the motion generation unit 104 and the end determination unit 105 .

In this embodiment, the screw center position is detected as the target position. Detection may be performed by template matching using an image of a screw taken in advance as a template, or by extracting a screw region based on the luminance value of the image and calculating its center of gravity.

(Step S106)
In step S106, the motion generation unit 104 receives the axial direction data and the target position data from the axial direction acquisition unit 102 and the target position detection unit 103, and calculates the position and orientation that define the motion of the tip of the robot arm 2. The motion generation unit 104 also sends the calculated position and orientation data to the control unit 106 . First, the tip of the robot arm 2 is moved so that the target position exists on the extension of the rotation axis, and then the tip of the robot arm 2 is made to reach the target position along the axial direction.

An example of motion generation will be described below with reference to FIG. In FIG. 5, 51 is an image captured by the camera, which is the imaging device 1, 52 is the driver, which is the end effector 3, captured in the image, 53 is the screw captured in the image, 54 is the obtained axial direction, and 55 is the detected target. Position 56 represents the distance between the axial direction and the target position. From these pieces of information, a flange control amount that brings the axial direction closer to the target position is calculated. Specifically, first, the controlled variable H' of the controlled object is solved by the following equations 1 to 4.

Here, Equation 1 is a relational expression between the camera as the imaging device 1, the driver as the end effector 3, and the screw. P′ is a screw position in the imaging device coordinate system 43 as a control target, H′ is a control amount of the driver, which is the end effector 3, and P is a screw position in the end effector coordinate system 44 as a control target. Formula 2 defines the control amount H' of the driver, which is the end effector 3, and is the case where the end effector coordinate system 44 is moved in the x-axis and y-axis directions. dx and dy represent the control amount. Formula 3 is a relational expression between the screw position P' and the image, where u' and v' are the projected positions of P' in the image, _fx and _fy are the focal lengths of the camera that is the imaging device 1, cx and cy represents the principal point position of the camera, which is the imaging device 1 . Equation 4 is the desired image Jacobian, where u and v are the detected screw positions, and u _d and v _d are the positions of the vertical feet down the shaft from the screw positions. By obtaining these from each image of the stereo camera, which is the imaging device 1, the control amount H' of the driver, which is the end effector 3, can be obtained. This is reflected in the current flange position/orientation Q by Equation (5).

Here, Q' is the position and orientation of the flange to be obtained, and Q is the position and orientation of the flange when the image is captured. Thereby, the flange position and orientation can be controlled based on the distance between the axial direction in the image and the target position. As a result, since the flange position/orientation calculation does not require an accurate three-dimensional position of the target position, it is possible to reduce the influence of the three-dimensional measurement accuracy due to the position/orientation calibration error between the two cameras, which are the imaging device 1, for example. can be done.

(Step S107)
In step S107, the end determination unit 105 receives axial direction data and target position data from the axial direction acquisition unit 102 and the target position detection unit 103, and determines whether to end or continue the control loop. In addition, end determination unit 105 sends end determination data to control unit 106 . Here, the control loop is a series of loop processes from S102 to S109 for controlling the robot arm 2 based on the imaging results. Termination may be determined based on the distance between the detected axis and the target position, the difference between the previous distance and the current distance, or the number of loops. For example, if the difference between the previous distance and the current distance is less than a certain value, it can be judged as "end", otherwise it can be judged as "continue". In other cases, it may be determined as "continuation". In this embodiment, if the distance between the axial direction and the target position is equal to or less than a certain value, it is determined to be "finished", otherwise it is determined to be "continued".

(Step S108)
In step S108, the control unit 106 receives the position/posture data defining the motion from the motion generation unit 104, and controls the robot arm 2 to move to the received position/posture.

(Step S109)
In step S109, the control unit 106 receives end determination data from the end determination unit 105, and if the end determination data is "continue", the control unit 106 sends an imaging trigger to the imaging device 1, and the process returns to S102. If the end determination data is "end", the process proceeds to S110.

(Step S110)
At step S110, the control unit 106 executes the task operation. Here, the task operation is to control the robot arm 2 and the end effector 3 with preset movements. The task operation of this embodiment is a screw removing operation in which a screwdriver, which is the end effector 3, removes a screw at a target position. For example, after the driver, which is the end effector 3, is advanced and the contact between the driver, which is the end effector 3, and the screw is detected by the proximity sensor, the screw is removed while rotating the driver, and the position and orientation before executing the task operation. back to

(Step S111)
In step S111, the control unit 106 determines to end the processing. If there is a next target screw, the process returns to S101. If task execution is completed for all targets, the process ends.

[effect]
According to this embodiment, robot control by visual servo can be realized from the axial direction and the target position in the image. As a result, the task can be performed with high accuracy even when sufficient features cannot be extracted from the image while reducing the load of the calibration work.

[Modification 1 of the first embodiment]
Although the controlled variable H′ of the controlled object is defined as the translation of the x-axis and the y-axis in the end effector coordinate system 44 according to the definition of Equation 2, it is not limited to this. The control amount H' may be rotation around the x-axis and y-axis, or translation and rotation may be used in combination.

Hereinafter, the control processing method of Modification 1 of the first embodiment will be described with reference to the flowchart of FIG. Steps S101 to S105 and steps S107 to S111 in FIG. 6 are substantially the same as those of the first embodiment, and therefore are omitted.

(Step S1061)
In step S1061, the motion generation unit 104 receives axial direction data and target position data from the axial direction acquisition unit 102 and the target position detection unit 103, and determines whether to change motion parameters to be generated. Here, the operating parameter is a component of the controlled variable H' of the controlled object. Specifically, they are translational components about the x-, y-, and z-axes of the controlled variable H', and rotational components about the x-, y-, and z-axes. If the motion generating unit 104 determines to change the motion parameter, the process proceeds to S1062, and if it determines not to change the motion parameter, the process proceeds to S1063. The determination may switch between change determination and non-change determination each time S1061 is executed a certain number of times, or may be performed by switching between change determination and non-change determination using a pre-registered pattern. You can judge by the distance from In this embodiment, when the distance between the axial direction and the target position is equal to or less than a certain value, and the next determination, the change determination is performed once. This allows visual servoing by combining multiple operating parameters.

(Step S1062)
In step S1062, the motion generator 104 switches between fixed components and variable components of the motion parameters. From Equation 2, the control amount H′ before the motion generator 104 switches the motion parameter has zero components other than the translational components in the x-axis and y-axis directions of the end effector coordinate system 44 . For example, if the motion parameters are changed so that the components other than the rotation around the x-axis and y-axis of the end effector coordinate system 44 are set to 0, the control amount H' is given by Equation (6).

As another example, if the operating parameters are changed to move forward in the z-axis direction of the end effector coordinate system 44 by a constant amount L, the control amount H' is given by Equation (7).

In the present embodiment, for example, in the first change determination, the motion parameter is changed so as to move forward by a distance of P′ _z /2 in the z-axis direction of the end effector coordinate system 44 . In the second change determination, the end effector coordinate system 44 is changed so that the components other than the rotation around the x-axis and y-axis become zero.

If the feature of the controlled object or the target position cannot be sufficiently extracted from the image, the translational component and the rotational component of the motion parameter cannot be controlled at the same time. However, as in the present embodiment, the translational component error and the rotational component error can be separated and controlled by sequentially switching between the translation control, the forward control, and the rotation control. Therefore, even if the characteristics of the controlled object or the target position cannot be sufficiently extracted from the image, visual servoing of the position and orientation can be realized.

[Modification 2 of the first embodiment]
Although the driver that is the end effector 3 is used as the controlled object and the screw is used as the target position, it is not necessary to be limited to this. A suction hand may be used as the end effector 3, or a multi-fingered hand may be used. A part to be gripped or a part to be assembled may be used as the target position.

For example, as in an image 71 captured by the imaging device 1 in FIG. 7, the control target is a multi-fingered hand 72 that is the end effector 3, the axial direction is a center line 74 of the multi-fingered hand that is the end effector 3, and the target position is a target position. A center position 75 of the gripping component 73 may be used. Control is performed so as to reduce the distance between the axial direction and the target position in substantially the same manner as in the first embodiment. After that, the multifingered hand, which is the end effector 3, is advanced, closed, and returned to its original position and posture, so that the part to be grasped can be accurately grasped.

As another example, the case of detecting the control position of the controlled object and the axial direction of the target from the image will be described with reference to FIG. 8 (position detection and direction detection). Reference numeral 81 denotes an image captured by the imaging device 1, and 82 denotes an assembly component that is a control target and is gripped by the multi-fingered hand that is the end effector 3. FIG. Reference numeral 83 denotes a control position at which the center of the assembly part to be controlled is detected, 84 the assembly target part, and 85 the target axial direction detected based on the assembly target part. Control is performed so as to reduce the distance between the axial direction and the control position in substantially the same manner as in the first embodiment. After that, the multifingered hand, which is the end effector 3, is moved forward, opened, and returned to its original position and orientation, so that the control position is set for the controlled object, and the task is accurately performed even when the axial direction is set for the controlled object. can run. In this embodiment, grasping includes concepts of grasping (for example, grasping or sandwiching with a plurality of fingers) and holding (for example, sucking using a vacuum suction pad or electromagnetic force).

[Modification 3 of the first embodiment]
A motion display section (not shown) may be used to display the motion of the robot. An example of the operation display unit will be described below with reference to FIG. In FIG. 9, reference numeral 91 denotes the screen of the action display unit presented to the user, 92 denotes the image received from the image acquisition unit 101, and 93 denotes the axial direction received from the axial direction acquisition unit 102. FIG. Reference numeral 94 denotes the target position received from the target position detection unit 103, 95 denotes the direction and distance in which the axis moves, and 96 denotes a pause button that is pressed to temporarily stop the control loop. Reference numeral 97 denotes a step execution button that is pressed when executing the control loop once, and 98 denotes a restart button that is pressed when resuming the control loop that has been paused. Button operations are performed by a UI device (for example, a mouse) not shown. In FIG. 9, the direction of the arrow 95 indicates the direction of movement of the shaft, and the length of the arrow indicates the distance of movement of the shaft. However, the distance by which the axis moves may be a numerical value (actual size unit, pixel unit). Moreover, both the direction and the distance in which the axis moves may be displayed, or only one of them may be displayed.

First, the motion display unit receives image data, axial direction data, and target position data from the image acquiring unit 101, the axial direction acquiring unit 102, and the target position detecting unit 103, and displays the axial direction and the target position superimposed on the image. . The motion display unit also receives the position and orientation data from the motion generation unit 104, and superimposes and displays the direction and distance of movement of the controlled object. Next, when the user presses the pause button 96 , the controller 106 pauses control of the robot arm 2 . When the user presses the step execution button 97, the control unit 106 resumes control of the robot, and after reaching step S108, suspends control of the robot again. Finally, when the user presses resume button 98, controller 106 resumes control of the robot.

In this way, the movement display unit is used to display the robot's movement, and by moving the robot little by little while confirming the detection result and the movement of the robot, it is possible to adjust the parameters necessary for control and detection, and to respond when a problem occurs. Can assist in root cause analysis.

[Modification 4 of the first embodiment]
Although the position and orientation set by the teaching operation using the teaching pendant are acquired in S101, the present invention is not limited to this. For example, the position and orientation may be obtained based on the result of detecting a screw and measuring its approximate three-dimensional position based on an image captured by a stereo camera, which is the imaging device 1 . Alternatively, the position and orientation may be set by a simulator using a three-dimensional model such as CAD data. Also, a plurality of target positions may be set together with the task execution order.

(Second embodiment)
In the second embodiment, as in the first embodiment, an imaging device attached to the robot images the controlled object and the target position, and guides the controlled object to the target position. However, the second embodiment differs from the first embodiment in that it includes a light projection device that projects light and a plane measurement unit that measures a target plane on which a screw is fastened from an image projected with light. different from By measuring the plane on which the target screw is fastened, it is possible to realize the work of controlling the plane and the axial direction to form a predetermined angle (for example, perpendicular). That is, the target posture at the target position can be obtained. By obtaining the target position and the target posture in this way and controlling the robot, the robot can stably remove the screw by applying the driver, which is an end effector, to the screw substantially perpendicularly. In other words, the task can be stably performed by the robot.

[Device configuration]
The block diagram of FIG. 10 shows an example of a robot system 2000 including the control device 200 of the second embodiment. The robot system 2000 includes an imaging device 1 , a robot arm 2 , an end effector 3 , a light projection device 4 and a control device 200 . Note that the imaging device 1, the robot arm 2, and the end effector 3 are substantially the same as those in the first embodiment, so description thereof will be omitted.

The light projection device 4 is a device for projecting characteristic light onto an object for planar measurement. For example, the light projection device 4 is configured by a cross-line laser projector that projects cross-shaped line laser light, and projects the laser light according to a projection trigger received from the control unit 106 .

The control device 200 includes an image acquisition unit 101 , an axial direction acquisition unit 102 , a target position detection unit 103 , a plane measurement unit 201 , a motion generation unit 202 , an end determination unit 105 and a control unit 106 . Note that the image acquisition unit 101, the axial direction acquisition unit 102, the target position detection unit 103, the end determination unit 105, and the control unit 106 are substantially the same as those in the first embodiment, so description thereof will be omitted.

The plane measurement unit 201 receives image data on which laser light is projected from the image acquisition unit 101, detects a three-dimensional normal line of the plane from the image data, and converts target plane data (normal line data) to a motion generation unit. 202.

The motion generator 202 defines the motion of the tip of the robot arm 2 based on the axial direction data, target position data, and normal line data received from the axial direction acquisition unit 102, target position detection unit 103, and plane measurement unit 201. Calculate the position and orientation of the Then, the calculated position and orientation data is sent to the control unit 106 .

[Control processing]
A control processing method by the control device 200 and the robot system 2000 of the second embodiment will be described with reference to the flowchart of FIG. Steps S101 to S105 and steps S107 to S111 are substantially the same as in the first embodiment, respectively, so description thereof will be omitted.

(Step S201)
In step S201, the light projection device 4 receives a projection trigger from the control unit 106 and projects laser light. As a result, even a featureless plane can be stably measured by the stereo camera, which is the imaging device 1 .

(Step S202)
In step S<b>202 , the imaging device 1 receives an imaging trigger from the control unit 106 , performs imaging, and sends an image signal of the imaged image to the image acquiring unit 101 .

(Step S203)
In step S<b>203 , the image acquisition unit 101 receives an image signal from the imaging device 1 and sends image data to the plane measurement unit 201 .

(Step S204)
In step S<b>204 , the plane measurement unit 201 receives the image data from the image acquisition unit 101 , detects the normal line of the plane from the image data, and sends the normal line data to the motion generation unit 202 . Detection of the normal line is performed by, for example, measuring the three-dimensional point group of the laser area projected on the image by stereo matching, based on the left and right images of the stereo camera, which is the imaging device 1 . The plane measurement unit 201 detects the normal line of the plane by calculating the third principal component through principal component analysis of the measured three-dimensional point group.

(Step S205)
In step S205, the motion generation unit 202 receives axial direction data, target position data and normal line data from the axial direction acquisition unit 102, the target position detection unit 103 and the plane measurement unit 201, and calculates the motion of the tip of the robot arm 2. Calculate the specified pose. Also, the motion generation unit 202 sends the position and orientation data to the control unit 106 .

An example of motion generation will be described below. In this embodiment, the shaft of the driver, which is the end effector 3, is controlled so as to be applied approximately perpendicularly to the plane on which the screw is fastened. When the end effector coordinate system 44 is represented by the robot coordinate system 41, Equation 8 is obtained.

Further, when the planar orientation O measured by the imaging device coordinate system 43 is represented by the robot coordinate system 41, Equation 9 is obtained.

In order to control the driver axis so that it hits the plane perpendicularly, the position component of the end effector coordinate system 44 is left as is, and the z-axis of the attitude component is roughly aligned with the normal to the plane. With the normal to the plane as the target orientation, the driver position and orientation obtained so that the z-axis direction of the orientation component is the target orientation is used as the initial position and orientation, and substantially the same motion generation as in the first embodiment is performed.

In this way, when the end effector 3 is moved forward, it is possible to control the shaft of the driver, which is the end effector 3, to hit the plane on which the screw is fastened approximately perpendicularly. Note that in the present embodiment, the plane measurement unit 201 performs plane measurement only once after the initialization in S101, but it is not necessary to limit to this. The plane measurement unit 201 may perform the plane measurement immediately before the motion generation in S205, or the plane measurement unit 201 and the motion generation unit 202 may perform the plane measurement and motion generation again immediately before the task execution in S110. good.

[effect]
According to this embodiment, by performing plane measurement, visual servo can be executed to control the angle between the axial direction associated with the control target and the plane. As a result, a task in which the attitude of the controlled object at the target position is important can be realized.

[Modification 1 of Second Embodiment]
Although the plane measurement is performed using the image projected by the laser beam in S204, it is not necessary to limit to this. For example, a pattern on a plane may be used, a pattern may be projected by a projector, or a depth camera may be used as the imaging device 1 .

(Third embodiment)
In the third embodiment, as in the first embodiment, an imaging device attached to the robot images the controlled object and the target position, and guides the controlled object to the target position. However, the third embodiment differs from the first embodiment in that the derived result is reflected in the calibration value. Here, the calibration value is a parameter representing the position and orientation relationship among the imaging device 1 , the robot arm 2 and the end effector 3 . The greater the error in the calibration value, the longer it takes for the visual servo operation to converge, but the method of the present embodiment can quickly converge.

[Device configuration]
The block diagram of FIG. 12 shows an example of a robot system 3000 including the control device 300 of the third embodiment. A robot system 3000 includes an imaging device 1 , a robot arm 2 , an end effector 3 , and a control device 300 . Note that the imaging device 1, the robot arm 2, and the end effector 3 are substantially the same as those in the first embodiment, so description thereof will be omitted.

The control device 300 includes an image acquisition unit 101, an axial direction acquisition unit 102, a target position detection unit 103, a motion generation unit 301, a calibration value correction unit 302, a calibration value storage unit 303, and an end determination unit. 105 and a control unit 106 . Note that the image acquisition unit 101, the axial direction acquisition unit 102, the target position detection unit 103, the end determination unit 105, and the control unit 106 are substantially the same as those in the first embodiment, so description thereof will be omitted.

The motion generator 301 generates the motion of the tip of the robot arm 2 based on the axial direction data, the target position data, and the calibration values received from the axial direction acquisition unit 102, the target position detection unit 103, and the calibration value storage unit 303. Calculate the specified pose. Then, the position/orientation data is sent to the control unit 106 and the calibration value correction unit 302 .

The calibration value correction unit 302 receives the position and orientation data from the motion generation unit 301 , corrects the calibration values, and sends the corrected calibration values to the calibration value storage unit 303 .

The calibration value storage unit 303 receives and stores the calibration values from the calibration value correction unit 302 . Also, the calibration value storage unit 303 sends the calibration value to the motion generation unit 301 . For example, the calibration value storage unit 303 is composed of memory.

[Control processing]
A control processing method by the control device 300 and the robot system 3000 of the third embodiment will be described with reference to the flowchart of FIG. Steps S101 to S105 and steps S107 to S111 are substantially the same as in the first embodiment, respectively, so description thereof will be omitted.

(Step S301)
In step S<b>301 , the motion generation unit 301 receives axial direction data, target position data and calibration values from the axial direction acquisition unit 102 , target position detection unit 103 and calibration value storage unit 303 . The motion generation unit 301 uses the received calibration values to calculate the position and orientation that define the motion of the tip of the robot arm 2 in substantially the same manner as in the first embodiment. Also, the motion generation unit 301 sends the position and orientation data to the control unit 106 and the calibration value correction unit 302 .

(Step S302)
In step S302, the calibration value correction unit 302 receives the result (position/orientation data) generated by the motion generation unit 301 from the motion generation unit 301, and corrects the calibration values. Then, the corrected calibration value is sent to the calibration value storage unit 303 . By correcting the calibration value while executing visual servo, the error can be gradually reduced. The calibration value to be corrected may be a transformation matrix G representing the relationship in position and orientation between the tip of the robot arm 2 and the camera as the imaging device 1, or the camera as the imaging device 1 and the driver as the end effector 3. may be a transformation matrix H representing the relationship between the positions and orientations of . In this embodiment, the transformation matrix G is corrected. Assuming that the transformation matrix after correction is G′, Equation 10 holds from Equation 5.

(Step S303)
In step S303, the calibration value storage unit 303 receives and stores the corrected calibration value from the calibration value correction unit 302. FIG.

[effect]
According to this embodiment, by correcting the calibration value while controlling the controlled object, the error in the calibration value can be gradually reduced. As a result, the control can be finished more quickly when the next visual servo is executed.

[Modification 1 of the third embodiment]
Although only the calibration values are saved in S303, this need not be the case. For example, a set of the current robot position/orientation and calibration values may be saved, or a set of an initial position/orientation ID set by teaching work and calibration values may be saved. As a result, even if there is a calibration error that depends on the position and orientation of the robot, the error can be reduced for each position and orientation.

[Hardware configuration]
The control devices 100, 200 and 300 shown in FIGS. 2, 10 and 12 can be implemented using a general PC (personal computer). Computer programs and data for causing the CPU of the PC to execute the function of each unit in the control devices 100, 200, and 300 are stored in the hard disk device. The CPU can appropriately load computer programs and data stored in the hard disk device into a memory such as a RAM and execute processing using the computer programs and data. As a result, the PC can implement the functions of the control devices 100 , 200 , and 300 .

[Other modification 1]
Although two grayscale cameras are used as the imaging device 1 in all the embodiments, it is not necessary to be limited to this. Three or more grayscale cameras, RGB color cameras, or depth cameras may be used.

[Other modification 2]
Although a six-axis robot is used as the robot arm 2 in all the embodiments, it is not necessary to be limited to this. Other articulated robots, parallel link robots, or orthogonal robots may be used.

[Other modification 3]
In the first embodiment, the proximity sensor is used to detect the contact between the driver, which is the end effector 3, and the screw, but it is not necessary to be limited to this. Detection may be performed based on an image, a distance sensor may be used, or a force sensor may be used.

<Effects of Embodiment>
According to the first embodiment, robot control by visual servo can be realized from the axial direction and the target position in the image. As a result, the task can be performed with high accuracy even when sufficient features cannot be extracted from the image while reducing the load of the calibration work.

According to the second embodiment, by performing a plane measurement, visual servoing can be performed to control the angle between the axial direction associated with the controlled object and the plane. As a result, a task in which the attitude of the controlled object at the target position is important can be realized.

According to the third embodiment, by correcting the calibration value while controlling the controlled object, the error in the calibration value can be gradually reduced. As a result, the control can be finished more quickly when the next visual servo is executed.

<Definition>
A target position is the position at which the end effector performs a task. The axial direction is a line segment in the image associated with the controlled object for measuring the position and orientation relationship between the controlled object and the target position. A task motion is to control a robot arm and an end effector with a preset motion. An operating parameter is a component of a controlled variable to be controlled. A calibration value is a parameter representing the position/orientation relationship among an imaging device, a robot arm, and an end effector.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

1000: robot system, 1: imaging device, 2: robot arm, 3: end effector, 100: control device, 101: image acquisition unit, 102: axial direction acquisition unit, 103: target position detection unit, 104: motion generation unit , 105: end determination unit, 106: control unit

Claims (14)

An imaging device attached to a robot acquires an image of a controlled object whose rotation can be controlled by the robot and a target position, which is a destination of movement of the controlled object, and is a target position in a concave portion. an image acquisition means;
A progression for acquiring a two-dimensional direction in the image representing the traveling direction of the rotational axis of the controlled object and along the rotational axis of the controlled object, based on the edges of the controlled object in the image. a direction acquisition means;
target position detection means for detecting a two-dimensional position in the image representing the target position from the image ;
The target position exists on the extension of the traveling direction of the rotating shaft, and the controlled object performs an operation of changing the attitude of the controlled object so that the traveling direction of the rotating shaft and the depth direction of the recess match. a motion generation means for generating a motion of moving the controlled object closer to the target position along the traveling direction of the controlled object whose attitude has been changed, which is generated before reaching the target position;
a control means for controlling the controlled object according to the operation;
with
The motion generating means obtains a control amount of the posture of the control object that brings the line segment along the two-dimensional direction closer to the two-dimensional position, and updates the posture of the control object based on the control amount. A control device characterized by: 2. The control device according to claim 1, further comprising end determination means for determining whether or not to end the control by said control means based on said traveling direction and said target position. The motion generating means switches motion parameters, which are components of the control amount of the controlled object, based on the traveling direction and the target position, and generates the motion of the controlled object according to the switched motion parameters. A control device according to claim 1 . 2. The apparatus according to claim 1, further comprising operation display means for superimposing and displaying at least one of the target position, the traveling direction, and the direction or distance in which the axis to be controlled moves on the image. controller. Further comprising plane measurement means for measuring the target plane,
2. The control device according to claim 1, wherein the motion generating means obtains the depth direction of the concave portion based on the target plane. calibration value storage means for storing a calibration value, which is a parameter representing a position and orientation relationship between the imaging device, a robot arm of the robot, and the controlled object;
calibration value correcting means for correcting the calibration value based on the calibration value and the result generated by the action generating means;
2. The controller of claim 1, further comprising: a. 7. The control device according to claim 6, wherein the controlled object is an end effector attached to the robot arm. 2. The control device according to claim 1, wherein the recess is a screw hole, and the controlled object is a driver. an image acquisition means for acquiring an image of a controlled object whose rotation can be controlled by the robot and a target, which is a destination of movement of the controlled object and which is located in a concave portion, by an imaging device attached to the robot;
position detection means for detecting from the image a two-dimensional position in the image representing a control position associated with the control object;
direction detection means for detecting a two-dimensional direction in the image representing a direction of travel along an axis of rotation of the controlled object associated with the target, based on edges of the controlled object in the image;
The controlled object moves to a target position by changing the attitude of the controlled object so that the advancing direction of the rotating shaft and the depth direction of the concave portion are aligned based on the advancing direction of the rotating shaft and the control position. a motion generating means for generating a motion for bringing the controlled object closer to the control position along the advancing direction of the rotation axis of the controlled object whose attitude has been changed;
a control means for controlling the controlled object according to the operation;
with
The motion generating means obtains a control amount of the posture of the control object that brings the line segment along the two-dimensional direction closer to the two-dimensional position, and updates the posture of the control object based on the control amount. A control device characterized by: An imaging device attached to a robot acquires an image of a controlled object whose rotation can be controlled by the robot and a target position, which is a destination of movement of the controlled object, and is a target position in a concave portion. an image acquisition means;
A progression for acquiring a two-dimensional direction in the image representing the traveling direction of the rotational axis of the controlled object and along the rotational axis of the controlled object, based on the edges of the controlled object in the image. a direction acquisition means;
target position detection means for detecting a two-dimensional position in the image representing the target position from the image ;
The target position exists on the extension of the traveling direction of the rotating shaft, and the controlled object performs an operation of changing the attitude of the controlled object so that the traveling direction of the rotating shaft and the depth direction of the recess match. a motion generation means for generating a motion of moving the controlled object closer to the target position along the traveling direction of the controlled object whose attitude has been changed, which is generated before reaching the target position;
a control means for controlling the controlled object according to the operation;
with
The motion generating means obtains a control amount of the attitude of the controlled object that brings the line segment along the two-dimensional direction closer to the two-dimensional position, and performs control for updating the attitude of the controlled object based on the control amount. a device;
the robot;
the imaging device;
A robot system comprising: A method of operating a controller, comprising:
An imaging device attached to a robot acquires an image of a controlled object whose rotation can be controlled by the robot and a target position, which is a destination of movement of the controlled object, and is a target position in a concave portion. process and
obtaining a two-dimensional direction in the image representing the direction of travel of the axis of rotation of the controlled object and along the axis of rotation of the controlled object, based on the edges of the controlled object in the image; and,
detecting from the image a two-dimensional position in the image representing the target position;
The target position exists on the extension of the traveling direction of the rotating shaft, and the controlled object performs an operation of changing the attitude of the controlled object so that the traveling direction of the rotating shaft and the depth direction of the recess match. a step of generating a motion for bringing the controlled object closer to the target position along a traveling direction of the controlled object generated before reaching the target position and having a changed attitude;
a step of controlling the controlled object according to the operation;
has
In the step of generating the motion, a control amount of the attitude of the controlled object that brings the line segment along the two-dimensional direction closer to the two-dimensional position is obtained, and the attitude of the controlled object is updated based on the control amount. A method of operating a control device, characterized by: A method of operating a controller, comprising:
obtaining an image of a controlled object whose rotation can be controlled by the robot and a target to which the controlled object moves and which is located in a concave portion, by an imaging device attached to the robot;
detecting from the image a two-dimensional position in the image representing a control position associated with the controlled object;
Detecting a two-dimensional direction in the image representing a direction of travel along an axis of rotation of the controlled object associated with the target based on edges of the controlled object in the image;
The controlled object moves to a target position by changing the attitude of the controlled object so that the advancing direction of the rotating shaft and the depth direction of the concave portion are aligned based on the advancing direction of the rotating shaft and the control position. a step of generating a motion for bringing the controlled object closer to the control position along the advancing direction of the rotation axis of the controlled object generated before reaching and further changing the attitude;
a step of controlling the controlled object according to the operation;
has
In the step of generating the motion, a control amount of the attitude of the controlled object that brings the line segment along the two-dimensional direction closer to the two-dimensional position is obtained, and the attitude of the controlled object is updated based on the control amount. A method of operating a control device, characterized by: A method of operating a controller, comprising:
An imaging device attached to a robot acquires an image of a controlled object whose rotation can be controlled by the robot and a target position, which is a destination of movement of the controlled object, and is a target position in a concave portion. process and
obtaining a two-dimensional direction in the image representing the direction of travel of the axis of rotation of the controlled object and along the axis of rotation of the controlled object, based on the edges of the controlled object in the image; and,
detecting from the image a two-dimensional position in the image representing the target position;
The target position exists on the extension of the traveling direction of the rotating shaft, and the controlled object performs an operation of changing the attitude of the controlled object so that the traveling direction of the rotating shaft and the depth direction of the recess match. a step of generating a motion for bringing the controlled object closer to the target position along a traveling direction of the controlled object generated before reaching the target position and having a changed attitude;
a step of controlling the controlled object according to the operation;
has
In the step of generating the motion, a control amount of the attitude of the controlled object that brings the line segment along the two-dimensional direction closer to the two-dimensional position is obtained, and the attitude of the controlled object is updated based on the control amount. A program that causes a computer to execute the operation method of the control device. A method of operating a controller, comprising:
obtaining an image of a controlled object whose rotation can be controlled by the robot and a target to which the controlled object moves and which is located in a concave portion, by an imaging device attached to the robot;
detecting from the image a two-dimensional position in the image representing a control position associated with the controlled object;
Detecting a two-dimensional direction in the image representing a direction of travel along an axis of rotation of the controlled object associated with the target based on edges of the controlled object in the image;
The controlled object moves to a target position by changing the attitude of the controlled object so that the advancing direction of the rotating shaft and the depth direction of the concave portion are aligned based on the advancing direction of the rotating shaft and the control position. a step of generating a motion for bringing the controlled object closer to the control position along the advancing direction of the rotation axis of the controlled object generated before reaching and further changing the attitude;
a step of controlling the controlled object according to the operation;
has
In the step of generating the motion, a control amount of the attitude of the controlled object that brings the line segment along the two-dimensional direction closer to the two-dimensional position is obtained, and the attitude of the controlled object is updated based on the control amount. A program that causes a computer to execute the operation method of the control device.

Images