JP2014151377A - Robot control method, robot control device, robot system, robot, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable work continuously and safely even in a case where a target article retained by an end effector cannot be recognized with an image.SOLUTION: An image including a first article and a second article retained by an end effector is obtained to recognize both a first position that is a discretionary position of a surface of the first article and a second position that is a discretionary position of a surface of the second article. Whether the first and second positions are recognized is determined, and, based on the determination result, the end effector controls a movable part disposed at a tip part so as to move the second position close to a third position that is apart from the first position toward inside of the first article by a given distance. Here, if the first and second positions are recognized, the movable part is controlled on the basis of the recognition result of the first and second positions, while if at least one of the first and second positions cannot be recognized, force applied to the end effector is obtained to control the movable part on the basis of the obtained force applied to the end effector.

Description

  The present invention relates to a robot control method, a robot control device, a robot system, a robot, and a program.

  Patent Document 1 discloses switching from control based on imaging information of the front camera to control based on imaging information of the side camera immediately before or immediately after the hand grips the object to be gripped.

  In Patent Document 2, the position and orientation of a workpiece are measured by a camera, the robot is feedback-controlled based on the measurement result, and the workpiece follows the movement of the workpiece. It is disclosed to perform re-supplementation control for re-supplementing a workpiece by changing the target position and moving speed of the camera based on the recognized final position of the workpiece when it is out of the field of view.

JP 2011-235386 A JP 2011-136377 A

  The invention described in Patent Document 1 only switches the camera from the front camera to the side camera immediately before or immediately after the hand grips the gripping object, and performs visual servoing immediately before or after the hand grips the gripping object. Is going. However, the invention described in Patent Document 1 does not describe how to deal with the case where the side camera loses sight of the object to be grasped.

  The invention described in Patent Literature 2 can re-supplement a workpiece even when the workpiece is lost. However, in the invention described in Patent Document 2, there is no description about how to deal with the case where the workpiece cannot be recognized even if the workpiece is re-supplemented.

  This is because the inventions described in Patent Documents 1 and 2 control a movable part that grips an object by visual servo. That is, in the inventions described in Patent Documents 1 and 2, the case where the workpiece or the grasped object cannot be recognized from the image is not considered.

  However, there may be a case where a work or a grasped object cannot be recognized from an image during visual servo work. If the workpiece or the gripping object cannot be recognized from the image, control based on the image cannot be performed. Therefore, in this case, generally, the movable part is only stopped (work stop).

  If the workpiece or the gripping target cannot be recognized from the image, it may be assumed that the workpiece or the gripping target is at the position where the workpiece or the gripping target was last recognized and the operation can be continued. However, in this case, there is a possibility that the workpiece or the gripping object is destroyed during the operation because the estimated position of the workpiece or the gripping object is different from the actual position.

  Therefore, the present invention provides a robot control method, a robot control device, a robot system, a robot, and a program that can continue work safely even when an object held by an end effector cannot be recognized from an image. Objective.

  A first aspect for solving the above problem is a robot control method, the first step of acquiring an image including a first object and a second object held by an end effector, and the acquisition A second step of recognizing a first position which is an arbitrary position on the surface of the first object and a second position which is an arbitrary position on the surface of the second object from the obtained image; The third step for determining whether or not the first position and the second position can be recognized in step 2 and the control method using different control methods based on the determination result of the third step, A fourth unit that controls the movable portion provided at the tip of the end effector so as to bring the second position closer to a third position that is a predetermined distance away from the first position to the inside of the first object; And the fourth step. When the first position and the second position can be recognized, the top controls the movable part based on the recognition result of the first position and the second position, and the first position When at least one of the position and the second position cannot be recognized, a force applied to the tip of the movable part is acquired, and the movable part is controlled based on the acquired force applied to the tip of the movable part. It is characterized by.

  According to the first aspect, when the first position that is an arbitrary position on the surface of the first object and the second position that is an arbitrary position on the surface of the second object can be recognized from the image. If the movable part is controlled based on the recognition result of the first position and the second position, and the first position or the second position cannot be recognized from the image, the force applied to the tip of the movable part is applied. The movable part is acquired and controlled based on the force applied to the tip of the acquired movable part. Thereby, even when the object held by the end effector cannot be recognized from the image, the work can be safely continued. Furthermore, by controlling the movable portion based on the force applied to the tip of the movable portion, the movable portion can be controlled so that the second position is brought close to the third position that cannot be recognized from the image.

  Here, in the fourth step, when at least one of the first position and the second position cannot be recognized, the movable part may be controlled by impedance control. Thereby, even when an external force is applied to the first object, the second object, and the movable part, the movable part can be reliably moved while preventing the first object and the second object from being destroyed. it can.

  Here, in the fourth step, when the first position and the second position are recognized, the movable unit is controlled so that the second position moves by a first amount, When at least one of the first position and the second position cannot be recognized, the movable unit is controlled so that the second position moves by a second amount smaller than the first amount. May be. Thereby, when an object cannot be recognized from an image, a movable part can be moved safely.

  Here, in the third step, when at least one of the first position and the second position is not recognized, at least one of the first position and the second position is the image. If at least one of the first position and the second position is not included in an arbitrary range of the image, at least one of the first position and the second position May not be recognized by image processing, and may be at least one of a case where the positional relationship between the first position and the second position is outside a set range. Thereby, work can be continued corresponding to various situations.

  Here, in the first step, the step of acquiring the image acquires two images obtained by capturing the first object and the second object from different directions, and in the fourth step, If at least one of the first position and the second position cannot be recognized from at least one of the two images, at least one of the first position and the second position can be recognized. You may control the said movable part as it is when it was not. Thereby, work can be continued more safely.

  Here, in the fourth step, the third position is calculated by moving the second object in the direction toward the inside of the second object from the second position by the predetermined distance. The movable portion may be controlled to bring the second position closer to the third position. Thereby, a movable part can be controlled to the 3rd position which is inside the 2nd object and cannot be visually recognized.

  Here, in the fourth step, when the first position and the second position can be recognized, the second position is obtained from the recognition result, and the first position and the second position are obtained. When at least one of the positions cannot be recognized, the second position may be obtained based on the amount of movement of the movable part. Accordingly, even when the first position or the second position cannot be recognized from the image, the second position can be obtained.

  A second aspect for solving the above-described problem is a robot control device, the image acquisition unit acquiring an image including a first object and a second object held by the end effector, and the acquired An image processing unit for recognizing a first position, which is an arbitrary position on the surface of the first object, and a second position, which is an arbitrary position on the surface of the second object, from the image; Based on the position and the recognition result of the second position, the third position is moved away from the first position by a predetermined distance from the first position to the inside of the first object using different control methods. A control unit that controls a movable unit provided at a tip of the end effector so as to approach the position, and when the first position and the second position are recognized, the first position and the second position Based on the recognition result of the second position, the movable part Control, when at least one of the first position and the second position cannot be recognized, obtain a force applied to the tip of the movable part, and based on the obtained force applied to the tip of the movable part And a control unit that controls the movable unit. Thereby, even when the object held by the end effector cannot be recognized from the image, the work can be safely continued. Furthermore, by controlling the movable portion based on the force applied to the tip of the movable portion, the movable portion can be controlled so that the second position is brought close to the third position that cannot be recognized from the image.

  A third aspect for solving the above-described problem is a robot control apparatus, which is an image acquisition unit that acquires an image, and an image that recognizes a first object and a second object held by an end effector from the image. Based on the recognition result of the processing unit and the image processing unit, the position of the end effector is adjusted so as to bring the position of the second object closer to a target position separated from the position of the first object by a predetermined distance. A control unit that controls the position of the first object and the second object when the image processing unit can recognize the first object and the second object. The position of the end effector is controlled based on the position of the first effector, and when the image processing unit cannot recognize at least one of the first object and the second object, a force applied to the end effector is obtained. And, and controlling the position of the end effector on the basis of the acquired power. Thereby, even when the object held by the end effector cannot be recognized from the image, the work can be safely continued.

  A fourth aspect for solving the above-described problem is a robot system, which has a movable part and a robot having an end effector provided at the tip of the movable part, a first object, and the movable part holds the robot system. An imaging unit that captures an image including the second object, a first position that is an arbitrary position on the surface of the first object, and a second position that is an arbitrary position on the surface of the second object The second position is changed from the first position to the first object using a different control method based on the recognition result of the first position and the second position. A control unit that controls the movable unit so as to approach a third position that is a predetermined distance inward of the first position, and when the first position and the second position can be recognized, And the recognition result of the second position, When the moving part is controlled and at least one of the first position and the second position cannot be recognized, the force applied to the tip of the movable part is acquired, and the force applied to the acquired tip of the movable part And a control unit that controls the movable unit based on the above. Thereby, even when the object held by the end effector cannot be recognized from the image, the work can be safely continued.

  Here, the imaging unit includes two imaging devices that capture the first object and the second object from different directions, and the image processing unit is captured by each of the two imaging devices. The first position and the second position are recognized from two images, and the first position and the second position are detected from an image captured by at least one of the two imaging devices. When at least one of the first position and the second position cannot be recognized, it may be a case where at least one of the first position and the second position cannot be recognized. Thereby, work can be continued more safely.

  According to a fifth aspect of the present invention, there is provided a robot system including a robot having an end effector, an image acquisition unit that acquires an image, a first object from the image, and a first object held by the end effector. Based on the recognition result of the image processing unit and the image processing unit that recognizes the second object, the position of the second object is brought close to a target position that is separated from the position of the first object by a predetermined distance. A control unit that controls the position of the end effector, and when the image processing unit can recognize the first object and the second object, the control unit controls the position of the first object. The position of the end effector is controlled based on the position and the position of the second object, and the image processing unit cannot recognize at least one of the first object and the second object Acquires the force applied to the end effector, and controlling the position of the end effector on the basis of the acquired power. Thereby, even when the object held by the end effector cannot be recognized from the image, the work can be safely continued.

  A sixth aspect for solving the above-described problem is a robot system, in which a robot having a movable part provided with an end effector for holding a first object at a tip, an imaging unit having a lens, and the imaging A control unit that controls the movable unit to bring the first object closer to the second object based on a result of imaging the first object and the second object by a unit, and the control unit includes: When there is the second object, the end effector or the movable part on a line virtually connecting the lens and the first object, the movable part is moved based on the force applied to the end effector. It is characterized by controlling. Thereby, even when the first object overlaps with another object as viewed from the lens, that is, when the first object is covered with another object on the image, the operation can be safely continued. .

  A seventh aspect for solving the above-described problem is a robot system, in which a robot having a movable part provided with an end effector holding a first object at a tip, an imaging unit having a lens, and the imaging A control unit that controls the movable unit to bring the first object closer to the second object based on a result of imaging the first object and the second object by a unit, and the control unit includes: When the first object, the end effector, or the movable part is on a line virtually connecting the lens and the second object, the movable part is moved based on the force applied to the end effector. It is characterized by controlling. Accordingly, even when the second object overlaps with another object as viewed from the lens, that is, when the second object is covered with another object on the image, the operation can be safely continued. .

  Here, when the control unit controls the movable unit based on a force applied to the end effector, a moving speed and a stability according to the force received by the end effector in the direction of the force received by the end effector. The end effector may be moved by sex. Thereby, it is possible to work on the robot while preventing the object or the robot from being destroyed.

  Here, when the control unit controls the movable unit based on the force applied to the end effector, the end effector receives the force other than the direction in which the end effector is moved. The end effector may be moved in the direction of the applied force. Thereby, it is possible to work on the robot while preventing the object or the robot from being destroyed.

  An eighth aspect for solving the above-described problem is a robot system, in which an imaging unit, a robot having an end effector that holds the first object, and the first object are brought close to the second object. A control unit that controls the position of the end effector, and the control unit, when the first object and the second object do not overlap in the imaging direction of the imaging unit, The end effector is controlled based on the imaging result of the imaging unit, and when the first object and the second object overlap in the imaging direction of the imaging unit, the end effector is controlled based on the force applied to the end effector. It controls the effector. Accordingly, even when the first object and the second object overlap in the imaging direction of the imaging unit, that is, when the first object and the second object overlap on the image, the work can be performed safely. Can continue.

  Here, when the control unit controls the movable unit based on a force applied to the end effector, a moving speed and a stability according to the force received by the end effector in the direction of the force received by the end effector. The end effector may be moved by sex. Thereby, it is possible to work on the robot while preventing the object or the robot from being destroyed.

  Here, when the control unit controls the movable unit based on the force applied to the end effector, the end effector receives the force other than the direction in which the end effector is moved. The end effector may be moved in the direction of the applied force. Thereby, it is possible to work on the robot while preventing the object or the robot from being destroyed.

  According to a ninth aspect of the present invention, there is provided a robot system comprising: an imaging unit; and a robot having an end effector that holds the first object. When the first object and the second object do not overlap, the first object is moved closer to the second object based on the imaging result of the imaging unit, and the first imaging object is in the imaging direction of the imaging unit. And a step of controlling the position of the end effector based on a force applied to the end effector when the object and the second object overlap each other. Accordingly, even when the first object and the second object overlap in the imaging direction of the imaging unit, that is, when the first object and the second object overlap on the image, the work can be performed safely. Can continue.

  A tenth aspect for solving the above problem is a robot, which is a movable part, an end effector provided at a tip of the movable part, a first object, and a second object held by the end effector. An image acquisition unit that acquires an image including an object, and a first position that is an arbitrary position on the surface of the first object and a second position that is an arbitrary position on the surface of the second object The second position is moved from the first position to the inside of the first object using a different control method based on the recognition result of the image processing unit and the first position and the second position. A control unit that controls the movable unit to approach a third position separated by a predetermined distance from the first position when the first position and the second position can be recognized. And based on the recognition result of the second position, If at least one of the first position and the second position cannot be recognized, a force applied to the tip of the movable part is acquired, and based on the acquired force applied to the tip of the movable part And a control unit that controls the movable unit. Thereby, even when the object held by the end effector cannot be recognized from the image, the work can be safely continued.

  Here, two movable parts are provided, one of the two movable parts holds the second object, and the other one of the two movable parts holds the first object. May be. As a result, work such as combining two objects held by each of the two movable parts can be safely continued even when the object held by the end effector cannot be recognized from the image.

  An eleventh aspect for solving the above problem is a robot, which is an end effector, an image acquisition unit that acquires an image, a first object from the image, and a second object held by the end effector. And the image processing unit for recognizing the position of the second object based on the recognition result of the image processing unit so as to bring the position of the second object closer to a target position separated from the position of the first object by a predetermined distance. A control unit that controls the position of the effector, and when the image processing unit is able to recognize the first object and the second object, the control unit and the position of the first object Based on the position of the second object, the position of the end effector is controlled, and when the image processing unit cannot recognize at least one of the first object and the second object, the end effector. Gets the force applied to the coater, and controlling the position of the end effector on the basis of the acquired power. Thereby, even when the object held by the end effector cannot be recognized from the image, the work can be safely continued.

  A twelfth aspect for solving the above-described problem is a robot, in which an imaging unit having a lens, a movable unit having an end effector for holding a first object provided at a tip, and the imaging unit A control unit that controls the movable unit so as to bring the first object closer to the second object based on the imaging result of the first object and the second object, and the control unit includes the lens When the second object, the end effector or the movable part is on a line virtually connecting the first object and the first object, the movable part is controlled based on a force applied to the end effector. It is characterized by. Thereby, even when the first object overlaps with another object as viewed from the lens, that is, the first object is covered with the other object on the image, the operation can be safely continued. .

  A thirteenth aspect for solving the above problem is a robot, which is an imaging unit having a lens, a movable unit provided with an end effector holding a first object at a tip, and the imaging unit A control unit that controls the movable unit so as to bring the first object closer to the second object based on the imaging result of the first object and the second object, and the control unit includes the lens When the first object, the end effector, or the movable part is on a line that virtually connects the second object and the second object, the movable part is controlled based on a force applied to the end effector. It is characterized by. Thereby, even when the second object overlaps with another object as viewed from the lens, that is, the second object is covered with the other object on the image, the operation can be safely continued. .

  Here, when the control unit controls the movable unit based on a force applied to the end effector, a moving speed and a stability according to the force received by the end effector in the direction of the force received by the end effector. The end effector may be moved by sex. Thereby, it is possible to work on the robot while preventing the object or the robot from being destroyed.

  Here, when the control unit controls the movable unit based on the force applied to the end effector, the end effector receives the force other than the direction in which the end effector is moved. The end effector may be moved in the direction of the applied force. Thereby, it is possible to work on the robot while preventing the object or the robot from being destroyed.

  A fourteenth aspect for solving the above problem is a robot, in which an acquisition unit that acquires an image captured by an imaging unit, an end effector that holds a first object, and a first object that A control unit that controls the position of the end effector so as to approach the second object, and the control unit is configured such that the first object and the second object do not overlap in the imaging direction of the imaging unit. The end effector is controlled based on the acquired image, and when the first object and the second object overlap in the imaging direction of the imaging unit, the force applied to the end effector is The end effector is controlled based on the above. Accordingly, even when the first object and the second object overlap in the imaging direction of the imaging unit, that is, when the first object and the second object overlap on the image, the work can be performed safely. Can continue.

  Here, when the control unit controls the movable unit based on a force applied to the end effector, a moving speed and a stability according to the force received by the end effector in the direction of the force received by the end effector. The end effector may be moved by sex. Thereby, it is possible to work on the robot while preventing the object or the robot from being destroyed.

  Here, when the control unit controls the movable unit based on the force applied to the end effector, the end effector receives the force other than the direction in which the end effector is moved. The end effector may be moved in the direction of the applied force. Thereby, it is possible to work on the robot while preventing the object or the robot from being destroyed.

  A fifteenth aspect for solving the above problem is a robot, comprising: an imaging unit; and an end effector that holds the first object; and the first object in the imaging direction of the imaging unit. And when the second object does not overlap, the step of bringing the first object closer to the second object based on the imaging result of the imaging unit, and in the imaging direction of the imaging unit, the first object and And a step of controlling the position of the end effector based on a force applied to the end effector when the second object overlaps. Accordingly, even when the first object and the second object overlap in the imaging direction of the imaging unit, that is, when the first object and the second object overlap on the image, the work can be performed safely. Can continue.

It is a figure which shows an example of a structure of the robot system 1 in the 1st Embodiment of this invention. 2 is a block diagram illustrating an example of a functional configuration of a robot system 1. FIG. It is a figure explaining an example of the process which drives a movable part. 2 is a diagram illustrating an example of a hardware configuration of a robot control unit 20. FIG. It is a flowchart which shows an example of the flow of the process which drives a movable part. It is a figure which shows an example of a structure of the robot system 2 in the 2nd Embodiment of this invention. It is the modification of the 1st Embodiment of this invention, and 2nd Embodiment, and is a form which provides an imaging part in a ceiling, when a robot and a worker work in order with respect to the same product. It is a modification of the first embodiment and the second embodiment of the present invention, and is a form in which an imaging unit is provided on the ceiling when a robot performs work in one production cell. This is a modification of the first embodiment and the second embodiment of the present invention, in which an imaging unit is provided in a portion corresponding to the head of the robot. It is the modification of the 1st Embodiment of this invention and 2nd Embodiment, and is a form which provides an imaging part in the front-end | tip of an arm. It is the modification of the 1st Embodiment of this invention and 2nd Embodiment, and is a form which provides an imaging part in the front-end | tip of an arm.

  An embodiment of the present invention will be described with reference to the drawings.

<First Embodiment>
FIG. 1 is a system configuration diagram showing an example of a configuration of a robot system 1 according to an embodiment of the present invention. The robot system 1 in this embodiment mainly includes a robot 10, a robot control unit 20, a first imaging unit 30, and a second imaging unit 40.

  The robot 10 is an arm type robot having an arm 11 (corresponding to a movable part) including a plurality of joints (joints) 12 and a plurality of links 13. At the tip of the arm 11, there is a hand 14 (so-called end effector or hand effector) that can hold an object (work) or can hold a tool to perform a predetermined operation on the object. Provided.

  The joint 12 and the hand 14 are provided with an actuator (not shown) for operating them. The actuator includes, for example, a servo motor and an encoder. The encoder value output by the encoder is used for feedback control of the robot 10 by the robot control unit 20.

  Further, a force sensor (not shown) is provided in the hand 14 or the tip of the arm 11 or the like. The force sensor detects a force applied to the hand 14. As the force sensor, for example, a six-axis force sensor that can simultaneously detect six components, ie, a force component in the translational three-axis direction and a moment component around the three rotation axes, can be used. The physical quantities used in the force sensor are current, voltage, charge amount, inductance, strain, resistance, electromagnetic induction, magnetism, air pressure, light, and the like. The force sensor can detect six components by converting a desired physical quantity into an electrical signal. The force sensor is not limited to six axes, and may be, for example, three axes. The position where the force sensor is provided is not particularly limited as long as the force applied to the hand 14 can be detected.

  Note that the configuration of the robot 10 is not limited to the above-described configuration because the main configuration has been described in describing the features of the present embodiment. This does not exclude the configuration of a general gripping robot. For example, although a 6-axis arm is shown in FIG. 1, the number of axes (number of joints) may be further increased or decreased. The number of links may be increased or decreased. In addition, the shape, size, arrangement, structure, and the like of various members such as the arm, hand, link, and joint may be appropriately changed. The end effector is not limited to the hand 14. The present invention can also be applied to any kind of robot, for example, a self-propelled robot, as long as it can create a placement space.

  The robot control unit 20 performs processing for controlling the entire robot 10. The robot control unit 20 may be installed at a location away from the main body of the robot 10 or may be built in the robot 10 or the like. When the robot control unit 20 is installed at a location away from the main body of the robot 10, the robot control unit 20 is connected to the robot 10 by wire or wirelessly.

  The first imaging unit 30 and the second imaging unit 40 are units that generate image data by imaging the vicinity of the work area of the robot 10 from different angles. The first imaging unit 30 and the second imaging unit 40 include, for example, a camera and are provided on a work table, a ceiling, a wall, or the like. As the first imaging unit 30 and the second imaging unit 40, a visible light camera, an infrared camera, or the like can be employed. The first imaging unit 30 and the second imaging unit 40 have a plurality of lenses, and acquire an image of an object in a predetermined region along the optical axis of the lenses.

  The first image capturing unit 30 and the second image capturing unit 40 are connected to the robot control unit 20, and images captured by the first image capturing unit 30 and the second image capturing unit 40 are input to the robot control unit 20. The first imaging unit 30 and the second imaging unit 40 may be connected to the robot 10 instead of the robot control unit 20 or may be built in the robot 10. In this case, images captured by the first imaging unit 30 and the second imaging unit 40 are input to the robot control unit 20 via the robot 10.

  Next, a functional configuration example of the robot system 1 will be described. FIG. 2 is an example of a functional block diagram of the robot control unit 20.

  The robot 10 includes an operation control unit 101 that controls the arm 11 and the hand 14 based on an encoder value of an actuator, a sensor value of a sensor, and the like.

  The operation control unit 101 controls the arm 11 and the hand 14 based on the information output from the robot control unit 20, the encoder value of the actuator, the sensor value of the sensor, and the like. For example, the operation control unit 101 drives the actuator so as to move the arm 11 in the movement direction and movement amount output from the robot control unit 20. In the present embodiment, the hand 14 grips the workpiece B (corresponding to the second object), and the trajectory point B1 (corresponding to the second position) on the surface of the workpiece B is the reference workpiece A (corresponding to the first object). The operation control unit 101 drives the arm 11 so as to reach the target position A1 (corresponding to the third position) on the surface of

  In the present embodiment, since the standard workpiece A is placed on the floor or the like, the standard workpiece A is used as the standard workpiece. However, the reference workpiece is defined for convenience in the explanation of the control. The reference workpiece may be any object that can determine the target position, and is not particularly limited. Further, the expression “reference” does not relate to the shape, size, position, etc. of the workpiece.

  The robot control unit 20 mainly includes an image acquisition unit 200, an image processing unit 201, a target position calculation unit 203, a control unit 205, and a servo system 206. The image processing unit 201 includes a reference workpiece recognition unit 202 and a workpiece recognition unit 204.

  The image acquisition unit 200 acquires an image captured by the first imaging unit 30 (hereinafter referred to as a first image) and an image captured by the second imaging unit 40 (hereinafter referred to as a second image). The first image and the second image acquired by the image acquisition unit 200 are output to the image processing unit 201.

  The image processing unit 201 recognizes the reference workpiece A and the workpiece B from the first image and the second image output from the image acquisition unit 200, and recognizes the target position A1 and the trajectory point B1.

  Here, when the reference workpiece A and the workpiece B (target position A1 and trajectory point B1) can be recognized from the first image and the second image, the lenses and the reference workpiece A of the first imaging unit 30 and the second imaging unit 40 are used. And other objects (reference workpiece A, workpiece B, arm 11, etc.) are not included on the line connecting workpiece B (target position A1 and orbit point B1). In this case, the reference workpiece A and workpiece B (target position A1 and trajectory point B1) do not overlap with other objects as viewed from the lens, and the reference workpiece A and workpiece B (target Position A1 and trajectory point B1) are included.

  When the reference workpiece A and workpiece B (target position A1 and trajectory point B1) cannot be recognized from the first image and the second image, the lenses of the first imaging unit 30 and the second imaging unit 40, the reference workpiece A and the workpiece B are used. This is a case where another object is included on the line connecting (target position A1 and orbit point B1), that is, when there is another object closer to the lens than reference work A and work B (target position A1 and orbit point B1). . Specifically, when any of the arm 11, hand 14, and work B is on the lens side from the reference work A, or when any of the arm 11, hand 14, and reference work A is on the lens side from the work B. is there. In this case, the reference workpiece A and workpiece B (target position A1 and trajectory point B1) overlap with other objects as viewed from the lens. Accordingly, in the first image and the second image, the reference workpiece A and the workpiece B (target position A1 and trajectory point B1) are covered with other objects and cannot be visually recognized.

  Note that the “lens” here is assumed to be a lens closest to the object to be photographed (generally an objective lens), but all lenses constituting the imaging unit are included in the object. In addition, when the 1st imaging part 30 and the 2nd imaging part 40 employ | adopt a bending | flexion optical system, the lens of the objective lens side (a side far from an image pick-up element) is contained in the object from the optical element which bends an optical axis. . Further, when a cover glass is provided on the front surface of the objective lens, the cover glass is also included in the object.

  In addition, the case where the reference workpiece A and the workpiece B (target position A1 and trajectory point B1) overlap with other objects as viewed from the lens includes the case where all overlap and the case where a portion overlaps. . Moreover, when all overlap, the case where it overlaps exactly and the case where it hides behind are included.

  The reference workpiece recognition unit 202 recognizes image information from which the reference workpiece A and the target position A1 can be calculated from the first image and the second image output from the image acquisition unit 200.

  3A and 3B show examples of the first image. FIG. 3A shows a case where image information that can calculate the target position A1 can be detected, and FIG. 3B shows a case that image information that can calculate the target position A1 cannot be detected.

  In the present embodiment, as shown in FIG. 1, the target position A1 is the bottom of the hole and cannot be visually recognized directly in the image. However, since the depth of the hole is known (for example, stored in the memory 22 (described later) or the like), if the position of the hole opening A2 (corresponding to the first position) is known, the target position A1 is It can be calculated. Therefore, the reference workpiece recognition unit 202 detects an image A3 (see FIG. 3A) including the opening A2 from the first image and the second image as image information that can calculate the target position A1. Then, the reference workpiece recognition unit 202 recognizes the opening A2 from the image A3.

  As shown in FIG. 3A, when the opening A2 is visible, the image A3 can be detected. On the other hand, as shown in FIG. 3B, when the opening A2 is covered with another member (here, work B) and the opening A2 cannot be visually recognized, the image A3 cannot be detected. Therefore, it is desirable that the image A3 is a minimum range image including the opening A2.

  Note that the image recognition processing performed by the reference workpiece recognition unit 202 is not particularly limited, and various known techniques can be used, and thus description thereof is omitted.

  When the opening A2 is recognized from the first image and the second image, the reference workpiece recognition unit 202 outputs information indicating the position of the opening A2 to the target position calculation unit 203. In addition, when the opening A2 is not recognized from the first image or the second image, the reference workpiece recognition unit 202 outputs nothing to the target position calculation unit 203.

  The target position calculation unit 203 detects the opening A2 from the image A3 when the target position A1 can be calculated. Further, the target position calculation unit 203 is information indicating where the target position A1 is based on the detected position of the opening A2 and information on the depth of the hole (hereinafter referred to as position information of the target position A1). Is calculated. The calculated position information of the target position A1 is output to the control unit 205.

  In the present embodiment, since the hole has a circular cross section, the target position calculation unit 203 moves in the direction toward the inside of the reference workpiece A by the depth of the hole from the center position of the opening A2 (see FIGS. 1 and 3). In this case, the position moved downward) is calculated as the position of the target position A1.

  In the present embodiment, the target position A1 is calculated from the opening A2 of the circular hole and the depth of the hole. However, the target position is not limited to the bottom of the hole, and various forms are conceivable. However, the target position cannot be visually recognized in any form. Therefore, in the present invention, the target position is calculated from the position serving as the index exposed on the surface of the workpiece and the distance between the previously known index and the target position (corresponding to a predetermined distance). Further, the shape of the opening A2 is not limited to this, and various forms are conceivable. However, the opening A2 remains the same at any position on the surface of the reference workpiece A. Further, the target position A1 is not changed from an arbitrary position on the surface of the reference workpiece A to a position inside the reference workpiece A by a distance between an arbitrary position on the surface of the reference workpiece A and the target position.

  The workpiece recognition unit 204 detects the workpiece B and the trajectory point B1 from the first image and the second image output from the image acquisition unit 200. In the present embodiment, the trajectory point B1 is the tip of the work B. Therefore, as shown in FIG. 3A, the workpiece recognition unit 204 detects an image B2 including the tip of the workpiece B (orbit point B1) from the first image and the second image. Then, the workpiece recognition unit 204 recognizes the trajectory point B1 from the image B2.

  In the present embodiment, the workpiece B has a cylindrical shape. Therefore, the workpiece recognition unit 204 sets the center position in the radial direction of the tip as the trajectory point B1. Note that the cross-sectional shape of the workpiece B is not limited to a circle, and may be any shape as long as it is the same shape as the opening A2 of the reference workpiece A. Further, depending on the shape of the workpiece B, the position of the trajectory point B1 can be various positions. However, the trajectory point B1 is not changed at an arbitrary position on the surface of the workpiece B.

  As shown in FIG. 3A, when the trajectory point B1 is visible, the image B2 can be detected. On the other hand, as shown in FIG. 3B, when the trajectory point B1 is covered with another member (here, the reference workpiece A) and the trajectory point B1 cannot be visually recognized, the image B2 cannot be detected. Note that the image B2 is desirably a minimum range image including the trajectory point B1.

  When the image recognition unit 204 can recognize the image B2 from the first image and the second image, the workpiece recognition unit 204 detects the trajectory point B1 from the image B2. Then, the workpiece recognition unit 204 outputs information indicating the position of the trajectory point B1 to the control unit 205. The workpiece recognition unit 204 outputs nothing to the control unit 205 when the image B2 cannot be recognized from the first image or the second image.

  Note that the image recognition processing performed by the workpiece recognition unit 204 is not particularly limited, and various known techniques can be used, and thus description thereof is omitted.

  Based on the information output from the reference workpiece recognition unit 202, the target position calculation unit 203, and the workpiece recognition unit 204, the control unit 205 can recognize the opening A2 and the trajectory point B1, and if not, The arm 11 and the hand 14 are controlled using different control methods. Specifically, the control unit 205 determines whether or not the opening A2 and the trajectory point B1 can be recognized based on outputs from the reference workpiece recognition unit 202 and the workpiece recognition unit 204. Further, the control unit 205 switches between using visual servo or impedance control based on the determination result. Then, the control unit 205 moves the work B gripped by the hand 14 so as to bring the trajectory point B1 closer to the target position output from the target position calculation unit 203 so that the trajectory point B1 matches the target position. .

  The visual servo is a control method for tracking a target by measuring a change in position relative to the target as visual information and using it as feedback information. In this embodiment, a position-based method for controlling the robot based on the three-dimensional position information of the target calculated using a method such as stereo vision is adopted.

  In impedance control, mechanical impedance (inertia, damping coefficient, rigidity) generated when force is applied to the hand of the robot (hand 14) from the outside is set to a value convenient for the intended work. This is a method for controlling the position and force of the robot. For example, when the attenuation coefficient is set low, the robot moves easily with a light force, but is easy to move. However, the stability is low, and the characteristics are not suitable for precise operations such as positioning. Here, low stability means that it is difficult to stop at the target position after movement, or that the robot comes into contact with another object with a strong force (a force that may cause destruction). On the other hand, when the damping coefficient is set high, the resistance increases and the movement becomes difficult, but the stability increases and the characteristics are suitable for precise operations such as positioning. Here, high stability means that it is easy to stop at the target position after movement, and that the robot comes into contact with an appropriate contact strength even when it comes into contact with another object.

  When using impedance control, by setting the mechanical impedance (softness) to an appropriate target value, in the same direction as the external force, the appropriate softness (movement speed, stability) according to the external force received by the manipulator Etc.). Thereby, it is possible to prevent the workpiece and the robot itself from being destroyed, or to reduce the possibility of destroying the workpiece and the robot itself. In the present embodiment, an active impedance method is employed in which the impedance is changed by feedback control using measured values such as the position, speed, and force of the hand.

  Note that visual servo and impedance control are already known control methods, and thus detailed description thereof is omitted.

  In the case of visual servo, the control unit 205 calculates the movement amount and movement direction of the trajectory point B1 so that the trajectory point B1 is moved by the virtual displacement V (corresponding to the first amount). In the case of impedance control, the control unit 205 calculates the movement amount and the movement direction of the trajectory point B1 so that the trajectory point B1 is moved by the virtual displacement I (corresponding to the second amount). The virtual displacements V and I are displacements of the trajectory point B1, that is, the hand 14. The virtual displacement I is set smaller than the virtual displacement V. Details of processing performed by the control unit 205 will be described later.

  The servo system 206 outputs information to the motion control unit 101 so as to control the arm 11 based on the information output from the control unit 205. As a result, the position and orientation of the hand 14 can be changed.

  In addition, when the control unit 205 uses impedance control, the servo system 206 outputs the trajectory point B <b> 1 by the control of the arm 11 by the operation control unit 101, that is, the movement amount of each directional component of the hand 14 to the control unit 205. . Further, when the work is not finished, the servo system 206 outputs an instruction to the first imaging unit 30 and the second imaging unit 40 so as to capture the first image and the second image. Details of processing performed by the servo system 206 will be described later.

  FIG. 4 is a block diagram illustrating an example of a schematic configuration of the robot control unit 20. As shown in the figure, a robot control unit 20 composed of, for example, a computer includes a CPU (Central Processing Unit) 21 that is an arithmetic device, a RAM (Random Access Memory) that is a volatile storage device, and a nonvolatile storage device. A memory 22 including a ROM (Read only Memory), an external storage device 23, a communication device 24 for communicating with an external device such as the robot 10, an input device 25 such as a mouse and a keyboard, and an output such as a display. The apparatus 26 and the interface (I / F) 27 which connects the robot control part 20 and another unit are provided.

  Each functional unit described above is realized, for example, when the CPU 21 reads a predetermined program stored in the memory 22 or the like and executes the program. The predetermined program may be installed in advance in the memory 22 or the like, or may be downloaded from the network via the communication device 24 and installed or updated.

  The configuration of the robot system 1 described above is not limited to the above-described configuration because the main configuration has been described in describing the features of the present embodiment. For example, the robot 10 may include the robot control unit 20, the first imaging unit 30, and the second imaging unit 40. Further, the configuration of a general robot system is not excluded.

  Next, a characteristic process of the robot system 1 having the above configuration in the present embodiment will be described.

  FIG. 5 is a flowchart showing a flow of control processing of the arm 11 and the hand 14 of the present invention. This process is started at an arbitrary timing, for example, when a visual servo start instruction is input via a button (not shown) or the like. In the present embodiment, a description will be given by taking as an example a press-fitting operation in which the robot 10 press-fits a pin, which is the work B, into a hole of the reference work A.

  The image acquisition unit 200 acquires the first image and the second image from the first imaging unit 30 and the second imaging unit 40 (step S100). The acquired image data is output to the reference workpiece recognition unit 202 and the workpiece recognition unit 204.

  The reference workpiece recognition unit 202 recognizes the opening A2 of the reference workpiece A from the first image and the second image (step S102). Then, the reference workpiece recognition unit 202 determines whether or not the opening A2 has been recognized (step S104).

  If it is determined that the opening A2 can be recognized (YES in step S104), the reference workpiece recognition unit 202 outputs information indicating the position of the opening A2 to the target position calculation unit 203. The target position calculation unit 203 updates the position information of the target position A1 based on the acquired information (Step S106).

  The workpiece recognition unit 204 recognizes the trajectory point B1 of the workpiece B from the first image and the second image (step S108). Then, the workpiece recognition unit 204 determines whether or not the trajectory point B1 has been recognized (step S110).

  When the trajectory point B1 can be recognized (YES in step S110), the workpiece recognition unit 204 outputs information indicating the position of the trajectory point B1 to the control unit 205. Further, the control unit 205 controls the arm 11 by visual servo (step S112). Hereinafter, a method for controlling the arm 11 by the visual servo will be described in detail.

  Based on the information output from the workpiece recognition unit 204, the control unit 205 calculates information indicating the position of the orbit point B1 (hereinafter referred to as position information of the orbit point B1). Then, the control unit 205 calculates the virtual displacement V and the moving direction of the orbit point B1 based on the calculated position information of the orbit point B1 and the position information of the target position A1 acquired from the target position calculation unit 203. To do. The control unit 205 outputs, to the servo system 206, information indicating that the trajectory point B1, that is, the hand 14 is moved in the movement direction calculated by the virtual displacement V by visual servo.

  The servo system 206 obtains the trajectory point B1, that is, the moving direction and the moving amount of the hand 14, based on the information output from the control unit 205. Specifically, the movement amount in the x direction, the movement amount in the y direction, and the movement amount in the z direction are calculated by decomposing the virtual displacement V into each direction (x direction, y direction, z direction) component. Then, the servo system 206 outputs an instruction to the motion control unit 101 so as to move the hand 14 in the x direction, the y direction, and the z direction by the calculated movement amount by the visual servo.

  The operation control unit 101 moves the arm 11 based on the instruction output from the servo system 206. Thereby, the step (step S112) of controlling the arm 11 by the visual servo is completed.

  When the trajectory point B1 cannot be recognized (NO in step S110), the control unit 205 controls the arm 11 by impedance control (step S114). Hereinafter, a method for controlling the arm 11 by impedance control will be described in detail.

  The control unit 205 calculates position information of the trajectory point B1 based on information output from the workpiece recognition unit 204 and the servo system 206. Specifically, the control unit 205 moves the amount of movement of each directional component of the hand 14 output from the servo system 206 to the information acquired last from the workpiece recognition unit 204, that is, the position of the trajectory point B1 recognized last. Is added to calculate the position information of the current trajectory point B1. Based on the calculated position information of the trajectory point B1 and the position information of the target position A1 acquired from the target position calculation section 203, the control unit 205 moves the trajectory point B1, that is, the virtual displacement I of the hand 14 and the movement. Calculate the direction.

  The impedance control is performed when the trajectory point B1 (or the opening A2) cannot be detected from the image. Therefore, there is a possibility that the reference workpiece A and the workpiece B are colliding. Therefore, the virtual displacement I at the time of impedance control is set smaller than the virtual displacement V in order to perform work more safely.

  The control unit 205 outputs information indicating that the hand 14 is moved in the moving direction calculated by the virtual displacement I by impedance control to the servo system 206.

  The servo system 206 obtains the moving direction and moving amount of the hand 14 based on the information acquired from the control unit 205. Specifically, the servo system 206 decomposes the virtual displacement I into each direction (x direction, y direction, z direction) component, thereby moving the x direction movement amount, the y direction movement amount, and the z direction movement amount. Is calculated. Then, the servo system 206 outputs an instruction to the operation control unit 101 so as to move the hand 14 in the x direction, the y direction, and the z direction by the calculated movement amount by impedance control.

  The operation control unit 101 moves the arm 11 based on the instruction output from the servo system 206. Thereby, the step (step S114) of controlling the arm 11 by impedance control is completed.

  In this embodiment, since the shaft is press-fitted, there is a high possibility that an external force is applied to the workpiece B. However, by performing impedance control, the workpiece B can be reliably moved by the virtual displacement I even when an external force is applied.

  If the opening A2 cannot be recognized (NO in step S104), no information is input from the reference workpiece recognition unit 202 to the target position calculation unit 203, and thus the target position calculation unit 203 cannot calculate the target position A1. The position information of the target position A1 calculated immediately before is output to the control unit 205. Thereafter, the process proceeds to step S109.

  The workpiece recognition unit 204 recognizes the trajectory point B1 of the workpiece B from the first image and the second image (step S109). This process is the same as the process of step S108. If the trajectory point B1 can be recognized, the workpiece recognition unit 204 outputs information indicating the position of the trajectory point B1 to the control unit 205. The control unit 205 controls the movable unit by impedance control (step S114). In this case, since the position information of the target position A1 is not known from the image, impedance control is performed regardless of whether or not the trajectory point B1 has been recognized. Since the process of step S114 has already been described, the description thereof is omitted here.

  Thus, in the present embodiment, impedance control is performed when either the opening A2 or the trajectory point B1 is not detected from the image. Impedance control is also performed when both the opening A2 and the trajectory point B1 are not detected from the image. In the impedance control, since the impedance is set, even if the reference workpiece A and the workpiece B collide, the workpiece B cannot be forcibly moved in the set direction, so that the reference workpiece A and the workpiece B are destroyed. So you can continue working safely.

  After the trajectory point B1, that is, the hand 14 is moved by the virtual displacements V and I by the visual servo or impedance control (steps S112 and S114), the servo system 206 detects the position information of the trajectory point B1 after the movement and the target position A1. It is determined whether or not the position information of the orbital point B1 has reached the target position A1 (step S116).

  When the trajectory point B1 has not reached the target position A1 (NO in step S116), the servo system 206 outputs an instruction to the first imaging unit 30 and the second imaging unit 40. Then, the first imaging unit 30 and the second imaging unit 40 capture the first image and the second image, and the image acquisition unit 200 acquires the first image and the second image (step S100). Thereby, steps S100 to S116 are repeatedly performed. If the trajectory point B1 has reached the target position A1 (YES in step S116), the process ends.

  As described above, when the flow shown in FIG. 5 is repeatedly performed and the target position A1 and the trajectory point B1 are not visible from the image, the visual servo is switched to the impedance control. Further, when the target position A1 or the trajectory point B1 is returned from a state where the target position A1 or the trajectory point B1 is not visible, the impedance control is switched to the visual servo.

  According to the present embodiment, impedance control is performed when a workpiece or the like cannot be recognized from the image, and therefore the work can be safely continued even when the workpiece or the like cannot be recognized from the image.

  Further, according to the present embodiment, when the trajectory point and the target position can be recognized from the image, the visual servo is used, so that the work can be performed even if the position of the reference work A is different every time the work is performed. it can. Therefore, it is not necessary to install the reference workpiece A at the same position every time, and work efficiency can be increased.

  Furthermore, according to the present embodiment, regardless of what control method is used, the work gripped by the hand can be moved to a target position inside the work that cannot be visually recognized.

  In the present embodiment, as an example of switching between visual servo and impedance control, the shaft press-fitting operation has been described. However, the case where switching between visual servo and impedance control is effective is not limited thereto. For example, any work that combines a plurality of parts, such as a work for inserting a connector and a work for attaching a lid, can be applied to various work. Further, in this embodiment, a hand is used as an end effector, and a work is gripped by the hand, but various forms of the end effector can be considered depending on the type of work. Moreover, although the hand has gripped the workpiece, the workpiece can be held in various forms depending on the type of the end effector.

  Moreover, in this embodiment, the case where opening A2 is covered with the workpiece | work B is shown as a case where opening A2 cannot be visually recognized, and the case where track point B1 is covered with the reference | standard workpiece A as a case where track point B1 cannot be visually recognized. Although shown (refer FIG. 3 (B)), when the opening part A2 and the track point B1 cannot be visually recognized, it is not restricted to this. As a case where the opening A2 and the trajectory point B1 cannot be visually recognized, there may be a case where the opening A2 and the trajectory point B1 are physically covered by other parts such as the arm 11 or the like. In addition, when the opening A2 and the orbit point B1 cannot be visually recognized, even if the opening A2 and the orbit point B1 are not physically covered, the opening A2 and the orbit point B1 are recognized by reflection of metallic luster or the like. There may be a case where it cannot be recognized by processing. Furthermore, as the case where the opening A2 and the trajectory point B1 cannot be visually recognized, the opening A2 and the trajectory point B1 are not in the range assumed by the first image and the second image (for example, a predetermined range in the central portion) There may be a case where the opening A2 and the trajectory point B1 are not included in the first image and the second image. Unlike the above case, even if the opening A2 and the trajectory point B1 can be recognized, the positional relationship between the opening A2 and the trajectory point B1 is not in the assumed positional relationship (for example, the workpiece B is a reference) The case in which the opening A2 and the trajectory point B1 are not visible may be included.

  In the present embodiment, the imaging instruction is output from the servo system 206 to the first imaging unit 30 and the second imaging unit 40. However, the present invention is not limited to this. For example, the imaging instruction is output to the image acquisition unit 200. Then, an instruction may be output from the image acquisition unit 200 to the first imaging unit 30 and the second imaging unit 40.

  In this embodiment, processing (control of the arm 11, movement of the hand 14, etc.) is performed based on some information such as force, value, position information, imaging result, recognition result, judgment result, etc. Processing based on information means processing using information itself, processing using results obtained by processing information, processing by extracting part of information, etc. including.

  For example, when processing is performed based on the imaging result, the processing may be performed using an image obtained by imaging, or the result obtained by performing arbitrary image processing on the image may be used. Processing may be performed, luminance and color data included in the image may be extracted and processed, or only a partial area of the image may be extracted and processed. Or the like. The results obtained by performing arbitrary image processing on the image include, for example, position information (coordinates), size information, shape information, positional relationship information, information indicating the presence or absence of an object, etc. Is included.

  In addition, for example, when processing is performed based on force, processing may be performed using output values from sensors, measurement values from actuators, etc. (hereinafter referred to as measurement values) as they are, May be processed using the calculated value, or may be processed using the result of combining the measured value and the value calculated using the measured value. You may use it for processing.

<Second Embodiment>
In the first embodiment of the present invention, the workpiece B is press-fitted into the reference workpiece A placed on a plane such as a floor or a stand using a one-arm robot, but the scope of application of the present invention is not limited to this.

  The second embodiment of the present invention is a form in which the present invention is applied to a double-arm robot. Hereinafter, a second embodiment of the present invention will be described. Since the processing flow is the same as that of the first embodiment, only the configuration will be described. Further, the same portions as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  FIG. 6 is a system configuration diagram showing an example of the configuration of the robot system 2 in one embodiment of the present invention. The robot system 2 in the present embodiment mainly includes a robot 10A, a robot control unit 20, a first imaging unit 30, and a second imaging unit 40.

  The robot 10A is an arm type robot having a plurality of arms 11 (two in this embodiment) including a plurality of joints (joints) 12 and a plurality of links 13, that is, a so-called double-arm robot. A hand 14 (a so-called end effector) is provided at the tip of each of the two arms 11.

  One of the two hands 14 holds the workpiece B, and the other one holds the reference workpiece A. Which one of the two hands 14 holds which workpiece (reference workpiece A or workpiece B) is arbitrary. As in the first embodiment, the workpiece on which the workpiece B is inserted is referred to as the reference workpiece A for convenience, but the workpiece B may be used as the reference workpiece. Further, the expression “reference” does not relate to the shape, size, position, etc. of the workpiece.

  As described above, the present invention is not limited to the case where other workpieces are inserted into the placed workpiece, but is also applied to a case where a plurality of movable parts each grip the workpiece and combine the gripped workpieces. it can. For example, the present invention can be applied to a case where gears are arranged so that a plurality of gears mesh appropriately and a plurality of objects are aligned and placed. Furthermore, the present invention can also be applied to a case where a device for discharging adhesive, paste-like solder, or the like is held by a hand and the adhesive, paste-like solder, or the like is discharged to an arbitrary position.

<Other variations>
In the first and second embodiments, visual servo and impedance control are switched. However, switching to visual servo is not limited to impedance control. For example, the following control method is also conceivable.

  Consider the press-fitting of the shaft used in the first and second embodiments, and the insertion operation into a hole in a round bar without friction and clearance. When the axis parallel to the insertion direction is the z-axis and the x-axis and y-axis are set on a plane perpendicular to the z-axis, in the above operation, the movement around the x- and y-axes and the x- and y-axes and the z-axis Direction and generation of force around the z-axis are constrained (natural constraint). Therefore, in order to achieve the work, it is necessary to set the x, y axis direction and the allowable interference force around x, y, the insertion speed in the z axis direction, and the allowable rotation speed (artificial constraint) around the z axis. Therefore, when performing press-fitting and insertion work, the work is effectively executed by performing compliance control having a compliance (adaptation) function for correcting the movement of the end effector according to the interference force based on the constraint condition. be able to.

  In the compliance control, the end effector is displaced according to the force applied to the end effector. For example, when performing the work of inserting a pin into a hole, it is pressed with a constant force in the insertion direction (the direction in which the end effector is moved, the axial direction of the hole), and no external force is applied in directions other than the insertion direction. Set the pressing force for each direction. When the hole and the pin are in contact, an external force is applied in a direction other than the insertion direction, so the end effector is displaced in the direction of the force received by the end effector so that the hole and the pin do not contact. Familiarize with the pins. Therefore, by using compliance control, even when the axis of the hole and the axis of the pin are deviated, the pin can be inserted into the hole. Thereby, it is possible to prevent the robot, the pin, the hole and the like from being destroyed, or to reduce the possibility of destroying the robot, the pin, the hole and the like.

  In addition, when assembling, polishing, deburring, cranking, etc. on the arm 11 and the hand 14 etc., not only the position of the end effector such as the hand 14 but also the force applied to the object by the robot is controlled. It is necessary to do. Impedance control is a method for adjusting the relationship (impedance) between the interference force acting on the end effector and the movement of the end effector, and is therefore effective for such work. In addition to this, even when using hybrid control that determines the direction to control the position and the direction to control the force and applies multiple control methods so that the position and force in each direction become the desired value. It is also effective for work.

  In other words, when the reference workpiece A or workpiece B cannot be recognized from the image, even if the visual servo is switched to a control method capable of skillfully controlling the interference force such as hybrid control and compliance control, the work can be performed safely. Thus, the effect of the present invention can be achieved.

  In order to perform a control method capable of skillfully controlling the interference force such as hybrid control and compliance control, it is preferable to detect the force applied to the end effector such as the hand 14. The force applied to the end effector can be directly measured by a force sensor or the like, as in the first and second embodiments, and the external force exerted on the end effector is estimated from each axis torque value of the arm 11. You can also. Therefore, the control method capable of controlling the interference force can be executed if the arm 11 or the like has means for acquiring a force applied to the end effector directly or indirectly.

  In the first and second embodiments, visual servo and impedance control are switched. However, the scope of the present invention is not limited to switching between visual servo and impedance control. Visual servo is performed when the orbit point and the target position can be visually recognized from the first image and the second image, and impedance control may be performed when the orbit point and the target position cannot be visually recognized from the first image and the second image. For example, when the trajectory point and the target position can be visually recognized from the first image and the second image, a control method for performing both visual servo and impedance control is performed, and the trajectory point and the target position are visually recognized from the first image and the second image. If it becomes impossible, the visual servo may be stopped and only impedance control may be performed. By using a control method that performs both visual servoing and impedance control, the movable part can be moved more greatly.

  In the first and second embodiments, two imaging units, the first imaging unit 30 and the second imaging unit 40, are provided. However, the number of imaging units may be one, or three or more. . For example, when there is only one imaging unit, it is difficult to recognize the position in the depth direction when viewed from the imaging unit. However, even when there is only one imaging unit, the position in the depth direction can be determined by setting a virtual constraint plane. Further, when there is only one imaging unit, impedance control, compliance control, and the like may be performed by setting a physical constraint surface.

  In the first and second embodiments, since the first imaging unit 30 and the second imaging unit 40 are provided, the opening A2 and the trajectory point B1 are visible from either the first image or the second image. In the case where the opening A2 and the trajectory point B1 cannot be visually recognized, the opening A2 and the trajectory point can be viewed only when the opening A2 and the trajectory point B1 are not visible from both the first image and the second image. It is good also as a case where B1 cannot be visually recognized. However, in order to improve the safety of work, when the opening A2 and the trajectory point B1 cannot be visually recognized from either the first image or the second image, the opening A2 or the trajectory point B1 cannot be visually recognized. It is desirable.

  In the first and second embodiments, the two image capturing units, the first image capturing unit 30 and the second image capturing unit 40, are provided on a work table (may be a floor) (see FIGS. 1 and 6). However, the position where the imaging unit is provided is not limited to the workbench. The imaging unit may be provided on the ceiling or may be provided on the robot.

  7 and 8 are forms in which the imaging unit is provided on the ceiling. FIG. 7 shows a production system in which the robot 10A and an operator work on the same product in order. FIG. 8 shows a cell production system in which the robot 10A performs work in one production cell. As illustrated in FIGS. 7 and 8, the third imaging unit 50 is provided on the ceiling so that an image in which the work area of the robot 10 </ b> A is captured can be captured. 7 and 8, the illustration of the ceiling is omitted.

  In FIG. 7, one third imaging unit 50 is provided for one robot 10A. On the other hand, in FIG. 8, the work area of the robot 10A is large, and one third image pickup unit 30 cannot cover the entire work area, so that three third image pickup parts 50 are provided for one robot 10A. . In the case where a plurality of third imaging units 50 are provided, it is desirable to provide a part of the angle of view so that information in the depth direction can be obtained for a region where the angle of view overlaps.

  Also, as shown in FIG. 7, when one third imaging unit 50 is provided for one robot 10A, information in the depth direction of the image cannot be obtained. Therefore, when information in the depth direction of the image is required, the first imaging unit 30 and the second imaging unit 40 may be provided together with the third imaging unit 50 for one robot 10A. A plurality of three imaging units 50 may be provided.

  9, 10 and 11 are forms in which an imaging unit is provided in the robot itself. FIG. 9 is a form in which the imaging units 15 and 16 are provided in a portion corresponding to the head of the robot 10B. In FIG. 9, since the imaging units 15 and 16 are provided, information in the depth direction of the image can be obtained. However, in the form in which the imaging unit is provided in the portion corresponding to the head of the robot, the number of imaging units is not limited to two, and may be one or three or more.

  10 and 11 are forms in which the imaging unit is provided at the tip of the arm 11. In FIG. 10, the imaging unit 17 is provided at the tip of the arm 11 so that the axis 17 a of the imaging unit 17 and the axis 11 a of the arm 11 are parallel (including a case where they are slightly shifted). In FIG. 11, the imaging unit 18 is provided at the tip of the arm 11 so that the axis 18 a of the imaging unit 18 and the axis 11 a of the arm 11 are orthogonal (including a case where they are slightly shifted). In the form shown in FIG. 11, if the workpiece or the like is gripped by the arm 11 provided with the imaging unit 18, the gripped workpiece or the like can be photographed by the imaging unit 18 because the shaft 18 a and the shaft 11 a are not parallel. Therefore, visual servo cannot be performed. However, if the arm 11 that is not gripping a workpiece or the like has the imaging unit 18, visual servoing can be performed.

  The third imaging unit 50 and the imaging units 15, 16, 17, and 18 have the same functions and configurations as the first imaging unit 30 and the second imaging unit 40, and thus description thereof is omitted.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiment. In addition, it is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention. In particular, in the first and second embodiments, a case where a robot system in which a robot and a robot control unit are separately provided is illustrated. However, as a robot system in which a robot and a robot control unit are separately provided, You may provide as a robot by which the robot control part etc. were included in the robot, and you may provide as a robot control apparatus which consists only of a robot control part, or a robot control part and an imaging part. The present invention can also be provided as a program for controlling a robot or the like or a storage medium storing the program.

  And when providing as a robot control part, 1. 1. When an imaging unit is included in the robot controller, When the imaging unit is not included in the robot control device, the following two methods are included in the technical scope of the present invention. In the case of providing as a robot system and a robot, 1. When the robot includes an imaging unit and a robot control unit 2. When the imaging unit is included in the robot and the robot control unit is not included 3. When the robot includes a robot control unit and does not include an imaging unit. When the robot does not include the imaging unit and the robot control unit and the imaging unit and the robot control unit are included in different housings, or when the imaging unit and the robot control unit are included in the same housing, the four types of the present invention are included. Is included in the technical scope.

DESCRIPTION OF SYMBOLS 1, 2 ... Robot system 10, 10A, 10B ... Robot, 11 ... Arm, 12 ... Joint, 13 ... Link, 14 ... Hand, 15, 16, 17, 18 ... Imaging part, 20 ... Robot control part, 21 ... CPU, 22 ... Memory, 23 ... External storage device, 24 ... Communication device, 25 ... Input device, 26 ... Output device, 27 ... I / F, 30 ... First imaging unit, 40 ... Second imaging unit, 50 ... First 3 imaging units, 101 ... operation control unit, 200 ... image acquisition unit, 201 ... image processing unit, 202 ... reference workpiece recognition unit, 203 ... target position calculation unit, 204 ... workpiece recognition unit, 205 ... control unit, 206 ... servo system

Claims (31)

A first step of acquiring an image including a first object and a second object held by the end effector;
A second step of recognizing a first position, which is an arbitrary position on the surface of the first object, and a second position, which is an arbitrary position on the surface of the second object, from the acquired image;
A third step of determining whether or not the first position and the second position can be recognized in the second step;
Based on the determination result of the third step, the second position is moved from the first position to the inside of the first object by a predetermined distance using a different control method. A fourth step in which the end effector controls a movable part provided at the tip so that the end effector approaches,
In the fourth step, when the first position and the second position are recognized, the movable unit is controlled based on the recognition result of the first position and the second position, and When at least one of the first position and the second position cannot be recognized, a force applied to the end effector is acquired, and the movable unit is controlled based on the acquired force applied to the end effector. A robot control method characterized by the above. The robot control method according to claim 1,
In the fourth step, when at least one of the first position and the second position cannot be recognized, the movable unit is controlled by impedance control. The robot control method according to claim 1 or 2,
In the fourth step, when the first position and the second position are recognized, the movable portion is controlled so that the second position moves by a first amount, and the first position When at least one of the second position and the second position cannot be recognized, the movable portion is controlled so that the second position moves by a second amount smaller than the first amount. A robot control method. In the robot control method according to any one of claims 1 to 3,
In the third step, when at least one of the first position and the second position cannot be recognized, at least one of the first position and the second position is included in the image. If at least one of the first position and the second position is not included in an arbitrary range of the image, at least one of the first position and the second position is image processing. The robot control method is characterized in that the robot control method is at least one of the case where the position relationship is not recognized and the case where the positional relationship between the first position and the second position is outside a set range. In the robot control method according to any one of claims 1 to 4,
In the first step, the step of acquiring the image acquires two images obtained by capturing the first object and the second object from different directions,
In the fourth step, when at least one of the first position and the second position cannot be recognized from at least one of the two images, the first position and the second position The robot control method, wherein the movable part is controlled as being when at least one of the positions cannot be recognized. In the robot control method according to any one of claims 1 to 5,
In the fourth step, the third position is calculated by moving from the second position in a direction toward the inside of the second object by the predetermined distance, and the calculated third position The robot control method characterized by controlling the movable part so that the second position approaches. In the robot control method according to any one of claims 1 to 6,
In the fourth step, when the first position and the second position can be recognized, the second position is obtained from the recognition result, and at least the first position and the second position are determined. The robot control method characterized in that, when one of them is not recognized, the second position is obtained based on the amount of movement of the movable part. An image acquisition unit that acquires an image including a first object and a second object held by the end effector;
An image processing unit for recognizing a first position that is an arbitrary position on the surface of the first object and a second position that is an arbitrary position on the surface of the second object from the acquired image;
Based on the recognition result of the first position and the second position, the second position is separated from the first position to the inside of the first object by a predetermined distance using different control methods. The end effector is a control unit that controls the movable part provided at the tip so as to approach the third position, and when the first position and the second position can be recognized, the first effector The movable portion is controlled based on the recognition result of the position and the second position, and when at least one of the first position and the second position cannot be recognized, the movable portion is A control unit that acquires applied force and controls the movable unit based on the acquired force applied to the tip of the movable unit;
A robot control device comprising: An image acquisition unit for acquiring images;
An image processing unit for recognizing a first object and a second object held by the end effector from the image;
A control unit that controls the position of the end effector so as to bring the position of the second object closer to a target position that is separated from the position of the first object by a predetermined distance based on the recognition result of the image processing unit. And comprising
When the image processing unit is able to recognize the first object and the second object, the control unit is configured to use the end effector based on the position of the first object and the position of the second object. When the image processing unit cannot recognize at least one of the first object and the second object, a force applied to the end effector is acquired, and based on the acquired force A robot control apparatus for controlling a position of the end effector. A robot having a movable part and an end effector provided at the tip of the movable part;
An imaging unit that captures an image including a first object and a second object held by the movable unit;
An image processing unit for recognizing a first position that is an arbitrary position on the surface of the first object and a second position that is an arbitrary position on the surface of the second object;
Based on the recognition result of the first position and the second position, the second position is separated from the first position to the inside of the first object by a predetermined distance using different control methods. A control unit that controls the movable unit so as to approach the third position, and when the first position and the second position are recognized, the first position and the second position. Based on the recognition result, the movable part is controlled, and when at least one of the first position and the second position cannot be recognized, a force applied to the tip of the movable part is obtained, and the acquisition is performed. A control unit that controls the movable unit based on a force applied to the tip of the movable unit,
A robot system comprising: The robot system according to claim 10, wherein
The imaging unit includes two imaging devices that capture the first object and the second object from different directions,
The image processing unit recognizes the first position and the second position from two images picked up by each of the two image pickup devices, and picks up an image by at least one of the two image pickup devices. When at least one of the first position and the second position cannot be recognized from the captured image, it is assumed that at least one of the first position and the second position cannot be recognized. Characteristic robot system. A robot having an end effector;
An image acquisition unit for acquiring images;
An image processing unit for recognizing a first object and a second object held by the end effector from the image;
A control unit that controls the position of the end effector so as to bring the position of the second object closer to a target position that is separated from the position of the first object by a predetermined distance based on the recognition result of the image processing unit. And comprising
When the image processing unit is able to recognize the first object and the second object, the control unit is configured to use the end effector based on the position of the first object and the position of the second object. When the image processing unit cannot recognize at least one of the first object and the second object, a force applied to the end effector is acquired, and based on the acquired force A robot system for controlling a position of the end effector. A robot having a movable part with an end effector holding a first object provided at a tip;
An imaging unit having a lens;
A control unit that controls the movable unit to bring the first object closer to the second object based on the imaging result of the first object and the second object by the imaging unit;
When the second object, the end effector, or the movable part is on a line virtually connecting the lens and the first object, the control unit is based on a force applied to the end effector. The robot system characterized by controlling said movable part. A robot having a movable part with an end effector holding a first object provided at a tip;
An imaging unit having a lens;
A control unit that controls the movable unit to bring the first object closer to the second object based on the imaging result of the first object and the second object by the imaging unit;
The control unit is configured based on a force applied to the end effector when the first object, the end effector, or the movable unit is on a line virtually connecting the lens and the second object. The robot system characterized by controlling said movable part. The robot system according to claim 13 or 14,
When the control unit controls the movable unit based on a force applied to the end effector, the control unit moves in the direction of the force received by the end effector with a moving speed and stability according to the force received by the end effector. A robot system characterized by moving an end effector. The robot system according to claim 13 or 14,
When the control unit controls the movable unit based on the force applied to the end effector, the control unit receives the force received by the end effector so that the end effector does not receive any force other than the direction in which the end effector is moved. A robot system characterized by moving the end effector in a direction. An imaging unit;
A robot having an end effector for holding a first object;
A control unit that controls the position of the end effector so that the first object approaches the second object;
The control unit controls the end effector based on an imaging result of the imaging unit when the first object and the second object do not overlap in the imaging direction of the imaging unit, and the imaging unit When the first object and the second object overlap with each other in the imaging direction, the robot control system controls the end effector based on a force applied to the end effector. The robot system according to claim 17, wherein
When the control unit controls the movable unit based on a force applied to the end effector, the control unit moves in the direction of the force received by the end effector with a moving speed and stability according to the force received by the end effector. A robot system characterized by moving an end effector. The robot system according to claim 17, wherein
When the control unit controls the movable unit based on the force applied to the end effector, the control unit receives the force received by the end effector so that the end effector does not receive any force other than the direction in which the end effector is moved. A robot system characterized by moving the end effector in a direction. An imaging unit, and a robot having an end effector that holds the first object,
When the first object and the second object do not overlap in the imaging direction of the imaging unit, the step of bringing the first object closer to the second object based on the imaging result of the imaging unit;
Controlling the position of the end effector based on the force applied to the end effector when the first object and the second object overlap in the imaging direction of the imaging unit;
A robot system characterized by comprising: Moving parts;
An end effector provided at the tip of the movable part;
An image acquisition unit for acquiring an image including a first object and a second object held by the end effector;
An image processing unit for recognizing a first position that is an arbitrary position on the surface of the first object and a second position that is an arbitrary position on the surface of the second object;
Based on the recognition result of the first position and the second position, the second position is separated from the first position to the inside of the first object by a predetermined distance using different control methods. A control unit that controls the movable unit so as to approach the third position, and when the first position and the second position are recognized, the first position and the second position. Based on the recognition result, the movable part is controlled, and when at least one of the first position and the second position cannot be recognized, a force applied to the tip of the movable part is obtained, and the acquisition is performed. A control unit that controls the movable unit based on a force applied to the tip of the movable unit,
A robot characterized by comprising: The robot according to claim 21, wherein
Two movable parts are provided,
One of the two movable parts holds the second object, and the other one of the two movable parts holds the first object. An end effector,
An image acquisition unit for acquiring images;
An image processing unit for recognizing a first object and a second object held by the end effector from the image;
A control unit that controls the position of the end effector so as to bring the position of the second object closer to a target position that is separated from the position of the first object by a predetermined distance based on the recognition result of the image processing unit. And comprising
When the image processing unit is able to recognize the first object and the second object, the control unit is configured to use the end effector based on the position of the first object and the position of the second object. When the image processing unit cannot recognize at least one of the first object and the second object, a force applied to the end effector is acquired, and based on the acquired force A robot that controls a position of the end effector. An imaging unit having a lens;
A movable part provided with an end effector for holding the first object at the tip;
A control unit that controls the movable unit to bring the first object closer to the second object based on the imaging result of the first object and the second object by the imaging unit;
When the second object, the end effector, or the movable part is on a line virtually connecting the lens and the first object, the control unit is based on a force applied to the end effector. A robot characterized by controlling the movable part. An imaging unit having a lens;
A movable part provided with an end effector for holding the first object at the tip;
A control unit that controls the movable unit to bring the first object closer to the second object based on the imaging result of the first object and the second object by the imaging unit;
The control unit is configured based on a force applied to the end effector when the first object, the end effector, or the movable unit is on a line virtually connecting the lens and the second object. A robot characterized by controlling the movable part. The robot according to claim 24 or 25, wherein
When the control unit controls the movable unit based on a force applied to the end effector, the control unit moves in the direction of the force received by the end effector with a moving speed and stability according to the force received by the end effector. A robot characterized by moving an end effector. The robot according to claim 24 or 25, wherein
When the control unit controls the movable unit based on the force applied to the end effector, the control unit receives the force received by the end effector so that the end effector does not receive any force other than the direction in which the end effector is moved. A robot characterized by moving the end effector in a direction. An image acquisition unit that acquires an image captured by the imaging unit;
An end effector holding a first object;
A control unit that controls the position of the end effector so that the first object approaches the second object;
The control unit controls the end effector based on the acquired image when the first object and the second object do not overlap in the imaging direction of the imaging unit, and the imaging unit When the first object and the second object overlap with each other in a direction, the robot controls the end effector based on a force applied to the end effector. The robot according to claim 28, wherein
When the control unit controls the movable unit based on a force applied to the end effector, the control unit moves in the direction of the force received by the end effector with a moving speed and stability according to the force received by the end effector. A robot characterized by moving an end effector. The robot according to claim 28, wherein
When the control unit controls the movable unit based on the force applied to the end effector, the control unit receives the force received by the end effector so that the end effector does not receive any force other than the direction in which the end effector is moved. A robot characterized by moving the end effector in a direction. An image acquisition unit that acquires an image captured by the imaging unit; and an end effector that holds the first object;
A step of bringing the first object closer to the second object based on the acquired image when the first object and the second object do not overlap in the imaging direction of the imaging unit;
Controlling the position of the end effector based on the force applied to the end effector when the first object and the second object overlap in the imaging direction of the imaging unit;
A robot characterized by comprising:

Images