JP2015089598A - Teaching device, robot, robot system, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize stable visual servo control not influenced by working environment.SOLUTION: A teaching device includes: a first guide setting section for accepting a setting of a first guide with respect to image data of a first object; a second guide setting section for accepting a setting of a second guide with respect to image data of a second object; a display section for displaying the first guide and the second guide; a teaching data generation section for generating teaching data in which the first guide and the second guide indicate a predetermined positional relationship; and a teaching data transmission section for transmitting the generated teaching data to a robot performing visual servo control on the basis of the teaching data.

Description

  The present invention relates to a teaching device, a robot, a robot system, a method, and a program.

  Patent Literature 1 describes that visual servoing is performed based on a reference image with a marker (target image) and a captured image with a marker (current image).

  In the visual servo as described in Patent Document 1, the robot is designed so that the evaluation value of the difference between the target image including the target position and posture of the robot and the captured image including the current position and posture of the robot is small. Is controlled.

JP 2012-254518 A

  However, the visual servo as described above cannot accurately evaluate the difference between the target image and the captured image when the work environment differs between when the target image is taught and when the visual servo is executed. There is a problem that the robot cannot be controlled properly because the evaluation value of the above does not fall within a predetermined range. The case where the work environment is different is, for example, a case where the position or posture of the camera changes or the brightness of the illumination changes.

  Accordingly, an object of the present invention is to realize a stable visual servo that is not affected by the work environment.

  A first aspect of the present invention that solves the above-described problem is a teaching device, in which a first guide setting unit that receives setting of a first guide for image data of a first object, and a second object A second guide setting unit that receives setting of a second guide for image data, a display unit that displays the first guide and the second guide, and a predetermined positional relationship between the first guide and the second guide And a teaching data transmission unit that transmits the generated teaching data to a robot that performs visual servoing based on the teaching data. According to the first aspect of the present invention, guides are set for two objects, and for example, visual servo teaching data that brings these set guides close is sent to the robot. The robot can generate these guides for the captured image and perform visual servoing using these guides. Therefore, it is not necessary to evaluate the difference between the target image and the captured image, and a stable visual servo that is not affected by the work environment can be realized.

  The first guide setting unit receives the setting of the first guide in association with the feature of the image data of the first object, and the second guide setting unit is associated with the feature of the image data of the second object. Receiving the setting of the second guide, the teaching data generation unit, the feature of the image data of the first object and the first guide, the feature of the image data of the second object and the second guide May be generated. Thereby, the robot can generate a guide easily and accurately based on the feature of the image of the object included in the captured image.

  The first guide setting unit receives a plurality of settings of the first guide in association with a plurality of features of the image data of the first object, and the teaching data generation unit is configured to receive the image data of the first object. Teaching data indicating the plurality of features and the plurality of first guides may be generated. Thereby, since the robot can generate a plurality of guides for the object included in the captured image, it is possible to realize a stable visual servo that is not easily affected by changes in the work environment.

  The first guide setting unit extracts a plurality of features of image data of the first object using each of a plurality of image processes, and associates the plurality of first guides with each of the extracted features. You may make it accept the setting of. As a result, the robot can generate a plurality of guides by different image processing for the object included in the captured image, and thus can realize a stable visual servo that is not easily affected by changes in the work environment. .

  The first guide setting unit extracts features of the image data of the first object using predetermined image processing, and specifies a plurality of features and a plurality of the specified features from the extracted features. You may make it receive the setting of the said some 1st guide linked | related with each. As a result, the robot can generate a plurality of guides for the object included in the captured image by the same image processing, and thus can realize a stable visual servo that is not easily affected by changes in the work environment. .

  A second aspect of the present invention that solves the above problem is a teaching device, and a first guide setting unit that receives setting of a first guide for image data of a first object, and an imaging range of the imaging device On the other hand, the second guide setting unit for receiving the setting of the second guide, the display unit for displaying the first guide and the second guide, and the first guide and the second guide exhibit a predetermined positional relationship. A teaching data generation unit that generates teaching data; and a teaching data transmission unit that transmits the generated teaching data to a robot that performs visual servoing based on the teaching data. According to the second aspect of the present invention, a guide is set for one object, and a guide is set for the imaging range. For example, the teaching of the visual servo such that these set guides are brought close to each other. Data is sent to the robot. The robot can generate these guides for the captured image and perform visual servoing using these guides. Therefore, it is not necessary to evaluate the difference between the target image and the captured image, and a stable visual servo that is not affected by the work environment can be realized. In addition, since the guide for the imaging range is used, image processing for extracting features from the captured image is not necessary, and the processing load is reduced.

  A third aspect of the present invention that solves the above problem is a teaching method of a teaching device, in which a first guide setting step for receiving setting of a first guide for image data of a first object, A second guide setting step for receiving setting of a second guide for the image data of the object; a display step for displaying the first guide and the second guide; and the first guide and the second guide are predetermined. Teaching data generation step for generating teaching data indicating the positional relationship between the teaching data and the teaching data transmission step for transmitting the generated teaching data to a robot that performs visual servoing based on the teaching data. According to the third aspect of the present invention, guides are set for two objects, and for example, visual servo teaching data that brings these set guides close is sent to the robot. The robot can generate these guides for the captured image and perform visual servoing using these guides. Therefore, it is not necessary to evaluate the difference between the target image and the captured image, and a stable visual servo that is not affected by the work environment can be realized.

  A fourth aspect of the present invention that solves the above problem is a program, a first guide setting unit that receives setting of a first guide for image data of a first object, and an image of a second object A second guide setting unit that receives the setting of the second guide, a display unit that displays the first guide and the second guide, and the first guide and the second guide have a predetermined positional relationship with respect to the data. A computer is caused to function as a teaching data generation unit that generates teaching data to be shown and a teaching data transmission unit that transmits the generated teaching data to a robot that performs visual servoing based on the teaching data. According to the fourth aspect of the present invention, guides are set for two objects, and for example, visual servo teaching data that brings these set guides close is sent to the robot. The robot can generate these guides for the captured image and perform visual servoing using these guides. Therefore, it is not necessary to evaluate the difference between the target image and the captured image, and a stable visual servo that is not affected by the work environment can be realized.

  A fifth aspect of the present invention that solves the above problem is a robot, which is set for the first guide set for the image data of the first object and for the image data of the second object. A second guide includes a teaching data storage unit that stores teaching data indicating a predetermined positional relationship, an image acquisition unit that acquires a captured image including the first object and the second object, and the teaching data A first guide generating unit that generates the first guide for the image data of the first object included in the acquired captured image, and the acquired based on the teaching data. A second guide generating unit that generates the second guide for the image data of the second object included in the captured image, and the generated first guide and the generated based on the teaching data A second guide, the predetermined guide Having a visual servo control unit which performs control based on visual servo such that the location relationship. According to the fifth aspect of the present invention, guides are generated for two objects included in the captured image, and visual servoing using these guides is executed. Therefore, it is not necessary to evaluate the difference between the target image and the captured image, and a stable visual servo that is not affected by the work environment can be realized.

  In the teaching data, the first guide is shown in association with the feature of the image data of the first object, and the second guide is shown in association with the feature of the image data of the second object, The first guide generation unit generates the first guide based on the teaching data in association with the feature of the image data of the first object included in the acquired captured image, and the second guide The generation unit may generate the second guide in association with the feature of the image data of the second object included in the acquired captured image based on the teaching data. Thereby, a guide can be easily and accurately generated based on the characteristics of the image of the object included in the captured image.

  The teaching data indicates a plurality of the first guides associated with each of a plurality of features of the image data of the first object, and the first guide generating unit is acquired based on the teaching data. The first guide may be generated in association with any one of a plurality of features of the image data of the first object included in the captured image. As a result, a plurality of guides can be generated for the object included in the captured image, so that stable visual servoing that is less affected by changes in the work environment can be realized.

  A sixth aspect of the present invention that solves the above problem is a robot, which is a first guide set for the image data of the first object and a second set for the imaging range of the imaging device. A teaching data storage unit for storing teaching data indicating a predetermined positional relationship with the guide, an image acquisition unit for acquiring a captured image including the first object and the second object from the imaging device, A first guide generation unit that generates the first guide for the image data of the first object included in the acquired captured image based on the teaching data, and the acquisition based on the teaching data A second guide generation unit that generates the second guide for the image data of the second object included in the captured image, and the generated first guide and the generation based on the teaching data With the second guide Having a visual servo control unit which performs control based on visual servo so that the predetermined positional relationship. According to the sixth aspect of the present invention, a guide for an object included in a captured image and a guide for an imaging range are generated, and visual servoing using these guides is executed. Therefore, it is not necessary to evaluate the difference between the target image and the captured image, and a stable visual servo that is not affected by the work environment can be realized. In addition, since the guide for the imaging range is used, image processing for extracting features from the captured image is not necessary, and the processing load is reduced.

  A seventh aspect of the present invention that solves the above problem is a robot control method, wherein the first guide is set for the image data of the first object, and the image data of the second object. The second guide to be set includes a teaching data storage step for teaching data indicating a predetermined positional relationship, an image acquisition step for acquiring a captured image including the first object and the second object, A first guide generation step for generating the first guide for the image data of the first object included in the acquired captured image based on the teaching data, and the acquisition based on the teaching data A second guide generation step of generating the second guide for the image data of the second object included in the captured image, and the generated first guide and the generation based on the teaching data The And a second guide has a a visual servo control step of performing control based on visual servo so that the predetermined positional relationship. According to the seventh aspect of the present invention, guides are generated for two objects included in the captured image, and visual servoing using these guides is executed. Therefore, it is not necessary to evaluate the difference between the target image and the captured image, and a stable visual servo that is not affected by the work environment can be realized.

  An eighth aspect of the present invention that solves the above problem is a robot system, wherein the first guide setting unit that receives the setting of the first guide for the image data of the first object, and the second object A second guide setting unit that receives setting of a second guide for image data, a display unit that displays the first guide and the second guide, and a predetermined positional relationship between the first guide and the second guide A teaching data generation unit that generates teaching data indicating a teaching data, a teaching data storage unit that stores the generated teaching data, and an image acquisition unit that acquires a captured image including the first object and the second object A first guide generating unit that generates the first guide for the image data of the first object included in the acquired captured image based on the teaching data, and based on the teaching data , The above A second guide generation unit that generates the second guide for the image data of the second object included in the captured image, and the generated first guide and the generation based on the teaching data And a visual servo controller that performs control based on visual servo so that the predetermined positional relationship is obtained. According to the eighth aspect of the present invention, guides are generated for the two objects included in the captured image, and visual servoing using these guides is executed. Therefore, it is not necessary to evaluate the difference between the target image and the captured image, and a stable visual servo that is not affected by the work environment can be realized.

  A ninth aspect of the present invention that solves the above problem is a control method of a robot system, wherein a first guide setting step for receiving a setting of a first guide for image data of a first object, A second guide setting step for receiving setting of a second guide for the image data of the object; a display step for displaying the first guide and the second guide; and the first guide and the second guide are predetermined. A teaching data generating step for generating teaching data indicating the positional relationship of the teaching data, a teaching data storing step for storing the generated teaching data, and a captured image including the first object and the second object. Based on the image acquisition step and the teaching data, a first guide for generating the first guide is generated for the image data of the first object included in the acquired captured image. Generation step, a second guide generation step for generating the second guide for the image data of the second object included in the acquired captured image based on the teaching data, and the teaching data And a visual servo control step of performing control based on visual servo so that the generated first guide and the generated second guide have the predetermined positional relationship. According to the ninth aspect of the present invention, guides are generated for two objects included in a captured image, and visual servoing using these guides is executed. Therefore, it is not necessary to evaluate the difference between the target image and the captured image, and a stable visual servo that is not affected by the work environment can be realized.

It is a figure which shows an example of schematic structure of the robot system which concerns on embodiment of this invention. It is a figure which shows an example of a function structure of a robot system. (A) It is a figure which shows an example of the data structure of target object guide data. (B) It is a figure which shows an example of the data structure of fixed guide data. It is a figure which shows an example of the data structure of teaching data. It is a figure which shows an example of the hardware constitutions which implement | achieve the function of a teaching apparatus. It is a flowchart which shows an example of a teaching process. It is a figure which shows the 1st example of work which a robot performs. (A) It is a figure which shows an example of the guide set to the components W1. (B) It is a figure which shows an example of the guide set to the component W2. (C) It is a figure explaining an example of a fixed guide. It is a flowchart which shows an example of the visual servo process in a 1st operation example. It is FIG. (The 1) which illustrates visually progress of the visual servo in a 1st operation example. It is FIG. (The 2) which illustrates visually progress of the visual servo in a 1st operation example. It is a figure which shows the 2nd example of work which a robot performs. (A) It is a figure which shows an example of the guide set to the component W3. (B) It is a figure which shows an example of the guide set to the components W4. It is a flowchart which shows an example of the visual servo process in a 2nd operation example. It is FIG. (The 1) which illustrates visually progress of the visual servo in a 2nd operation example. It is FIG. (The 2) which illustrates visually progress of the visual servo in a 2nd operation example. (A1)-(A3) It is a figure which shows the example of the some guide set to the component W3. (A1)-(A3) It is a figure which shows the example of the some guide set to the component W4. It is a flowchart which shows the modification of the visual servo process in a 2nd operation example. It is FIG. (1) which illustrates visually progress of the modification of the visual servo in a 2nd operation example. It is FIG. (2) which illustrates visually progress of the modification of the visual servo in a 2nd operation example.

<Embodiment of the present invention>
Embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a schematic configuration of a robot system according to an embodiment of the present invention.

  The robot system 1 includes a robot 2, a teaching device 4, and an imaging device 5. The robot 2 has a control unit 3. The robot 2 and the teaching device 4 are connected to be communicable. In addition, the robot 2 and the imaging device 5 are connected to be communicable.

  The robot 2 is controlled by the control unit 3 and performs a desired work. The robot 2 is disposed near the work table T, for example, and performs work in a predetermined work area on the work table T. Although the work content of the robot 2 is not particularly limited, for example, a work of fitting a plurality of works W is conceivable.

  The robot 2 includes a body part 20, an arm 21, a hand 22, a leg part 24, and a head part 25.

  The arm 21 is provided on the trunk portion 20. The arm (also referred to as “manipulator”) 21 includes one or more joints (also referred to as “rotating shafts”) 21 a and one or more arm members (also referred to as “links”) 21 b.

  The joint 21a is provided with an actuator (not shown) for operating them. The actuator includes, for example, a servo motor and an encoder. The encoder value output by the encoder is used for feedback control of the robot 2 by the control unit 3.

  A force sensor 21 c is provided at the tip of the arm 21. The force sensor is a sensor that detects a force and a moment received as a reaction force to the force generated by the robot 2. As the force sensor, for example, a six-axis force sensor that can simultaneously detect six components, ie, a force component in the translational three-axis direction and a moment component around the three rotation axes, can be used. The force sensor is not limited to six axes, and may be, for example, three axes.

  By linking each rotation shaft, the posture of the arm member 21b is changed, and the position of interest (also referred to as “end point”) set at the tip of the arm 21 is freely moved within a predetermined movable range. And can be directed in any direction. Note that the position of the end point is not limited to the tip of the arm, and may be set to the tip of the end effector, for example.

  The arm 21 can be said to be a kind of manipulator. The manipulator is a mechanism that moves the position of the end point, and is not limited to an arm and can take various forms. Further, the number of manipulators is not limited to two as shown in the figure, and one or three or more manipulators may be provided.

  The hand 22 is provided at the tip of the arm 21. The hand 22 includes, for example, a plurality of fingers, and can hold or release the work object with at least two fingers. The hand 22 may be detachable from the tip of the arm 21.

  The hand 22 can be said to be a kind of end effector. The end effector is a member for gripping, lifting, lifting, adsorbing, or processing a work object. The end effector can take various forms such as a hand, a hook, and a suction cup. Further, a plurality of end effectors may be provided for one arm.

  The configuration of the robot 2 is not limited to the illustrated example. For example, FIG. 1 shows an example in which the number of joints is six (six axes), but the number of joints (also referred to as “number of axes”) and the number of links may be increased or decreased. In addition, the shape, size, arrangement, structure, and the like of various members such as joints, links, and hands may be changed as appropriate.

  The control unit 3 controls the entire robot 2. For example, the control unit 3 generates a control command for operating the robot 2 based on the teaching data output from the teaching device 4, and operates the robot 2 by driving an actuator according to the generated control command. In addition, the control unit 3 controls the imaging device 5 and acquires a captured image. Although details will be described later, the control unit 3 operates the robot 2 by visual servo.

  The imaging device 5 captures, for example, a work area on the work table T, generates captured image data, and outputs the captured image data to the robot 2. The imaging device 5 is a camera having, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like.

  The teaching device 4 generates teaching data indicating work contents to be executed by the robot 2 in accordance with a user operation. Further, the teaching device 4 teaches the robot 2 by outputting the generated teaching data to the control unit 3 of the robot 2. The teaching device 4 will be described in detail later.

  The configuration of the robot system 1 described above is the main configuration in describing the features of the present embodiment, and is not limited to the above configuration example. Further, the configuration of a general robot system is not excluded.

  For example, the installation position of the imaging device 5 is not particularly limited, and can be installed on a ceiling, a wall, or the like. In addition, the imaging device 5 may be provided on the head 25 of the robot 2, the tip of the arm 21, the trunk 20, and the like. For example, in addition to the imaging device 5, another imaging device may be provided. For example, the control unit 3 may be provided separately from the robot 2 as a control device and connected to the robot 2 from the outside. For example, the control unit 3 and the teaching device 4 may be integrated as a control device.

  FIG. 2 is a diagram illustrating an example of a functional configuration of the robot system.

  The teaching device 4 includes an input unit 40, a display unit 41, a guide setting unit (also referred to as “first guide setting unit” and “second guide setting unit”) 42, a teaching data generation unit 43, and teaching data. The transmitter 44, the model data storage 45, the guide data storage 46, and the teaching data storage 47 are provided.

  The input unit 40 receives user operations. The display unit 41 displays an interface screen and the like for the user.

  The guide setting unit 42 sets a virtual guide (also referred to as “target guide”) for an arbitrary position of the work target. The object guide is used as an image indicating the current position of the work object instead of the work object in the visual servo.

  Specifically, model data such as image data representing a work object in two dimensions or three dimensions is stored in advance in the model data storage unit 45 by a user operation via the input unit 40 or the like. The guide setting unit 42 acquires model data of the work object from the model data storage unit 45. Further, the guide setting unit 42 performs predetermined image processing (for example, edge detection, contour detection, corner detection, etc.) for extracting the feature position of the model on the acquired model data of the work target, The feature position of the work object is extracted. The feature position is not limited to one point, and may be a set of a plurality of points, or simply referred to as “feature”.

  The guide setting unit 42 sets a guide position having a predetermined positional relationship with respect to the extracted feature position of the work object. For example, the guide setting unit 42 causes the display unit 41 to display an image indicating the feature position of the work object together with the model data of the work object. In addition, the guide setting unit 42 receives the designation of the guide position on the displayed image via the input unit 40. The guide setting unit 42 stores object guide data indicating the relative positional relationship of the designated guide position with respect to the displayed feature position in the guide data storage unit 46. The guide position of the object is not limited to one point, and may be a set of a plurality of points, or simply referred to as “object guide”.

  FIG. 3A is a diagram illustrating an example of a data structure of the object guide data. As shown in FIG. 3A, the object guide data includes, for each object guide, an ID 451 that is identification information of the object guide, a guide position 452 of the object guide, and the object. A record in which the characteristic position 453 of the object that is the reference position of the guide is associated is stored. Note that the feature position 453 includes information regarding an image processing algorithm used for extracting the feature position.

  Returning to the description of FIG. The guide setting unit 42 sets a virtual guide (fixed guide) for an arbitrary position within the imaging range of the imaging device 5. The fixed guide is used as an image showing the target position of work in the visual servo.

  Specifically, the guide setting unit 42 causes the display unit 41 to display an image indicating the imaging range, for example. In addition, the guide setting unit 42 receives the designation of the guide position on the displayed image via the input unit 40. The guide setting unit 42 stores fixed guide data indicating the designated guide position in the guide data storage unit 46. The fixed guide position is not limited to one point, and may be a set of a plurality of points, or simply referred to as a “fixed guide”.

  FIG. 3B is a diagram illustrating an example of a data structure of fixed guide data. As shown in FIG. 3B, for each fixed guide, a record in which an ID 461 that is identification information of the fixed guide and a guide position 462 of the fixed guide are associated is stored for each fixed guide. The

  Returning to the description of FIG. The teaching data generation unit 43 generates teaching data for the purpose that the object guide and the fixed guide have a predetermined positional relationship (including a case where they coincide with each other at the same position). In addition, the teaching data generation unit 43 generates teaching data for which one object guide and another object guide have a predetermined positional relationship (including the case where they coincide at the same position).

  Specifically, the teaching data generation unit 43 receives, for example, an object guide designation, a fixed guide designation, a target position designation, and a command for moving the object guide to the target position via the input unit 40. . Further, for example, when there are a plurality of instructions, the teaching data generation unit 43 receives the process order of each instruction. The teaching data generation unit 43 generates teaching data including each received information and stores it in the teaching data storage unit 47.

  FIG. 4 is a diagram illustrating an example of the data structure of teaching data. As shown in FIG. 4, the teaching data stores a record in which a process number 471 and a teaching content 472 are associated with each other in the order of processes. The teaching content 472 includes guide data (object guide data, fixed guide data) corresponding to the designated guide.

  Returning to the description of FIG. The teaching data transmission unit 44 acquires the teaching data generated by the teaching data generation unit 43 from the teaching data storage unit 47 and transmits it to the control unit 3 of the robot 2.

  The model data storage unit 45 stores model data as described above. The guide data storage unit 46 stores guide data as described above. The teaching data storage unit 47 stores guide data as described above.

  The control unit 3 of the robot 2 includes a teaching data receiving unit 30, a visual servo control unit 31, an image acquisition unit 34, and a teaching data storage unit 35. The visual servo control unit 31 includes a feature extraction unit 32 and a guide generation unit (also referred to as “first guide generation unit” or “second guide generation unit”) 33.

  The teaching data receiving unit 30 receives teaching data from the teaching device 4 and stores it in the teaching data storage unit 35.

  The visual servo control unit 31 controls the operation of the robot 2 based on the teaching data stored in the teaching data storage unit 35. Therefore, the visual servo control unit 31 includes a feature extraction unit 32 and a guide generation unit 33.

  Specifically, the visual servo control unit 31 specifies a work object to be moved and a target position to move the work object based on the teaching data. Further, the visual servo control unit 31 acquires a captured image from the image acquisition unit 34. Further, the visual servo control unit 31 performs predetermined image processing (for example, edge detection, contour detection) in which the feature extraction unit 32 extracts the feature position of the work object to be moved from the captured image based on the teaching data. , Corner detection, etc.) and the feature position of the work object is extracted. Further, the visual servo control unit 31 generates an object guide of the work object to be moved to the guide position having a predetermined relative positional relationship with the extracted feature position by the guide generation unit 33 based on the teaching data. .

  In addition, when the target position is a predetermined position within the imaging range, the visual servo control unit 31 uses the guide generation unit 33 based on the teaching data to fix the fixed guide that is the target position at the specified position of the captured image. Is generated. On the other hand, when the target position is the work object, the visual servo control unit 31 uses the feature extraction unit 32 to set the feature position of the work object that is the target position with respect to the captured image based on the teaching data. Predetermined image processing (for example, edge detection, contour detection, corner detection, etc.) to be extracted is executed, and the feature position of the work object is extracted. Further, the visual servo control unit 31 generates an object guide as a target position at a guide position having a predetermined relative positional relationship with the extracted feature position by the guide generation unit 33 based on the teaching data.

  Further, the visual servo control unit 31 is based on an object guide to be moved and a guide (fixed guide or object guide) that is a target position generated on the captured image, and these guides are teaching data. Visual servo is performed so that the predetermined positional relationship is taught.

  Here, the visual servo control unit 31 deletes image information other than the generated guide from the captured image (that is, image information originally included in the captured image and the background of the guide image), and includes only the guide. Based on the visual servo. Of course, an image having the same size as the captured image may be generated, and a guide may be generated on the image.

  In the visual servo of this embodiment, an image including only a guide for a target position generated based on a captured image is used instead of the target image. Further, instead of the current image, an image including only a work target guide generated based on the captured image is used. These guides are used as input information (control amount) to the visual servo system. The visual servo control unit 31 compares the guides with each other based on these guides. If the guides match, the visual servo is terminated. If the guides do not match, a control command is generated and the robot 2 is operated.

  The feature extraction unit 32 refers to the guide data included in the teaching data, and specifies a predetermined image processing algorithm from the feature position associated with the designated guide ID. The feature extraction unit 32 executes the specified image processing algorithm on the captured image, and extracts and specifies the feature position of the work target.

  The guide generation unit 33 refers to the guide data included in the teaching data, and specifies the guide position associated with the guide ID of the designated object guide. Further, the guide generation unit 33 generates an object guide at a guide position having a predetermined relative positional relationship with the feature position of the work object specified by the feature extraction unit 32 based on the specified guide position. In addition, the guide generation unit 33 generates a fixed guide at the guide position associated with the guide ID of the designated fixed guide.

  The image acquisition unit 34 controls the imaging device 5 to perform imaging and acquires a captured image.

  The teaching data storage unit 35 stores teaching data as described above.

  FIG. 5 is a diagram illustrating an example of a hardware configuration for realizing the function of the teaching apparatus.

  The teaching device 4 includes, for example, an arithmetic device 90 such as a CPU (Central Processing Unit), a main storage device 91 such as a RAM (Random Access Memory), and an auxiliary device such as an HDD (Hard Disk Drive) as shown in FIG. A storage device 92, a communication interface (I / F) 93 for connecting to a communication network by wire or wireless, an input device 94 such as a mouse, keyboard, touch sensor, or touch panel, a display device 95 such as a liquid crystal display, This can be realized by a computer 9 including a read / write device 96 that reads / writes information from / to a portable storage medium such as a DVD (Digital Versatile Disk).

  For example, the guide setting unit 42, the teaching data generation unit 43, the teaching data transmission unit 44, and the like are realized by the arithmetic device 90 executing a predetermined program loaded from the auxiliary storage device 92 or the like to the main storage device 91. . The model data storage unit 45, the guide data storage unit 46, the teaching data storage unit 47, and the like are realized by the arithmetic device 90 using the main storage device 91 or the auxiliary storage device 92, for example. Communication with the robot 2 is realized, for example, when the arithmetic device 90 uses the communication I / F 93. The input unit 40 is realized, for example, when the arithmetic device 90 uses the input device 94. The display unit 41 is realized, for example, when the arithmetic device 90 uses the display device 95. The predetermined program may be installed from, for example, a storage medium read by the read / write device 96, or may be installed from a network via the communication I / F 93.

  The control unit 3 can be realized by, for example, a controller board including an arithmetic device, a storage device, a processing circuit, a drive circuit, and the like. For example, the teaching data receiving unit 30, the visual servo control unit 31, the image acquisition unit 34, and the like are realized by the arithmetic device executing a predetermined program loaded in the storage device. The teaching data storage unit 35 is realized, for example, when the arithmetic device uses a storage device.

  The functional configuration of the robot system 1 described above is classified according to main processing contents in order to facilitate understanding of the configuration of the robot system 1. The present invention is not limited by the way of classification and names of the constituent elements. The configuration of the robot system 1 can be classified into more components depending on the processing content. Moreover, it can also classify | categorize so that one component may perform more processes. Further, the processing of each component may be executed by one hardware or may be executed by a plurality of hardware.

  Further, the functions and processing sharing of the teaching device 4 and the control unit 3 are not limited to the example described. For example, at least some of the functions of the teaching device 4 may be included in the control unit 3 and realized by the control unit 3. For example, at least a part of the functions of the control unit 3 may be included in the teaching device 4 and realized by the teaching device 4.

  Next, processes and operations realized by the robot system 1 will be described.

  First, a description will be given based on a first working example performed by the robot 2 with reference to FIGS. Then, with reference to FIGS. 12 to 16, a description will be given based on a second work example performed by the robot 2.

  FIG. 7 is a diagram illustrating a first work example performed by the robot. The first work example is a work of fitting the part W2 into the recess formed in the part W1. Since the robot 2 can move the component by gripping the component with the hand 22, the description regarding the gripping operation is omitted.

  FIG. 6 is a flowchart showing an example of the teaching process. In the model data storage unit 45, model data of the parts W1 and W2 are stored in advance.

  First, the guide setting unit 42 receives the setting of the object guide (step S100). Specifically, the guide setting unit 42 receives the model of the parts (part W1, part W2) necessary for the work from the model data stored in the model data storage unit 45 via the input unit 40 from the user. Accept data selection. The guide setting unit 42 acquires, for example, an image including a component necessary for the operation from the imaging device 5 and the like, recognizes the component by image processing from the acquired image, and stores model data corresponding to the recognized component as a model. You may make it select from the model data stored in the data storage part 45. FIG.

  In addition, the guide setting unit 42 performs predetermined image processing for extracting a feature position on each selected model data, and extracts the feature position. The guide setting unit 42 may accept selection and setting of the image processing algorithm to be used for each model data from the user. Further, the guide setting unit 42 causes the display unit 41 to display an image indicating the feature position extracted together with the model data. For example, as shown in FIG. 8A (a diagram showing an example of a guide set for the part W1), the extracted feature position F1 (shown by a broken line) is displayed together with the model data W1 of the part W1. . Further, for example, as shown in FIG. 8B (a diagram showing an example of a guide set for the part W2), the extracted feature position F2 (indicated by a broken line) is displayed together with the model data W2 of the part W2. Is done.

  In addition, the guide setting unit 42 receives a guide position designation for the displayed image from the user via the input unit 40. For example, in FIG. 8A, the guide G1 is set so as to correspond to the concave portion of the component W1. Further, for example, in FIG. 8B, the guide G2 is set corresponding to one end of the component W2 (the side inserted into the concave portion of the component W1).

  Here, the object guide has a reference position and a direction component. The reference position of the guide has, for example, an XYZ coordinate value when the model data is three-dimensional, and an XY coordinate value when the model data is two-dimensional. The guide direction component is a three-dimensional XYZ-axis direction or a two-dimensional XY-axis direction with the reference position of the guide as the origin. The guide setting unit 42 receives designation of the reference position and direction component of the object guide as a guide position from the user via the input unit 40.

  When the guide position (reference position, direction component) with respect to the model data is designated, the guide setting unit 42 converts the reference position of the designated guide position into information indicating the relative position with respect to the extracted feature position. Then, the guide setting unit 42 assigns the ID assigned to the object guide or the ID specified by the user, the guide position (converted reference position, direction component), and the extracted feature position (feature position). A record associating information to be identified and information relating to an image processing algorithm used for extracting the feature position) is added to the object guide data in the guide data storage unit 46.

  Then, the guide setting unit 42 receives a fixed guide setting (step S110). Specifically, for example, the guide setting unit 42 causes the display unit 41 to display an image indicating the imaging range of the imaging device 5. In addition, the guide setting unit 42 receives a guide position designation for the displayed image from the user via the input unit 40. For example, in FIG. 8C (a diagram illustrating an example of the fixed guide), the fixed guide SG1 is set near the center of the imaging range.

  Here, the fixed guide has a reference position and a direction component in the same manner as the object guide. The guide setting unit 42 receives designation of the reference position and direction component of the fixed guide as a guide position from the user via the input unit 40.

  When the guide position (reference position, direction component) for the imaging range is determined, the guide setting unit 42 assigns the ID assigned to the fixed guide or the ID specified by the user and the guide position (reference position, direction component). ) Is added to the fixed guide data in the guide data storage unit 46.

Then, the teaching data generation unit 43 generates teaching data (step S120). Specifically, the teaching data generation unit 43 receives settings of each process of the robot 2 that realizes the first work example from the user via the input unit 40. The teaching data generation unit 43 receives a teaching including a designation of a guide and a command to move the guide to a target position, for example, as described below, and receives a process number 471 and a teaching content 472 (in the following, “process number: teaching content”). To the teaching data in the teaching data storage unit 47.
Step 1: The object guide G1 (reference position) is matched with the fixed guide SG1 (reference position).
Step 2: The object guide G2 (reference position) is matched with the fixed guide SG1 (reference position) at a predetermined offset distance (m pixels from the reference position in the X direction).
Step 3: Move the object guide G2 (reference position) by a predetermined distance (n pixels in the X direction).
Step 3 is a preparatory operation for the operation of fitting the component W2 into the component W1.

  In addition, the teaching data generation unit 43 sets the guide data of the guides (fixed guide SG1, object guide G1, object guide G2) specified by the instruction contents in the instruction contents 472 of each step.

  Then, the teaching data transmission unit 44 transmits the teaching data generated in step S120 to the robot 2 (step S130), and ends the processing of the flowchart shown in FIG.

  As described above, the setting of the guide position and the teaching data indicating the work contents of the robot 2 are set, and the robot 2 is taught. Note that the order of step S100 and step S110 may be reversed. Further, step S130 may be executed at an arbitrary timing designated by the user.

  FIG. 9 is a flowchart showing an example of visual servo processing in the first work example. FIG. 10 (a diagram for visually explaining the progress of visual servo in the first work example (part 1)) and FIG. 11 (a diagram for visually explaining the progress of visual servo in the first work example (part 2) )) To explain. For easy understanding, the hand 22 and the like of the robot 2 are omitted in FIGS. 10 and 11.

  It is assumed that the teaching data transmitted from the teaching device 4 is received by the teaching data receiving unit 30 and stored in the teaching data storage unit 35. The visual servo control unit 31 controls the operation of the robot 2 based on the teaching content of each process of the teaching data. Steps S200 to S250 correspond to the above-mentioned “Step 1”. Steps S260 to S310 correspond to the above-mentioned “Step 2”. Step S320 corresponds to the above-mentioned “Step 3”.

  First, the image acquisition unit 34 controls the imaging device 5 to perform imaging and acquires a captured image (step S200).

  Then, the visual servo control unit 31 generates the fixed guide SG1 (step S210). Specifically, the guide generation unit 33 acquires the guide position of the fixed guide SG1 from the guide data included in the teaching data in step 1. Further, the guide generation unit 33 generates the fixed guide SG1 on the captured image acquired in step S200 based on the acquired guide position (FIG. 10A).

  Then, the visual servo control unit 31 extracts the feature of the part W1 (step S220). Specifically, the feature extraction unit 32 acquires the feature position of the part W1 from the guide data included in the teaching data in step 1. In addition, the feature extraction unit 32 executes the image processing algorithm specified from the acquired feature position on the captured image acquired in step S200, and extracts the feature. In addition, the feature extraction unit 32 specifies a feature that matches the acquired feature position of the component W1 from the extracted features. The coincidence of the feature positions is determined by, for example, the coincidence rate (for example, a coincidence rate of 90% or more) between the arrangement of each pixel constituting the feature position of the component W1 and the arrangement of each pixel constituting the extracted feature. be able to. In this way, the feature position F1 of the component W1 is extracted and specified on the captured image (FIG. 10A).

  Then, the visual servo control unit 31 generates a guide G1 corresponding to the part W1 (step S230). Specifically, the guide generation unit 33 acquires the guide position (indicating a relative position with respect to the feature position) of the component W1 from the guide data included in the teaching data in step 1. In addition, the guide generation unit 33 sets the object guide G1 at the position indicated by the acquired guide position with reference to the characteristic position F1 of the component W1 extracted and specified in step S220. In this way, the object guide G1 is set on the captured image with the feature position F1 as a reference (FIG. 10A).

  Then, the visual servo control unit 31 executes visual servo (step S240). Specifically, the visual servo control unit 31 deletes image information other than the generated guide from the captured image. Then, the visual servo control unit 31 executes the visual servo so that the object guide G1 (reference position) matches the fixed guide SG1 (reference position) based on the teaching data in step 1 (FIG. 10). (A)). When the object guide G1 is not located within a predetermined distance from the fixed guide SG1, the visual servo control unit 31 generates a control command and operates the robot arm (moves the end point). On the other hand, when the object guide G1 is located within a predetermined distance from the fixed guide SG1, the visual servo control unit 31 determines that the visual servo in step 1 has converged without generating a control command ( FIG. 10B).

  Then, the visual servo control unit 31 determines whether or not the visual servo has converged (step S250). Specifically, when it is determined that the visual servo control unit 31 has converged in step S240 (step S250: Y), the process proceeds to step S260. On the other hand, if it is determined in step S240 that it has not converged (step S250: N), the process returns to step S200.

  When the visual servo of step S240 has converged (step S250: Y), the image acquisition unit 34 controls the imaging device 5 to perform imaging and acquires a captured image (step S260).

  Then, the visual servo control unit 31 generates the fixed guide SG1 (step S270). Specifically, the guide generation unit 33 acquires the guide position of the fixed guide SG1 from the guide data included in the teaching data in step 2. Further, the guide generation unit 33 generates the fixed guide SG1 on the captured image acquired in step S260 based on the acquired guide position (FIG. 10B).

  Then, the visual servo control unit 31 extracts the feature of the part W2 (step S280). Specifically, the feature extraction unit 32 acquires the feature position of the component W2 from the guide data included in the teaching data in step 2. In addition, the feature extraction unit 32 executes the image processing algorithm specified from the acquired feature position on the captured image acquired in step S260, and extracts the feature. In addition, the feature extraction unit 32 specifies a feature that matches the acquired feature position of the component W2 from the extracted features. The coincidence of the feature positions is determined by, for example, the coincidence rate (for example, a coincidence rate of 90% or more) between the arrangement of each pixel constituting the feature position of the component W2 and the arrangement of each pixel constituting the extracted feature. I can do it. In this way, the characteristic position F2 of the component W2 is extracted and specified on the captured image (FIG. 10B).

  Then, the visual servo control unit 31 generates a guide G2 corresponding to the part W2 (step S290). Specifically, the guide generation unit 33 acquires the guide position (indicating a relative position with respect to the feature position) of the component W2 from the guide data included in the teaching data in step 2. In addition, the guide generation unit 33 sets the object guide G2 at the position indicated by the acquired guide position on the basis of the characteristic position F2 of the component W2 extracted and specified in step S280. In this way, the object guide G2 is set on the captured image with the feature position F2 as a reference (FIG. 10B).

  Then, the visual servo control unit 31 executes visual servo (step S300). Specifically, the visual servo control unit 31 deletes image information other than the generated guide from the captured image. Then, based on the teaching data in step 2, the visual servo control unit 31 moves the object guide G2 (reference position) to the fixed guide SG1 (reference position) with a predetermined offset distance (m pixels in the X direction from the reference position). ) Is executed so as to match (FIG. 10B). When the object guide G2 is not located within a predetermined distance from the offset position E1 with reference to the fixed guide SG1, the visual servo control unit 31 generates a control command to operate the robot arm (end). Move point). On the other hand, when the object guide G2 is located within a predetermined distance from the offset position E1, the visual servo control unit 31 determines that the visual servo in step 2 has converged without generating a control command. (FIG. 11 (A)).

  Then, the visual servo control unit 31 determines whether or not the visual servo has converged (step S310). Specifically, when it is determined that the visual servo control unit 31 has converged in step S300 (step S310: Y), the process proceeds to step S320. On the other hand, if it is determined in step S300 that it has not converged (step S310: N), the process returns to step S260.

  When the visual servo in step S300 converges (step S310: Y), the visual servo control unit 31 performs a predetermined operation of the robot (step S320). Specifically, the visual servo control unit 31 moves the arm of the robot so that the object guide G2 (reference position) moves a predetermined distance (n pixels in the X direction) based on the teaching data in step 3. Operate (FIG. 11B). The process of step S320 may be performed by visual servoing or position control. Then, the visual servo control unit 31 ends the process of the flowchart shown in FIG. In addition, after completion | finish of the flowchart shown in FIG. 9, the control part 3 controls the robot 2 by position control, impedance control, etc., for example, and inserts the components W2 in the components W1.

  As described above, the object guide and the fixed guide are generated for the captured image, and the visual servo is performed so as to match these guides.

  FIG. 12 is a diagram illustrating a second work example performed by the robot. The second work example is to insert and tighten the screw W3b into the screw hole formed in the component W4. A driver W3a is used to move and tighten the screw W3b. Before starting the movement of the screw W3b by visual servo, the head of the screw W3a is fixed to the magnetic tip of the driver W3a. Since the robot 2 can move the component by gripping the component or the driver W3a with the hand 22, the description regarding the gripping operation is omitted.

  The teaching process is basically the same as that shown in FIG. 6, but the setting of the fixed guide (step S110 in FIG. 6) is not necessary. The model data storage unit 45 stores in advance model data of the part W4 and model data in a state where the screw W3b is fixed to the driver W3a (hereinafter referred to as “part W3”).

  First, the guide setting unit 42 receives model data of each part (part W3, part W4) necessary for the work from the model data stored in the model data storage unit 45 via the input unit 40 from the user. Accept selection.

  In addition, the guide setting unit 42 performs predetermined image processing for extracting a feature position on each selected model data, and extracts the feature position. The guide setting unit 42 may accept selection and setting of the image processing algorithm to be used for each model data from the user. Further, the guide setting unit 42 causes the display unit 41 to display an image indicating the feature position extracted together with the model data. For example, as shown in FIG. 13A (a diagram showing an example of a guide set for the part W3), the extracted feature position F3 (shown by a broken line) is displayed together with the model data W3 of the part W3. . Further, for example, as shown in FIG. 13B (a diagram showing an example of a guide set for the component W4), the extracted feature position F4 (shown by a broken line) is displayed together with the model data W4 of the component W4. Is done.

  In addition, the guide setting unit 42 receives a guide position designation for the displayed image from the user via the input unit 40. For example, in FIG. 13A, the guide G3 is set so as to correspond to the tip of the screw of the component W3. For example, in FIG. 13B, the guide G4 is set in correspondence with the screw hole of the component W4.

  When the guide position (reference position, direction component) with respect to the model data is designated, the guide setting unit 42 converts the reference position of the designated guide position into information indicating the relative position with respect to the extracted feature position. Then, the guide setting unit 42 assigns the ID assigned to the object guide or the ID specified by the user, the guide position (converted reference position, direction component), and the extracted feature position (feature position). A record associating information to be identified and information relating to an image processing algorithm used for extracting the feature position) is added to the object guide data in the guide data storage unit 46.

Then, the teaching data generation unit 43 receives settings of each process of the robot 2 that realizes the second work example from the user via the input unit 40. The teaching data generation unit 43 receives a teaching including a designation of a guide and a command for moving the guide to a target position, for example, as described below, and adds it to the teaching data of the teaching data storage unit 47 as a process number 471 and a teaching content 472. To do.
Step 1: Match the object guide G3 (reference position) with the object guide G4 (reference position) at a predetermined offset distance (m pixels from the reference position in the Z direction).
Step 2: Move the object guide G3 (reference position) by a predetermined distance (n pixels in the Z direction).
Step 2 is a preparatory operation for the screw tightening operation for the component W4 of the component W3.

  In addition, the teaching data generation unit 43 sets the guide data of the guides (target object guide G3, target object guide G4) specified by the teaching contents in the teaching contents 472 of each step.

  Then, the teaching data transmission unit 44 transmits the generated teaching data to the robot 2.

  FIG. 14 is a flowchart showing an example of visual servo processing in the second work example. FIG. 15 (a diagram for visually explaining the progress of visual servo in the second work example (part 1)) and FIG. 16 (a diagram for visually explaining the progress of visual servo in the second work example (part 2) )) To explain. For easy understanding, the hand 22 and the like of the robot 2 are omitted in FIGS. 15 and 16.

  It is assumed that the teaching data transmitted from the teaching device 4 is received by the teaching data receiving unit 30 and stored in the teaching data storage unit 35. The visual servo control unit 31 controls the operation of the robot 2 based on the teaching content of each process of the teaching data. Steps S400 to S470 correspond to the above-mentioned “Step 1”. Step S480 corresponds to the above-mentioned “Step 2”.

  First, the image acquisition unit 34 controls the imaging device 5 to perform imaging and acquires a captured image (step S400).

  Then, the visual servo control unit 31 extracts the feature of the part W4 (step S410). Specifically, the feature extraction unit 32 acquires the feature position of the component W4 from the guide data included in the teaching data in step 1. In addition, the feature extraction unit 32 executes an image processing algorithm specified from the acquired feature position on the captured image acquired in step S400, and extracts the feature. In addition, the feature extraction unit 32 specifies a feature position that matches the acquired feature position of the component W4 from the extracted features. In this way, the feature position F4 of the component W4 is extracted on the captured image (FIG. 15A).

  Then, the visual servo control unit 31 generates a guide G4 corresponding to the component W4 (step S420). Specifically, the guide generation unit 33 acquires the guide position (indicating a relative position with respect to the feature position) of the component W4 from the guide data included in the teaching data in step 1. In addition, the guide generation unit 33 sets the object guide G4 at the position indicated by the acquired guide position with reference to the characteristic position F4 of the component W4 extracted and specified in step S410. In this way, the object guide G4 is set on the captured image with the feature position F4 as a reference (FIG. 15A).

  Then, the visual servo control unit 31 extracts the feature of the part W3 (step S430). Specifically, the feature extraction unit 32 acquires the feature position of the part W3 from the guide data included in the teaching data in step 1. In addition, the feature extraction unit 32 executes an image processing algorithm specified from the acquired feature position on the captured image acquired in step S400, and extracts the feature. In addition, the feature extraction unit 32 specifies a feature position that matches the acquired feature position of the component W3 from the extracted features. In this way, the feature position F3 of the component W3 is extracted on the captured image (FIG. 15B).

  Then, the visual servo control unit 31 generates a guide G3 corresponding to the part W3 (step S440). Specifically, the guide generation unit 33 acquires the guide position (indicating a relative position with respect to the feature position) of the component W3 from the guide data included in the teaching data in step 1. In addition, the guide generation unit 33 sets the object guide G3 at the position indicated by the acquired guide position with reference to the characteristic position F3 of the component W3 extracted and specified in step S430. In this way, the object guide G3 is set on the captured image with the feature position F3 as a reference (FIG. 15B).

  Then, the visual servo control unit 31 executes visual servo (step S460). Specifically, the visual servo control unit 31 deletes image information other than the generated guide from the captured image. Then, based on the teaching data in step 1, the visual servo control unit 31 moves the object guide G3 (reference position) to the object guide G4 (reference position) with a predetermined offset distance (m pixels in the Z direction from the reference position). The visual servo is executed so as to match the position) (FIG. 15B). When the object guide G3 is not located within a predetermined distance from the offset position E4 with reference to the object guide G4, the visual servo control unit 31 generates a control command to operate the robot arm ( Move the endpoint). On the other hand, when the object guide G3 is located within a predetermined distance from the offset position E4, the visual servo control unit 31 determines that the visual servo in step 1 has converged without generating a control command. (FIG. 16 (A)).

  Then, the visual servo control unit 31 determines whether or not the visual servo has converged (step S470). Specifically, when it is determined that the visual servo control unit 31 has converged in step S460 (step S470: Y), the process proceeds to step S480. On the other hand, if it is determined in step S460 that convergence has not occurred (step S470: N), the process returns to step S400.

  When the visual servo in step S460 converges (step S470: Y), the visual servo control unit 31 performs a predetermined operation of the robot (step S480). Specifically, the visual servo control unit 31 moves the arm of the robot so as to move the object guide G3 (reference position) by a predetermined distance (n pixels in the Z direction) based on the teaching data in step 2. Operate (FIG. 16B). Then, the visual servo control unit 31 ends the process of the flowchart shown in FIG. After the end of the flowchart shown in FIG. 14, the control unit 3 controls the robot 2 by, for example, position control, impedance control, etc., and rotates the hand 22 or tightens the screw by the pressing operation of the driver W3a. If possible, the hand 22 is pressed and the screw W3b is attached to the component W4.

  As described above, a plurality of object guides are generated for the captured image, and visual servoing is performed to match these guides.

  According to the present embodiment, two guides are generated for the captured image (current image), and visual servo control is executed using these guides. Since there is no need to evaluate the difference between the target image and the current image, it is possible to realize a highly accurate and stable visual servo even when the work environment such as brightness changes.

  Further, according to the present embodiment, an image processing algorithm for extracting features is set by the user according to the work environment. Further, the guide is set by the set image processing algorithm. As a result, even in a work environment where the difference between the target image and the current image cannot be evaluated with high accuracy, the guide can be generated with high accuracy, so that stable visual servoing that is not affected by the work environment can be realized.

  In addition, according to the present embodiment, since a guide that is a target position is generated for a captured image (current image), it is not necessary to prepare a target image in advance. Therefore, a stable visual servo with high accuracy can be realized more easily.

  Further, in the present embodiment, the fixed guide is generated at a predetermined position with respect to the imaging range, and thus is not affected by the arrangement of the imaging device or the change in the imaging direction. Further, the processing load is reduced because image processing for extracting features from the captured image is not necessary. Furthermore, if a spatial range in which the posture of the robot with high mechanical accuracy (for example, the influence of deflection of the arm due to its own weight is small) can be taken with respect to the arrangement and imaging direction of the imaging device, By setting a guide (fixed guide or object guide) that is a target position, a stable robot operation with high accuracy can be realized easily.

<Modification of Embodiment of the Present Invention>
Next, a modified example of the visual servo process of the above embodiment will be described. In the description of the modification, the same configurations and processes as those in the above embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  In this modification, it is possible to set a plurality of image processing algorithms for extracting feature positions for one work object. In addition, the guide position can be set for each image processing algorithm. In addition, when extraction and identification of a feature position using a certain image processing algorithm fails, another image processing algorithm is used.

  Hereinafter, with reference to FIGS. 17 to 21, a description will be given based on a second working example performed by the robot 2. The model data storage unit 45 stores in advance model data of the part W4 and model data in a state where the screw W3b is fixed to the driver W3a (hereinafter referred to as “part W3”). In addition, it is assumed that a marker used for feature extraction is attached to the part W3 and its model data.

  The guide setting unit 42 receives, from the user, selection and setting of a plurality of image processing algorithms to be used for the model data selected by the user. Further, the guide setting unit 42 causes the display unit 41 to display an image indicating the feature position extracted together with the model data for each image processing algorithm. Further, the guide setting unit 42 receives a guide position designation for the displayed image for each image processing algorithm from the user via the input unit 40.

  For example, in FIG. 17A1, the guide setting unit 42 matches the model data of the part W3 with the model data of the screw W3b stored in advance in the model data storage unit 45, and the feature of the screw W3b. A position F3a (indicated by a broken line) is extracted. In addition, the guide setting unit 42 receives designation of a guide G3a (indicated by an X) corresponding to the feature position F3a from the user. In this figure, a guide G3a is set corresponding to the tip of the screw of the component W3.

  For example, in FIG. 17A2, the guide setting unit 42 performs image processing such as edge detection on the model data of the component W3, and extracts a feature position F3b (shown by a broken line). In addition, the guide setting unit 42 receives a designation of a guide G3b (indicated by an X) corresponding to the feature position F3b from the user. In this figure, a guide G3b is set corresponding to the tip of the screw of the component W3.

  Further, for example, in FIG. 17A3, the guide setting unit 42 performs image processing (for example, detecting the color of the marker) to detect the marker on the model data of the part W3, and the feature position F3c (broken line) Is extracted). In addition, the guide setting unit 42 receives a designation of a guide G3c (indicated by an X) corresponding to the feature position F3c from the user. In this figure, the guide G3c is set on the marker.

  For example, in FIG. 18A1, the guide setting unit 42 matches the model data of the part W4 with the model data of the screw holes stored in advance in the model data storage unit 45, and the feature of the screw holes. Extracted as a position F4a (indicated by a broken line). In addition, the guide setting unit 42 receives designation of a guide G4a (indicated by an X) from the user for the feature position F4a. In this figure, a guide G4a is set corresponding to the screw hole of the component W4.

  For example, in FIG. 18A2, the guide setting unit 42 performs image processing such as edge detection on the model data of the component W4, and extracts a feature position F4b (indicated by a broken line). Further, the guide setting unit 42 receives designation of a guide G4b (indicated by an X) from the user for the feature position F4b. In this figure, a guide G4b is set corresponding to the screw hole of the component W4.

  Also, for example, in FIG. 18 (A3), the guide setting unit 42 performs image processing such as edge detection different from that in FIG. 18 (A2) on the model data of the component W4, and the feature position F4c (broken line) Is extracted). In addition, the guide setting unit 42 receives designation of a guide G4c (indicated by an X) from the user for the feature position F4c. In this figure, the guide G4c is set at a position away from the screw hole of the component W4 by a certain distance.

  When the guide position (reference position, direction component) with respect to the model data is designated, the guide setting unit 42 converts the reference position of the designated guide position into information indicating the relative position with respect to the extracted feature position. Then, the guide setting unit 42 assigns the ID assigned to the object guide or the ID specified by the user, the guide position (converted reference position, direction component), and the extracted feature position (feature position). A record associating information to be identified and information relating to an image processing algorithm used for extracting the feature position) is added to the object guide data in the guide data storage unit 46. In this way, a plurality of object guides having different image processing algorithms are set for one object.

  In addition, the guide setting unit 42 receives designation of priority to be used from a user for a plurality of types of object guides set for each object, and stores them together with guide data. For example, for the guide of the part W3, the priority order is specified in the order of FIGS. 17A1, 17A2, and A3. For example, for the guide of the component W4, the priority order is designated in the order of FIGS. 18A1, 18A2, and A3.

  The teaching data generation unit 43 receives settings of each process of the robot 2 that realizes the second work example from the user via the input unit 40. The teaching data generation unit 43 receives the same teaching as the process of the second working example described above, and adds it to the teaching data of the teaching data storage unit 47 as a process number 471 and a teaching content 472.

  In addition, the teaching data generation unit 43 sets the guide data of the guides (object guides G3a, G3b, G3c, object guides G4a, G4b, G4c) specified by the teaching contents in the teaching contents 472 of each step. To do. Further, the teaching data generation unit 43 sets the priority order of each object guide to 472 in the teaching contents of each process.

  In addition, the teaching data generation unit 43 receives setting of visual servo target conditions using a plurality of guides from the user via the input unit 40, and sets the setting to teaching data. The target condition is, for example, any one of the object guides G3a, G3b, and G3c and any combination of the object guides G4a, G4b, and G4c. A plurality of combinations may be set.

  Then, the teaching data transmission unit 44 transmits the generated teaching data to the robot 2.

  FIG. 19 is a flowchart showing a modification of the visual servo process in the second work example. FIG. 20 (a diagram for visually explaining the progress of the visual servo modification in the second work example (part 1)) and FIG. 21 (the visual progress of the visual servo modification in the second work example) This will be described with reference to the diagram (part 2) to be described. For ease of explanation, the hand 22 and the like of the robot 2 are omitted in FIGS.

  Steps S500 to S630 correspond to the above-mentioned “Step 1”. Step S640 corresponds to the above-mentioned “Step 2”.

  First, the image acquisition unit 34 controls the imaging device 5 to perform imaging and acquires a captured image (step S500).

  Then, the visual servo control unit 31 selects a feature to be extracted from the part W4 (step S510). Specifically, the feature extraction unit 32 selects unselected guide data from a plurality of guide data of the part W4 included in the teaching data in step 1 in accordance with the designated priority order. For example, the priority order is specified in the order of (A1), (A2), and (A3) in FIG.

  Then, the visual servo control unit 31 extracts the feature of the component W4 (step S520). Specifically, the feature extraction unit 32 acquires the feature position of the component W4 from the guide data of the component W4 selected in step S510. In addition, the feature extraction unit 32 executes the image processing algorithm specified by the acquired feature position on the captured image acquired in step S500, and extracts the feature. In addition, the feature extraction unit 32 specifies a feature position that matches the acquired feature position of the component W4 from the extracted features.

  Here, in the process of step S520, the feature position of the part W4 may not be specified depending on the work environment such as the brightness of illumination and the positional relationship between the part W3 and the part W4. For example, as shown in FIG. 20A, when the screw hole of the component W4 is hidden by the component W3, the feature position F4a in FIG. 18A1 and the feature position F4b in FIG. 18A2 are extracted. And cannot be specified.

  Then, the visual servo control unit 31 determines whether or not the feature extraction of the object W4 has been successful (step S530). That is, the feature extraction unit 32 determines whether the feature position of the part W4 selected in step S510 has been successfully extracted and specified based on the result of step S520.

  If extraction of the feature position of the part W4 fails (step S530: N), the visual servo control unit 31 determines whether there is another feature of the part W4 (step S540). Specifically, the feature extraction unit 32 determines whether or not there is unselected guide data in the plurality of guide data of the part W4 included in the teaching data in step 1.

  If there is no unselected feature (step S540: N), the visual servo control unit 31 ends the process of the flowchart shown in FIG. If there is an unselected feature (step S540: Y), the visual servo control unit 31 returns the process to step S510.

  On the other hand, when the feature position of the part W4 is successfully extracted (step S530: Y), the visual servo control unit 31 generates a guide G4 corresponding to the part W4 (step S550). Specifically, the guide generation unit 33 acquires the guide position of the component W4 (indicating a relative position with respect to the feature position) from the guide data of the component W4 last selected in step S510. Further, the guide generation unit 33 sets the object guide G4 at the position indicated by the acquired guide position with reference to the characteristic position F4 of the component W4 extracted and specified last in step S520. In this way, the object guide G4 is set on the captured image with the selected feature position F4 as a reference.

  For example, in FIG. 20B, the feature position F4c in FIG. 18A3 is specified instead of the feature position F4a in FIG. 18A1 and the feature position F4b in FIG. The case where the corresponding guide G4c is set is shown.

  Then, the visual servo control unit 31 determines a feature to be extracted from the part W3 (step S560). Specifically, the feature extraction unit 32 determines unselected guide data from a plurality of guide data of the part W3 included in the teaching data of step 1 in accordance with the designated priority order. For example, the priority order is specified in the order of (A1), (A2), and (A3) in FIG.

  Then, the visual servo control unit 31 extracts the feature of the part W3 (step S570). Specifically, the feature extraction unit 32 acquires the feature position of the component W3 from the guide data of the component W3 selected in step S560. In addition, the feature extraction unit 32 executes the image processing algorithm specified by the acquired feature position on the captured image acquired in step S500, and extracts the feature. In addition, the feature extraction unit 32 specifies a feature position that matches the acquired feature position of the component W3 from the extracted features.

  Here, in the process of step S570, the characteristic position of the part W3 may not be specified depending on the work environment such as the brightness of illumination and the positional relationship between the part W3 and the part W4.

  Then, the visual servo control unit 31 determines whether or not the feature extraction of the object W3 has succeeded (step S580). That is, the feature extraction unit 32 determines whether or not the feature position of the part W3 selected in step S560 has been successfully extracted and specified based on the result of step S570.

  If extraction of the feature position of the part W3 fails (step S580: N), the visual servo control unit 31 determines whether there is another feature of the part W3 (step S590). Specifically, the feature extraction unit 32 determines whether or not there is unselected guide data in the plurality of guide data of the part W3 included in the teaching data in step 1.

  When there is no unselected feature (step S590: N), the visual servo control unit 31 ends the process of the flowchart shown in FIG. If there is an unselected feature (step S590: Y), the visual servo control unit 31 returns the process to step S560.

  On the other hand, if the feature position of the part W3 is successfully extracted (step S580: Y), the visual servo control unit 31 generates a guide G3 corresponding to the part W3 (step S600). Specifically, the guide generation unit 33 acquires the guide position (indicating a relative position with respect to the feature position) of the component W3 from the guide data of the component W3 last selected in step S560. In addition, the guide generation unit 33 sets the object guide G3 at the position indicated by the acquired guide position with reference to the feature position F3 of the part W3 extracted and specified last in step S570. In this way, the object guide G3 is set on the captured image with the selected feature position F3 as a reference.

  For example, in FIG. 21A, the feature position F3c in FIG. 17A3 is specified in place of the feature position F3a in FIG. 17A1 and the feature position F3b in FIG. The case where the corresponding guide G3c is set is shown.

  Then, the visual servo control unit 31 executes visual servo (step S610). Specifically, the visual servo control unit 31 deletes image information other than the generated guide from the captured image. Then, based on the teaching data in step 1, the visual servo control unit 31 moves the object guide G3 (reference position) to the object guide G4 (reference position) with a predetermined offset distance (m pixels in the Z direction from the reference position). The visual servo is executed so that the position) matches. Note that the object guide G4 is the last generated in step S550, and the object guide G3 is the last generated in step S600. When the object guide G3 is not located within a predetermined distance from the offset position with respect to the object guide G4, the visual servo control unit 31 generates a control command to operate the robot arm (the end point is set). Move). On the other hand, when the object guide G3 is located within a predetermined distance from the offset position, the visual servo control unit 31 determines that the visual servo in step 1 has converged without generating a control command.

  For example, FIG. 21A shows a case where the guide G3c in FIG. 17A3 is made to coincide with a predetermined offset position E4c from the guide G4c in FIG. 18A3. FIG. 21B shows a case where the guide G3c converges to an offset position E4c with reference to the guide G4c.

  Then, the visual servo control unit 31 determines whether or not the visual servo has converged (step S620). Specifically, when it is determined that the visual servo control unit 31 has converged in step S610 (step S620: Y), the process proceeds to step S630. On the other hand, if it is determined in step S610 that it has not converged (step S620: N), the process returns to step S500.

  When the visual servo in step S610 has converged (step S620: Y), the visual servo control unit 31 determines whether or not the visual servo satisfies the target condition (step S630). Specifically, for example, the visual servo control unit 31 determines whether or not the combination of the guide G4 and the guide G3 used last in step S610 is the same as the combination of the object guides specified in the teaching data. judge. When the combination of used object guides is different from the specified combination of object guides (step S630: N), the visual servo control unit 31 returns the process to step S500.

  When the combination of the used object guides is the same as the specified object guide combination (step S630: Y), the visual servo control unit 31 performs a predetermined operation of the robot (step S640). Specifically, the visual servo control unit 31 moves the arm of the robot so as to move the object guide G3 (reference position) by a predetermined distance (n pixels in the Z direction) based on the teaching data in step 2. Operate (FIG. 16B). Note that the object guide G3 is generated last in step S600. Then, the visual servo control unit 31 ends the process of the flowchart shown in FIG. After the end of the flowchart shown in FIG. 19, the control unit 3 controls the robot 2 by, for example, position control, impedance control, etc., and rotates the hand 22 or tightens the screw by pressing the driver W3a. If possible, the hand 22 is pressed and the screw W3b is attached to the component W4.

  As described above, when the guide setting fails, another guide is set and the visual servo is executed.

  According to the modification of the above embodiment, a plurality of image processing algorithms for extracting features are set by the user according to the work environment. Further, even if the guide setting fails due to an image processing algorithm, the guide is set according to another image processing algorithm. Thus, it is possible to prevent the guide from being generated due to a change in the work environment, and to continue the visual servoing. In addition, a stable visual servo that is not affected by the work environment can be realized.

  In the modification of the above embodiment, a plurality of guides are prepared for both the component W3 and the component W4. However, one guide may be prepared for one component. Moreover, the modification of said embodiment is not restricted to a 2nd operation example, It can apply similarly also to a 1st operation example. That is, a plurality of guides may be prepared for at least one of the parts W1 and W2, and when feature extraction based on certain guide data fails, feature extraction based on other guide data may be performed. In addition, a visual servo target condition based on the object guide G1 and the fixed guide SG1 and a visual servo target condition based on the object guide G2 and the fixed guide SG1 are set, respectively, immediately after the convergence determination of each visual servo. The target condition may be determined.

<Other variations>
In the above-described embodiment and its modifications, the guide setting unit 42 sets one guide position for the feature position extracted by a predetermined image processing algorithm (which may be specified by the user). . However, the guide setting unit 42 displays the extracted feature position (for example, an edge or the like) on the display unit 41, and selects an arbitrary feature position (for example, a part of the extracted edge or the like) from the extracted feature position. ) May be received from the user via the input unit 40. Then, the guide setting unit 42 may receive the designation of the guide position for the designated feature position and generate guide data. In this case, the guide data includes a guide position (converted reference position and direction component), a feature position specified by the user (information specifying the feature position, and image processing used to extract the feature position. Information on the algorithm). Of course, an arbitrary plurality of feature positions may be received from the extracted feature positions, and a guide position specification may be received for each specified feature position. By doing in this way, the flexibility of the setting of a characteristic position and a guide can be improved.

  Each processing unit in each illustrated flowchart is divided according to main processing contents in order to facilitate understanding of the processing of the teaching device 4 and the control unit 3. The present invention is not limited by the way of dividing the processing unit or the name. The processing of the teaching device 4 and the control unit 3 can be divided into more processing units depending on the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiment. In addition, it is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention. Various aspects of the provided aspects of the invention can also be employed. The present invention may be provided, for example, as a robot system having a robot, a control unit (control device), and a teaching device separately, or as a robot in which the function of the teaching device is included in the control unit of the robot. Alternatively, it may be provided as a robot system having a robot and a control unit (control device) including the function of the teaching device separately. The present invention can also be provided as a teaching method, a teaching device program, a robot control method, a robot control program, a robot system control method, a storage medium storing the program, and the like.

1: Robot system 2: Robot 3: Control unit 4: Teaching device 5: Imaging device 9: Computer 20: Body 21: Arm 21a: Joint 21b: Arm member 21c: Force sensor 22: Hand 24: Leg 25: Head 30: Teaching data receiving unit 31: Visual servo control unit 32: Feature extraction unit 33: Guide generation unit 34: Image acquisition unit 35: Teaching data storage unit 40: Input unit 41: Display unit 42: Guide setting unit 43: Teaching data generating unit 44: Teaching data transmitting unit 45: Model data storing unit 46: Guide data storing unit 47: Teaching data storing unit 90: Computing device 91: Main storage device 92: Auxiliary storage device 93: Communication I / F
94: Input device 95: Display device 96: Read / write device 451: ID
452: Guide position 453: Feature position 461: ID
462: Guide position 471: Process number 472: Teaching content E: Offset position F: Feature position G: Object guide SG: Fixed guide T: Worktable W: Workpiece, object, parts, model data

Claims (15)

A first guide setting unit for receiving settings of the first guide for the image data of the first object;
A second guide setting unit that receives a setting of the second guide for the image data of the second object;
A display unit for displaying the first guide and the second guide;
A teaching data generation unit that generates teaching data in which the first guide and the second guide indicate a predetermined positional relationship;
A teaching data transmitting unit that transmits the generated teaching data to a robot that performs visual servoing based on the teaching data;
A teaching device comprising: The teaching device according to claim 1,
The first guide setting unit accepts the setting of the first guide in association with the characteristics of the image data of the first object,
The second guide setting unit accepts the setting of the second guide in association with the characteristics of the image data of the second object,
The teaching data generation unit generates teaching data indicating the characteristics of the image data of the first object and the first guide, and the characteristics of the image data of the second object and the second guide.
A teaching device. The teaching device according to claim 2,
The first guide setting unit receives a plurality of settings of the first guide in association with a plurality of features of the image data of the first object,
The teaching data generation unit generates teaching data indicating a plurality of features of the image data of the first object and the plurality of first guides.
A teaching device. The teaching device according to claim 3,
The first guide setting unit extracts a plurality of features of image data of the first object using each of a plurality of image processes, and associates the plurality of first guides with each of the extracted features. Accept settings for
A teaching device. The teaching device according to claim 3,
The first guide setting unit extracts features of the image data of the first object using predetermined image processing, and specifies a plurality of features and a plurality of the specified features from the extracted features. Accept settings of the plurality of first guides to be associated with each;
A teaching device. A first guide setting unit for receiving settings of the first guide for the image data of the first object;
A second guide setting unit that receives the setting of the second guide for the imaging range of the imaging device;
A display unit for displaying the first guide and the second guide;
A teaching data generation unit that generates teaching data in which the first guide and the second guide indicate a predetermined positional relationship;
A teaching data transmitting unit that transmits the generated teaching data to a robot that performs visual servoing based on the teaching data;
A teaching device comprising: A teaching method for a teaching device, comprising:
A first guide setting step for accepting the setting of the first guide for the image data of the first object;
A second guide setting step for accepting the setting of the second guide for the image data of the second object;
A display step for displaying the first guide and the second guide;
A teaching data generation step for generating teaching data in which the first guide and the second guide indicate a predetermined positional relationship;
A teaching data transmission step of transmitting the generated teaching data to a robot that performs visual servoing based on the teaching data;
A teaching method characterized by comprising: A first guide setting unit for receiving settings of the first guide for the image data of the first object;
A second guide setting unit that receives a setting of the second guide for the image data of the second object;
A display unit for displaying the first guide and the second guide;
A teaching data generation unit that generates teaching data in which the first guide and the second guide indicate a predetermined positional relationship;
As a teaching data transmission unit that transmits the generated teaching data to a robot that performs visual servoing based on the teaching data,
A program characterized by operating a computer. Teaching data for storing teaching data indicating a predetermined positional relationship between a first guide set for the image data of the first object and a second guide set for the image data of the second object A storage unit;
An image acquisition unit for acquiring a captured image including the first object and the second object;
A first guide generating unit that generates the first guide for the image data of the first object included in the acquired captured image based on the teaching data;
A second guide generating unit that generates the second guide for the image data of the second object included in the acquired captured image based on the teaching data;
A visual servo control unit that performs control based on visual servo so that the generated first guide and the generated second guide have the predetermined positional relationship based on the teaching data;
A robot characterized by comprising: The robot according to claim 9,
In the teaching data, the first guide is shown in association with the feature of the image data of the first object, and the second guide is shown in association with the feature of the image data of the second object,
The first guide generation unit generates the first guide in association with the characteristics of the image data of the first object included in the acquired captured image based on the teaching data,
The second guide generation unit generates the second guide in association with the feature of the image data of the second object included in the acquired captured image based on the teaching data.
A robot characterized by that. The robot according to claim 10,
In the teaching data, a plurality of the first guides are shown in association with a plurality of features of the image data of the first object,
The first guide generation unit generates the first guide in association with any one of a plurality of features of image data of the first object included in the acquired captured image based on the teaching data. To
A robot characterized by that. A teaching data storage unit that stores teaching data indicating a predetermined positional relationship between a first guide set for the image data of the first object and a second guide set for the imaging range of the imaging device When,
An image acquisition unit that acquires a captured image including the first object and the second object from the imaging device;
A first guide generating unit that generates the first guide for the image data of the first object included in the acquired captured image based on the teaching data;
A second guide generating unit that generates the second guide for the image data of the second object included in the acquired captured image based on the teaching data;
A visual servo control unit that performs control based on visual servo so that the generated first guide and the generated second guide have the predetermined positional relationship based on the teaching data;
A robot characterized by comprising: A robot control method,
Teaching data storage in which a first guide set for the image data of the first object and a second guide set for the image data of the second object provide teaching data indicating a predetermined positional relationship Steps,
An image acquisition step of acquiring a captured image including the first object and the second object;
A first guide generating step for generating the first guide for the image data of the first object included in the acquired captured image based on the teaching data;
A second guide generating step for generating the second guide for the image data of the second object included in the acquired captured image based on the teaching data;
A visual servo control step for performing control based on visual servo so that the generated first guide and the generated second guide are in the predetermined positional relationship based on the teaching data;
A control method characterized by comprising: A first guide setting unit for receiving settings of the first guide for the image data of the first object;
A second guide setting unit that receives a setting of the second guide for the image data of the second object;
A display unit for displaying the first guide and the second guide;
A teaching data generation unit that generates teaching data in which the first guide and the second guide indicate a predetermined positional relationship;
A teaching data storage unit for storing the generated teaching data;
An image acquisition unit for acquiring a captured image including the first object and the second object;
A first guide generating unit that generates the first guide for the image data of the first object included in the acquired captured image based on the teaching data;
A second guide generating unit that generates the second guide for the image data of the second object included in the acquired captured image based on the teaching data;
A visual servo control unit that performs control based on visual servo so that the generated first guide and the generated second guide have the predetermined positional relationship based on the teaching data;
A robot system characterized by comprising: A control method for a robot system,
A first guide setting step for accepting the setting of the first guide for the image data of the first object;
A second guide setting step for accepting the setting of the second guide for the image data of the second object;
A display step for displaying the first guide and the second guide;
A teaching data generation step for generating teaching data in which the first guide and the second guide indicate a predetermined positional relationship;
A teaching data storage step for storing the generated teaching data;
An image acquisition step of acquiring a captured image including the first object and the second object;
A first guide generating step for generating the first guide for the image data of the first object included in the acquired captured image based on the teaching data;
A second guide generating step for generating the second guide for the image data of the second object included in the acquired captured image based on the teaching data;
A visual servo control step for performing control based on visual servo so that the generated first guide and the generated second guide are in the predetermined positional relationship based on the teaching data;
A control method characterized by comprising:

Images