JP2015071206A - Control device, robot, teaching data generation method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control device, a robot, a teaching data generation method, and a program capable of reducing a teaching cost for a robot operation on a holding operation and an operation on a workpiece by a held tool.SOLUTION: A control device 100 comprises: an image recognition unit 110 for performing image recognition processing; and a processing unit 130 for performing specific processing of position attitude data indicating the position attitude of a holding part 310 and generation processing of teaching data of the robot 300. Moreover, the image recognition unit 110 performs image recognition processing on a picked-up image of the holding state, in which the hand of a teacher holds a tool. On the basis of the result of the image recognition processing of the holding state, moreover, the processing unit 130 creates the teaching data containing the position attitude data of the holding part 310 in the holding state.

Description

  The present invention relates to a control device, a robot, a teaching data generation method, a program, and the like.

  In order to make the robot perform a desired action, for example, how to change the joints of the robot and how to move the robot's hand (TCP: tool center point) to which position It is necessary to give motion commands to the robot one by one. Such an operation command is described by a human, and after giving the operation command to the robot, it is necessary for the human to operate the robot while confirming the operation command. This procedure that humans perform is called "teaching".

  As a specific method of teaching, teaching playback (Teaching playback) in which a robot performs predetermined movements manually and others, and memorizing the movement at that time, or a teacher moves a robot arm or the like by hand. Direct teaching that teaches robot operation or offline teaching that teaches using a simulation on a computer without using a real robot is known. In addition, as inventions related to these robot operation teaching methods, there are conventional techniques described in Patent Document 1 and Patent Document 2.

JP 2001-38660 A JP-A-6-250730

  The teaching operation of robot operation takes a long time and a great amount of labor, which is a heavy burden on the teacher.

  According to some aspects of the present invention, it is possible to provide a control device, a robot, a teaching data generation method, a program, and the like that can reduce the teaching cost of the robot operation regarding the gripping work and the work on the work by the gripped tool be able to.

  An aspect of the present invention includes an image recognition unit that performs image recognition processing, and a processing unit that performs processing for specifying position and orientation data representing the position and orientation of the gripping unit of the robot, and processing for generating teaching data of the robot. The image recognition unit performs the image recognition process on a captured image in a gripped state in which a teacher's hand grips the tool, and the processing unit is based on a result of the image recognition process in the gripped state. , And a control device that generates the teaching data including the position and orientation data of the gripper in the gripping state.

  In one embodiment of the present invention, a gripping state in which a teacher's hand grips a tool is imaged by a plurality of imaging units. Then, image recognition processing is performed on the acquired captured image, and position and orientation data representing the position and orientation of the gripping unit when the gripping unit of the robot corresponding to the teacher's hand is in the gripping state are specified. Furthermore, robot teaching data including the specified position and orientation data is generated.

  Therefore, it is possible to reduce the teaching cost of the robot operation for the gripping work and the work on the work by the gripped tool.

  Moreover, in one aspect of the present invention, the image recognition unit includes a first captured image in the gripping state and a work state in which a work is performed on the workpiece with the tool gripped by the teacher's hand. Two captured images, and performs a first image recognition process on the first captured image and a second image recognition process on the second captured image, and the processing unit includes the first image recognition process. Based on the result of the image recognition process and the result of the second image recognition process, the specifying process of the position and orientation data of the gripper in the working state may be performed.

  As a result, even if it is difficult to specify the relative position and orientation relationship between the teacher's hand and the tool only from the captured image of the work state, It becomes possible to specify the position and orientation relationship.

  In one embodiment of the present invention, the processing unit obtains a relative position and orientation relationship between the hand of the teacher, the tool, and a work that is a work target of the robot, and the obtained position and orientation Based on the relationship, the specifying process of the position / orientation data of the grip portion may be performed in a working state in which the tool gripped by the hand of the teacher is working on the workpiece.

  This makes it possible to obtain the relative position and orientation relationship between the teacher's hand, the tool, and the workpiece.

  In one aspect of the present invention, the image recognition unit performs contour detection processing on at least one of the tool and the workpiece in the image recognition processing, and based on the result of the contour detection processing, A relative position / orientation relationship between at least one of the tool and the workpiece and the hand of the teacher may be obtained.

  Accordingly, it is possible to specify the relative position and orientation of at least one of the tool and the workpiece and the teacher's hand based on the contour line of at least one of the tool and the workpiece.

  In one aspect of the present invention, the image recognition unit performs a marker detection process on a marker provided on at least one of the hand of the teacher and the tool in the image recognition process, and the marker detection process. Based on the result, the relative position and orientation relationship between the hand of the teacher and the tool may be obtained.

  This makes it possible to more reliably obtain the relative position and orientation relationship between the hand and the tool.

  In one aspect of the present invention, the processing unit is configured to specify the position / orientation data of the gripper based on a correspondence relationship between the hand model of the teacher and the gripper of the robot. May be performed.

  This makes it possible to convert the relative position / posture relationship between the teacher's hand, the tool, and the workpiece into the relative position / posture relationship between the gripping portion of the robot, the tool, and the workpiece.

  In one aspect of the present invention, the image recognition unit performs a feature point detection process on the hand of the instructor in the image recognition process to generate a model of the hand, and the generated hand Based on the model, the specifying process of the position and orientation data of the grip portion may be performed.

  Thus, it is possible to specify the position and orientation of the teacher's hand by using the model of the teacher's hand.

  In the aspect of the invention, the image recognition unit performs a first image recognition process on the first captured image in the gripping state, and the processing unit is based on a result of the first image recognition process. The relative position and orientation relationship between the teacher's hand and the tool is obtained, and the image recognition unit performs work on the workpiece with the tool held by the teacher's hand. A second image recognition process is performed on the second captured image in a working state, and the processing unit is configured to perform a process based on the first position and orientation relationship and the result of the second image recognition process. The relative second position / orientation relationship between the hand, the tool, and the workpiece is obtained, the correspondence between the model of the hand of the teacher and the grip portion of the robot, and the obtained second position. Based on the posture relationship, A relative third position / orientation relationship among the gripper, the tool, and the workpiece of the robot is obtained, and the position / orientation data of the gripper in the working state is obtained based on the third position / orientation relationship. May be specified.

  This makes it possible to accurately specify the relative position and orientation relationship between the gripping part of the robot, the tool, and the workpiece in the working state.

  In one aspect of the present invention, the processing unit performs operation information of a jog tool by the teacher after performing the specifying process of the position and orientation data of the gripping unit based on a result of the image recognition process. Based on the above, the correction processing of the position and orientation data of the grip portion may be performed.

  As a result, the teaching data can be corrected when the position and orientation of the gripping portion of the robot included in the teaching data is deviated from the target position and orientation expected by the teacher.

  In another aspect of the present invention, an image recognition unit that performs image recognition processing and a processing unit that performs processing for specifying position and orientation data representing the position and orientation of the gripping unit of the robot, and processing for generating teaching data of the robot The image recognition unit performs the image recognition process on a captured image in a working state in which a work is performed on a work with a tool held by a teacher's hand, and the processing unit performs the work The present invention relates to a control device that generates the teaching data including the position and orientation data of the gripping unit in the working state based on the result of the image recognition processing in the state.

  As a result, in the working state, it is possible to cause the gripping portion of the robot to take the position and orientation intended by the teacher.

  In another aspect of the present invention, an image recognition unit that performs image recognition processing and a processing unit that performs processing for specifying position and orientation data representing the position and orientation of the gripping unit of the robot, and processing for generating teaching data of the robot The image recognition unit performs the image recognition process on a captured image in a gripped state in which a teacher's hand grips the tool, and the processing unit is a result of the image recognition process in the gripped state. And the robot that generates the teaching data including the position and orientation data of the gripping part in the gripping state.

  In another aspect of the present invention, a gripping state in which the teacher's hand grips the tool is captured by the imaging unit, and the robot gripping unit corresponding to the teacher's hand is based on the acquired captured image. The present invention relates to a teaching data generation method for specifying position and orientation data representing the position and orientation of the gripping part when in the gripping state, and generating teaching data for the robot including the specified position and orientation data.

  Another aspect of the present invention relates to a program that causes a computer to function as each of the above-described units.

The system configuration example of this embodiment. FIG. 2A to FIG. 2C are explanatory diagrams for simulating robot operation by a teacher. The flowchart explaining the flow of the process which simulates a holding | grip state and produces | generates teaching data. The flowchart explaining the flow of the process which simulates a working state and produces | generates teaching data. The flowchart explaining the flow of the process which produces | generates teaching data simulating a holding | grip state and a working state. Explanatory drawing of the generation process of a hand model, and the detection process of the outline of a tool and a workpiece | work. FIG. 7A to FIG. 7C are explanatory diagrams of hand model generation processing using markers. Explanatory drawing of the relative position orientation relationship of a hand, a tool, and a workpiece | work. FIG. 9A to FIG. 9C are explanatory diagrams of the correspondence relationship between the hand and the grip portion. Explanatory drawing of the correction | amendment process using a jog tool. FIG. 11A and FIG. 11B are explanatory diagrams of an example of the robot of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Technique of this embodiment In recent years, robots have become multi-axis and their hands are becoming dexterous. And the operation | work etc. which hold | grip a workpiece | work and a tool by the holding part of a robot are actually performed.

  In order to cause the robot to perform such a gripping operation, it is necessary to teach the robot to perform a gripping operation. For example, in the method disclosed in Patent Document 1 described above, teaching work is performed by a jog base method using a teaching pendant. This teaching method is one of teaching playback.

  Specifically, in the method disclosed in Patent Document 1, in order to teach the position and orientation of a state where a tool or the like is gripped by the gripping part, a dedicated dummy jig is manufactured in advance, and the dummy jig and the gripping part (end It is necessary to replace the effector). Then, in a dedicated state where only teaching is performed, a teacher (human) makes full use of the teaching pendant or remote controller and repeats jogging a considerable number of times to teach the position and orientation of the jig tip. However, such teaching work takes a long time and a lot of labor, which is a heavy burden on the teacher.

  As another teaching playback, there is a method called direct teaching. Direct teaching is a method in which a teacher directly holds a gripping part or an arm of a robot with a hand and moves it to a target position to store a series of operations in the robot.

  Direct teaching is very useful and can reduce the number of teaching steps because it can intuitively teach the target position (trajectory) of the robot rather than teaching using the teaching pendant. However, direct teaching has a problem that teaching cannot be performed unless the actual robot is used.

  On the other hand, before the actual robot is actually operated, a simulation of the robot operation on a PC (Personal Computer) is performed to check whether the robot operates safely and as expected. At this time, the operation of the robot is determined without using the actual robot. Further, the robot operation scenario data determined on the PC is used as teaching data for the actual robot. Such a teaching method that does not use the actual robot is called off-line teaching. Since offline teaching does not use the actual robot, it is necessary to examine the robot operation in advance when the actual robot has not been completed, or in an office away from the work site where the robot is placed. This is particularly useful when it is desired to determine the robot motion.

  However, offline teaching is often performed by a teacher who inputs a numerical value of a target coordinate of a robot, but this costs as much as teaching using a teaching pendant.

  Therefore, in practice, it is desired that intuitive teaching such as direct teaching can be performed even when performing offline teaching.

  Therefore, the control device or the like of the present embodiment described below performs teaching by a method similar to direct teaching while being offline teaching without using a robot. Specifically, the state in which the tool is gripped by the teacher's hand is imaged, and the position and orientation of the grip portion of the robot is taught based on the captured image. Therefore, in this embodiment, it is not necessary to use a teaching tool or the like, and it is not necessary to use an actual robot. This reduces teaching costs (teaching man-hours and preparation costs) in the robot gripping operation.

  Here, the teaching cost refers to the teaching man-hour required for the actual teaching work, the preparation cost necessary for performing the teaching work, and the like. In addition, the preparation cost refers to, for example, the cost (man-hour, cost) required for designing and manufacturing a jig used for teaching.

  Next, a configuration example of the control device 100 of this embodiment is shown in FIG. The control apparatus 100 according to the present embodiment performs processing for specifying position and orientation data representing the position and orientation of the image recognition unit 110 that performs image recognition processing and the gripping unit 310 of the robot 300, and performs processing for generating teaching data for the robot 300. And a processing unit 130.

  Then, the image recognition unit 110 performs image recognition processing on a captured image in a gripped state in which a teacher's hand grips the tool. Here, the image recognition processing is processing for recognizing an object appearing in a captured image by performing feature point detection processing, contour detection processing, or the like on the captured image. A specific example of the image recognition process will be described later.

  Further, the processing unit 130 generates teaching data including the position and orientation data of the gripping unit 310 in the gripping state based on the result of the image recognition processing in the gripping state, and outputs the generated teaching data to the robot 300.

  The functions of the image recognition unit 110 and the processing unit 130 can be realized by hardware such as various processors (such as a CPU) and ASIC (such as a gate array), a program, and the like. Moreover, the control apparatus 100 is not limited to the structure of FIG. 1, and various deformation | transformation implementations, such as abbreviate | omitting some of these components and adding another component, are possible.

  Here, the flow of processing will be described with reference to the flowcharts of FIGS. 2 (A), 2 (B) and 3. First, as shown in FIG. 2A, the gripping state in which the hand HD of the teacher TP grips the tool TL is imaged by a plurality of imaging units (CM1 to CM3) (S101). Then, image recognition processing is performed on the acquired captured image (S102), and as shown in FIG. 2B, when the gripping part HP of the robot RB corresponding to the hand HD of the teacher TP is in the gripping state. The position and orientation data representing the position and orientation of the gripper HP are specified (S103). Further, teaching data of the robot RB including the specified position / orientation data is generated (S104).

  That is, in the gripping state, the relative position and orientation relationship between the gripping portion HP of the robot RB and the tool TL (hereinafter referred to as a relative position and orientation relationship) is the relative position between the hand HD of the teacher TP reflected in the captured image and the tool TL. Teaching data that is the same as the posture relationship is generated.

  Here, the tool is a tool for performing work such as machining on a workpiece or work target, and refers to a tool such as a drill or a screwdriver, for example. Note that the tool may be gripped by the gripper 310 or may be provided directly at the end point of the arm of the robot 300.

  The gripping state refers to a state where the gripping unit 310 of the robot 300 or the teacher's hand is gripping the tool (or workpiece). For example, FIG. 2A shows a state when the teacher TP is in a gripping state.

  The captured image refers to an image obtained by capturing with the image capturing unit 200. The captured image may be an image stored in an external storage unit or an image acquired via a network. In the present embodiment, it is desirable to use a plurality of imaging units 200 in order to specify the position and orientation of the teacher's hand or work in the three-dimensional space. For example, three imaging units 200 are desirable, and at least two imaging units are necessary.

  The position / orientation data refers to data representing the position and orientation of the gripper 310. The position and orientation data is data representing position and orientation vectors such as VT1 and VT2 in FIG.

  The teaching data is data used to teach the robot 300 the operation performed by the robot 300, for example, scenario data. The scenario data is data that defines the processing performed by the robot 300. More specifically, the scenario data refers to data in which the processing performed by the robot 300 is described in chronological order.

  According to this, it is not necessary to cause the actual machine of the robot 300 to take the position and orientation that the robot 300 wants to teach, and it is possible for the instructor himself to take the position and orientation that the robot 300 wants to teach. Furthermore, it is not necessary for the teacher to repeat jogging for inputting detailed numerical values and the like a considerable number of times.

  As described above, this method is performed by the teacher (human) instead of the operation performed by the actual machine of the robot 300 by direct teaching or other teaching playback, and is performed using the actual machine of the robot 300. The labor can be further reduced than direct teaching. Therefore, it is possible to reduce the teaching cost of the robot operation for the gripping work and the work on the work by the gripped tool.

  In the present embodiment, in particular, in a state where the robot 300 actually performs a work on a workpiece using a tool (working state), it is desired that the gripper 310 of the robot 300 takes the position and orientation intended by the teacher.

  Therefore, in the present embodiment, teaching may be performed by simulating the gripping portion 310 of the robot 300 in a working state with a teacher's hand. That is, the teacher's own hand is placed in the same position and orientation as the position and orientation desired to be held by the gripping unit 310 of the robot 300 in the working state, and the teacher's hand at this time is imaged and teaching is performed based on the acquired captured image. I do.

  Specifically, the image recognition unit 110 performs image recognition processing on a captured image in a working state in which work is performed on a work with a tool held by a teacher's hand. Then, the processing unit 130 generates teaching data including the position and orientation data of the gripping unit 310 in the working state based on the result of the image recognition processing in the working state.

  Here, the flow of processing will be described with reference to the flowcharts of FIGS. 2B, 2C, and 4. FIG. For example, it is assumed that the state shown in FIG. 2B is the working state of the robot RB. Here, the working state is a state in which the gripping part HP of the robot RB or the hand HD of the teacher TP grips the tool TL and works on the workpiece WK with the tool TL. That is, it can be said that the working state is a special state of the gripping state. Specifically, FIG. 2B shows an operation in which the robot RB inserts the tip of the driver (tool TL) into the screw hole of the screw (work WK).

  Then, as shown in FIG. 2C, the teacher TP simulates the working state of the robot RB shown in FIG. This state is imaged by a plurality of imaging units CM1 to CM3 (S201), and an image recognition process is performed on the acquired captured image (S202), and the hand HD of the instructor TP, the tool TL, and the work WK in the working state. The relative position and orientation are specified (S203). Then, in the working state, the relative position and orientation relationship between the gripping portion HP of the robot RB, the tool TL, and the work WK is the same as the relative position and orientation relationship between the hand HD of the teacher TP, the tool TL, and the work WK shown in the captured image. Such teaching data is generated (S204).

  If the robot 300 is operated using the teaching data generated in this way, it is possible to cause the gripper 310 of the robot 300 to take the position and orientation intended by the teacher in the working state.

  Here, as described above, in order to specify the position and orientation of the gripping unit 310 in the working state, it is necessary to specify the relative position and orientation relationship between the teacher, the hand, the tool, and the workpiece. However, the captured image obtained by simulating the work state with the teacher's hand may include obstacles placed around the workpiece, so the relative position and orientation relationship between the teacher's hand and the tool It may be difficult to identify.

  Therefore, in this embodiment, after specifying the relative position / posture relationship between the teacher's hand and the tool, the relative position / posture relationship between these and the workpiece is specified. That is, the process of specifying the relative position and orientation relationship between the teacher, the hand, the tool, and the work is divided into two.

  Specifically, the image recognition unit 110 acquires a first captured image in a gripping state and a second captured image in a working state in which work is performed on a workpiece with a tool gripped by a teacher's hand. The first image recognition process for the first captured image and the second image recognition process for the second captured image are performed. Then, the processing unit 130 performs position / posture data specifying processing of the gripping unit 310 in the working state based on the result of the first image recognition process and the result of the second image recognition process.

  For example, as shown in FIG. 2A, the teacher TP simulates the gripping state, images the situation at this time, and acquires the first captured image. Then, based on the acquired first captured image, the relative position and orientation relationship between the hand HD of the teacher TP and the tool TL is specified. Note that the first image recognition process is a process for recognizing the teacher's hand and tool shown in the first captured image at this time.

  Next, as shown in FIG. 2C, the teacher TP simulates the working state, images the situation at this time, and obtains a second captured image. Then, based on the acquired second captured image, the relative position and orientation relationship between the hand HD of the teacher TP, the tool TL, and the work WK is specified. Since the relative position and orientation relationship between the hand HD of the teacher TP and the tool TL has already been specified from the first captured image, only the relative position and orientation relationship with the workpiece WK need be specified here. The second image recognition process is a process for recognizing at least the work shown in the second captured image at this time.

  Then, based on the above processing result, the position and orientation of the gripping part HP of the robot RB in the working state as shown in FIG.

  As a result, even if it is difficult to specify the relative position / posture relationship between the teacher's hand and the tool only from the captured image of the work state, the relative position / posture relationship between the teacher's hand, the tool, and the workpiece in the work state can be determined. It becomes possible to specify.

  Here, the flow of processing when performing the above-described first image recognition processing and second image recognition processing will be described using the flowchart of FIG.

  First, as shown in FIG. 2A, the imaging unit 200 captures the gripping state in which the teacher grips the tool, and the image recognition unit 110 acquires the first captured image in the gripping state (S301). A first image recognition process is performed on the first captured image (S302).

  Next, the processing unit 130 obtains a relative first position / orientation relationship between the teacher's hand and the tool based on the result of the first image recognition process (S303).

  Then, as shown in FIG. 2C, the imaging unit 200 captures an image of a work state in which work is performed on a work with a tool held by a teacher's hand. Next, the image recognition unit 110 acquires a second captured image in the working state (S304), and performs a second image recognition process on the second captured image (S305).

  Further, the processing unit 130 obtains a relative second position / orientation relationship between the teacher's hand, the tool, and the workpiece based on the first position / orientation relationship and the result of the second image recognition process (S306). ).

  Thereafter, the processing unit 130 acquires the correspondence between the model of the teacher's hand and the gripping unit 310 of the robot 300 (S307). Then, the processing unit 130, based on the acquired correspondence and the obtained second position and orientation relationship, the relative third position of the gripping unit 310 of the robot 300, the tool, and the workpiece in the working state. The posture relationship is obtained (S308).

  Finally, the processing unit 130 identifies position / posture data of the gripping unit 310 in the working state based on the third position / posture relationship (S309), and teach data including the position / posture data of the gripping unit 310 in the working state. Generate (S310).

  This makes it possible to accurately specify the relative position and orientation relationship between the gripping portion 310 of the robot 300 in the working state, the tool, and the workpiece. Details of each process will be described later.

  As a comparative example, there is an invention related to a robot motion teaching device disclosed in Patent Document 2 described above. In the invention of Patent Document 2, as shown in FIG. 1 of the same document, the teacher P works on the work (W1, W2), images the situation, and positional information of the work (W1, W2). Are acquired in time series, and an operation program of the robot 1 is generated based on the acquired position information.

  As described above, in the invention of Patent Document 2, only the position information of the workpiece is specified from the captured image, and the operation program (teaching data) of the robot 1 is generated based on the acquired position information of the workpiece.

  In step S5 of the flowchart of FIG. 2 of Patent Document 2, the position and orientation of the workpiece are converted into a position and orientation in the robot coordinate system based on data relating to the installation position of the robot and the gripping position of the hand (gripping unit). Yes. Here, the gripping position of the hand is a relative position and orientation relationship between the hand and the work (or tool) in the gripping state, and data regarding the gripping position of the hand is stored in the memory 6 in advance. Furthermore, in the invention of Patent Document 2, it is assumed that the hand holding position is uniquely determined as a premise. In other words, it is assumed that the direction in which the workpiece is gripped is always a certain direction (for example, from above), and the position where the workpiece and the hand are in contact is also a certain position (for example, the center position on both sides of the workpiece). It is done. In this case, since the data on the gripping position of the hand stored in the memory 6 correctly represents the actual gripping state, the position / posture of the workpiece is the same as the position / posture in the robot coordinate system. It is possible to convert.

  However, in practice, the gripping position of the hand often varies depending on circumstances due to the structure of the robot and the route setting. For example, even if the position and orientation of the workpiece are the same, the workpiece may be gripped from above or may be gripped from the side. In addition, the contact position between the workpiece and the hand usually changes depending on the type of the workpiece. As described above, when the gripping position of the hand changes each time, the data of the gripping position of the hand stored in the memory 6 does not necessarily accurately represent the actual gripping state. Therefore, if the gripping position is not uniquely determined, in step S5 of FIG. 2 of Patent Document 2, coordinate conversion that does not match the actual gripping state is performed, and teaching data as expected by the teacher is obtained. It may not be generated. In other words, the invention of Patent Document 2 cannot teach how the gripper should grip the tool.

  On the other hand, in this embodiment, not only the position information of the workpiece but also the relative position and orientation relationship between the teacher's hand (the gripping portion 310 of the robot 300) and the tool is specified from the captured image. Therefore, the specified relative position / posture relationship correctly represents the actual gripping state. Then, teaching data is generated based on the specified relative position and orientation relationship. As a result, the position / posture relationship between the gripper 310 and the tool in the gripping state can be taught as intended by the teacher. That is, in this embodiment, unlike the invention of Patent Document 2, it is possible to teach the robot various gripping positions and postures suitable for the situation.

2. Details of Processing Next, details of each processing of the present embodiment will be described.

  First, the image recognition process will be described. Each image recognition process described below can be performed in any of step S102 of the flowchart of FIG. 3 described above, step S202 of the flowchart of FIG. 4, and step S302 and step S305 of the flowchart of FIG.

  In the image recognition process, the image recognition unit 110 performs a feature point detection process on the teacher's hand to generate a hand model, and based on the generated hand model, the position / orientation data specifying process of the grip unit 310 I do.

  Specifically, for example, in the captured image PIM as shown in FIG. 6, a plurality of feature points (SP1 to SP8) of the teacher's hand HD are detected, and these feature points are connected to determine the model MD of the hand HD. Generate.

  Here, the feature point refers to a point that can be clearly observed from the image. For example, in FIG. 6, it is represented by points SP1 to SP8. In this example, a corner detection method or the like is used as a feature point detection method, but other general corner detection (eigenvalue, FAST feature detection) may be used, and a representative example is SIFT (Scale invariant feature transform). Local feature descriptors, SURF (Speeded Up Robust Feature), etc. may be used.

  The hand model refers to a two-dimensional figure representing a rough outline of a teacher's hand, three-dimensional CAD (Computer Aided Design) model data, or the like. For example, in FIG. 6, the hand model is represented by MD. Note that CAD is also called computer-aided design and refers to designing using a computer. CAD model data refers to data representing a design object designed in CAD.

  Thus, the position and orientation of the teacher's hand is specified based on the model of the teacher's hand, and the position and orientation of the grip unit 310 of the robot 300 is specified based on the specified position and orientation of the teacher's hand. Etc. becomes possible.

  Further, the image recognition unit 110 performs contour detection processing on at least one of the tool and the workpiece in the image recognition processing, and based on the result of the contour detection processing, at least one of the tool and the workpiece and the teacher's hand. The relative position and orientation relationship is obtained.

  Specifically, for example, the contour line OL1 of the tool TL and the contour line OL2 of the workpiece WK are detected in the captured image PIM as shown in FIG.

  Here, the contour detection process refers to a process of extracting contour lines of an object such as a work or a tool shown on a captured image and specifying contour information. For example, a contour image is extracted by generating a binary image in which a region where the magnitude of the brightness gradient of the image is equal to or greater than a given value is blacked out and performing a thinning process on the binary image. Note that the contour information is information representing the contour of an object. The contour information is, for example, a coordinate position on the image of the start point and end point of the contour, a vector, its size, and the like.

  Accordingly, it is possible to specify the relative position and orientation between at least one of the tool and the workpiece and the teacher's hand based on the contour line of at least one of the tool and the workpiece.

  Also, if you want to more reliably determine the relative position and orientation relationship between the teacher's hand and the tool, pre-apply a marker to the teacher's hand and the tool, and based on the detection result of the attached marker, The relative position and orientation relationship between the hand and the tool may be obtained.

  That is, in the image recognition process, the image recognition unit 110 performs a marker detection process for a marker provided on at least one of the teacher's hand and tool, and based on the result of the marker detection process, the teacher's hand and tool The relative position / orientation relationship may be obtained.

  Specifically, for example, in the captured image PIM as shown in FIG. 7A, the first marker MK1 attached to the teacher's hand HD is detected, and the positions of these first markers MK1 are connected. A hand HD model MD similar to that shown in FIG. 6 is generated. Note that it is assumed that a plurality of round first markers MK1 as shown in FIG. The first marker MK1 changes in appearance (that is, the shape of the first marker MK1 shown in the captured image PIM) depending on the position and orientation of the hand of the attached teacher. Therefore, the position and orientation of the teacher's hand can be specified.

  Similarly, a triangular second marker MK2 as shown in FIG. 7C is attached in advance to the tool TL, and the tool TL is based on the detection result of the second marker MK2 in the captured image PIM. The contour line OL is detected. Even in this process, the position and orientation of the tool TL can be estimated based on the way the second marker MK2 appears in the captured image PIM. Based on these results, the relative position and orientation relationship between the teacher's hand and the tool is obtained.

  Here, the marker is a tangible object that can be used as a mark, a character, a figure, a symbol, a pattern, a three-dimensional shape, a combination thereof, or a combination of these and a color, and can be fixed to an object. . For example, it is a sticker, a sticker, a label, or the like, and has a shape such as MK1 in FIG. 7B or MK2 in FIG. It does not matter what the shape, color, pattern, etc. of the marker is, but a marker that is easily distinguishable from other regions, for example, an image or sticker made of a single red color is desirable.

  This makes it possible to more reliably obtain the relative position and orientation relationship between the hand and the tool.

  Next, details of the processing in step S306 in the flowchart of FIG. 5 will be described. The processing unit 130 obtains a relative position and orientation relationship between the teacher's hand, the tool, and the work that is the work target of the robot 300 (S306). Then, the processing unit 130 performs processing for specifying the position and orientation data of the gripping unit 310 in a working state in which the work is performed on the workpiece with the tool gripped by the teacher's hand based on the obtained position and orientation relationship. (S309).

  Here, the relative position and orientation relationship (relative position and orientation relationship) refers to the position and orientation of the other object in the local coordinate system for one object.

  For example, as shown in FIG. 8, the relative position and orientation relationship between the teacher's hand and the tool is obtained as a vector VT1 representing the position and orientation of the tool TL in the local coordinate system for the teacher's hand HD. Note that the origin of the local coordinate system of the teacher's hand HD is, for example, the center of gravity or the center position of the model MD of the hand HD described with reference to FIG.

  Similarly, the relative position and orientation relationship between the tool and the work is obtained as a vector VT2 representing the position and orientation of the work WK in the local coordinate system for the tool TL, as shown in FIG. The origin of the local coordinate system of the tool TL is, for example, the position of the center of gravity or the center position of the region surrounded by the contour OL1 of the tool TL described with reference to FIG.

  Then, it is possible to obtain the relative position / posture relationship between the teacher's hand, the tool, and the workpiece by using the two relative position / posture relationships. The method for obtaining the relative position and orientation relationship between the teacher's hand, the tool, and the workpiece is not limited to the method described above, and may be obtained, for example, based on the local coordinate system of the workpiece.

  Next, it is necessary to replace the relative position / posture relationship between the teacher's hand, the tool, and the work obtained by the above processing with the relative position / posture relationship between the grasping unit 310 of the robot 300, the tool, and the work.

  Therefore, in steps S307 to S309 in the flowchart of FIG. 5, the processing unit 130 specifies the position and orientation data of the gripping unit 310 based on the correspondence between the model of the teacher's hand and the gripping unit 310 of the robot 300. Process.

  For this purpose, it is necessary to obtain the correspondence between the teacher's hand and the gripping part 310 of the robot 300. In this embodiment, for example, the hand MD model MD shown in FIG. 9A and the robot gripping part HP shown in FIG. 9B are displayed on the image or in the three-dimensional virtual space. Overlay as in (C). Then, a size difference or a shape difference between the hand MD model MD and the robot gripping part HP is specified as information indicating the correspondence. Note that the processing for obtaining the correspondence between the hand and the gripper may be performed in advance and stored in the storage unit, or may be performed when the hand model is first generated.

  This makes it possible to replace the relative position / posture relationship between the teacher's hand, tool, and workpiece with the relative position / posture relationship between the gripper 310 of the robot 300, the tool, and the workpiece.

  Then, after performing the above processing, the position and orientation data of the grip unit 310 of the robot 300 in the gripping state or working state is specified. At this time, for example, in step S309 in FIG. 5, a process of performing coordinate conversion of the position / orientation of the gripper 310 represented by the local coordinate system of the gripper 310 to the position / orientation in the world coordinate system is performed. Then, teaching data including the specified position / orientation data is generated. Accordingly, the robot 300 can be operated according to the teaching data, and the robot 300 can perform the same movement as the operation performed by the teacher.

  However, as shown in FIG. 10, as a result of simulating the robot operation in accordance with the teaching data, the position / posture P1 of the robot gripper HP may be slightly shifted from the target position / posture P2 expected by the teacher. In FIG. 10, for the sake of explanation, the positions of P1 and P2 are greatly shifted, but actually, a shift of several centimeters to several millimeters or less is assumed.

  At this time, the processing unit 130 performs the position / orientation data specifying process of the gripping unit 310 based on the result of the image recognition processing, and then, based on the operation information of the jog tool by the teacher, Position / orientation data correction processing may be performed.

  For example, in the example of FIG. 10, the teacher operates the jog tool to change the position / posture P1 of the gripper HP of the robot to the target position / posture P2 expected by the teacher, and the position / posture of the gripper HP at this time The data is specified as position and orientation data in the working state. Here, performing such fine adjustment is referred to as correction processing for the position and orientation data of the grip portion 310.

  Here, the jog tool is a tool operated by a teacher to perform a jog (JOG) operation. Specifically, it is a teaching pendant or a PC. The jog operation is a movement operation in which the movement distance is not specified, and is an operation for manually controlling the operation and stop of the motor when using an actual robot. On the other hand, in the computer simulation, it is equivalent to the jog operation that the instructor fine-tunes the control parameters of the robot while checking the simulation result of the robot operation.

  As a result, when the position and orientation of the robot gripping part HP included in the teaching data is deviated from the target position and orientation expected by the teacher, the teaching data can be corrected. Note that the position / orientation data correction processing of the gripper 310 based on the operation information of the jog tool may be performed during simulation or may be performed when the robot is actually driven based on the generated teaching data.

  Further, the imaging unit (camera) 200 used in the above-described embodiment includes an imaging element such as a charge-coupled device (CCD) and an optical system. The imaging unit 200 can include a device (processor) used for image processing and the like. The imaging unit 200 is disposed at an angle such that a teacher or a work falls within the imaging range, for example, on a ceiling or a work table. Note that the work refers to an object to be processed in the robot 300. Further, as described above, in order to specify the position and orientation of the teacher's hand or the like, at least two imaging units 200 are required, and in this embodiment, three imaging units 200 are used. Then, the imaging unit 200 outputs information of the captured image to the control device 100 or the like. However, in the present embodiment, the information of the captured image is output as it is to the control device 100, but the present invention is not limited to this. For example, the imaging unit 200 may have some of the functions of the image recognition unit 110 and the processing unit 130 of the control device 100. In this case, information after image processing is performed on the captured image is output.

3. Robot Next, FIG. 11A and FIG. 11B show a configuration example of a robot 300 to which the control device 100 of this embodiment is applied. In both cases of FIG. 11A and FIG. 11B, the robot 300 includes a grip portion 310.

  Here, the gripping part (hand) 310 refers to a part used for gripping, lifting, lifting, or attracting a workpiece. The gripping part 310 may be a hand, a hook, a suction cup or the like. A plurality of hands may be provided for one arm.

  For example, in the robot of FIG. 11A, the robot body 300 (robot) and the control device 100 are configured separately. In this case, some or all of the functions of the control device 100 are realized by a PC, for example. In addition, some or all of the functions of the control device 100 may be realized by a server connected to the robot 300 via a network including at least one of wired and wireless.

  On the other hand, the robot of the present embodiment is not limited to the configuration of FIG. 11A, and the robot body 300 and the control device 100 may be configured integrally as shown in FIG. That is, the robot 300 may include the control device 100. Specifically, as illustrated in FIG. 11B, the robot 300 includes a robot main body (having an arm and a hand) and a base unit that supports the robot main body, and the control device 100 is stored in the base unit. It may be done. A robot 300 in FIG. 11B is configured such that wheels and the like are provided in the base unit portion so that the entire robot can move. 11A shows an example of a single arm type, the robot 300 may be a multi-arm type robot such as a double arm type as shown in FIG. 11B. The robot 300 may be moved manually, or may be moved by providing a motor for driving wheels and controlling the motor by the control device 100. Moreover, the control apparatus 100 is not necessarily provided in the base unit part provided under the robot 300 as shown in FIG.

  Note that the control device, the robot, the teaching data generation method, and the like according to the present embodiment may implement part or most of the processing by a program. In this case, a control device, a robot, a teaching data generation method, and the like according to the present embodiment are realized by a processor such as a CPU executing a program. Specifically, a program stored in the information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), or memory (card type). It can be realized by memory, ROM, etc. A processor such as a CPU performs various processes according to the present embodiment based on a program (data) stored in the information storage medium. That is, in the information storage medium, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit) Is memorized.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configuration and operation of the control device, robot, teaching data generation method and program are not limited to those described in the present embodiment, and various modifications can be made.

100 control device, 110 image recognition unit, 130 processing unit, 200 imaging unit,
300 robot, 310 gripping part

Claims (13)

An image recognition unit for performing image recognition processing;
A processing unit that performs processing for specifying position and orientation data representing the position and orientation of the gripping unit of the robot, and processing for generating teaching data of the robot;
Including
The image recognition unit
Performing the image recognition process on the captured image of the gripped state where the teacher's hand grips the tool,
The processor is
A control apparatus that generates the teaching data including the position and orientation data of the gripping unit in the gripping state based on a result of the image recognition processing in the gripping state. In claim 1,
The image recognition unit
A first captured image in the gripping state and a second captured image in a working state in which a work is performed on the workpiece with the tool gripped by the hand of the teacher, and the first captured image is acquired. Performing a first image recognition process for the second captured image and a second image recognition process for the second captured image,
The processor is
A control apparatus that performs the specifying process of the position / orientation data of the gripper in the working state based on a result of the first image recognition process and a result of the second image recognition process. . In claim 1 or 2,
The processor is
A relative position / orientation relationship between the hand of the teacher, the tool, and a work that is a work target of the robot is obtained, and the gripper is grasped by the hand of the teacher based on the obtained position / orientation relationship. A control apparatus that performs the specifying process of the position / orientation data of the gripper in a working state in which the tool is working on the workpiece. In claim 3,
The image recognition unit
In the image recognition processing, contour detection processing is performed for at least one of the tool and the workpiece, and at least one of the tool and the workpiece and the hand of the instructor are performed based on the result of the contour detection processing. A control device characterized by obtaining a relative position and orientation relationship with the control unit. In any one of Claims 1 thru | or 4,
The image recognition unit
In the image recognition processing, marker detection processing is performed for a marker provided on at least one of the hand of the teacher and the tool, and the hand of the teacher and the tool are performed based on a result of the marker detection processing. A control device characterized by obtaining a relative position and orientation relationship with the control unit. In any one of Claims 1 thru | or 5,
The processor is
A control apparatus that performs the specifying process of the position / orientation data of the gripper based on a correspondence relationship between the model of the hand of the teacher and the gripper of the robot. In any one of Claims 1 thru | or 6.
The image recognition unit
In the image recognition process, a feature point detection process is performed on the hand of the teacher to generate a model of the hand, and the position and orientation data of the gripping unit based on the generated model of the hand A control device that performs specific processing. In claim 1,
The image recognition unit
Performing a first image recognition process on the first captured image in the gripping state;
The processor is
Based on the result of the first image recognition process, a relative first position / orientation relationship between the hand of the teacher and the tool is obtained,
The image recognition unit
Performing a second image recognition process on a second captured image in a working state in which a work is being performed on the workpiece with the tool held by the teacher's hand;
The processor is
Based on the first position and orientation relationship and the result of the second image recognition process, a relative second position and orientation relationship between the hand of the teacher, the tool, and the workpiece is obtained.
Based on the correspondence between the model of the hand of the teacher and the grip portion of the robot and the obtained second position and orientation relationship, the grip portion of the robot in the working state, the tool, , To obtain a third position and orientation relationship relative to the workpiece,
The control device that identifies the position and orientation data of the gripper in the working state based on the third position and orientation relationship. In any one of Claims 1 thru | or 8.
The processor is
Based on the result of the image recognition process, the position / orientation data of the gripper is corrected based on the operation information of the jog tool by the teacher after performing the specifying process of the position / orientation data of the gripper. A control device that performs processing. An image recognition unit for performing image recognition processing;
A processing unit that performs processing for specifying position and orientation data representing the position and orientation of the gripping unit of the robot, and processing for generating teaching data of the robot;
Including
The image recognition unit
Performing the image recognition process on a captured image in a working state in which a work is performed on a workpiece with a tool held by a teacher's hand;
The processor is
A control apparatus that generates the teaching data including the position and orientation data of the gripping unit in the working state based on a result of the image recognition processing in the working state. An image recognition unit for performing image recognition processing;
A processing unit that performs processing for specifying position and orientation data representing the position and orientation of the gripping unit of the robot, and processing for generating teaching data of the robot;
Including
The image recognition unit
Performing the image recognition process on the captured image of the gripped state where the teacher's hand grips the tool,
The processor is
A robot that generates the teaching data including the position and orientation data of the gripping unit in the gripping state based on a result of the image recognition processing in the gripping state. The grasping state where the teacher's hand grasps the tool is imaged by the imaging unit,
Based on the acquired captured image, the position and orientation data representing the position and orientation of the gripping part when the gripping part of the robot corresponding to the hand of the teacher is in the gripping state are specified,
A teaching data generation method comprising generating teaching data of the robot including the specified position and orientation data. An image recognition unit for performing image recognition processing;
As a processing unit that performs processing for specifying position and orientation data representing the position and orientation of the gripping unit of the robot and performs processing for generating teaching data of the robot,
Make the computer work,
The image recognition unit
Performing the image recognition process on the captured image of the gripped state where the teacher's hand grips the tool,
The processor is
A program that generates the teaching data including the position and orientation data of the gripping unit in the gripping state based on a result of the image recognition processing in the gripping state.

Images