JP6042291B2 - Robot, robot control method, and robot control program - Google Patents

Description

  The present invention relates to a robot that performs work on a workpiece, a robot control method, and a robot control program.

  As a method for easily teaching a robot the work to be performed by the robot, a direct teaching in which a user directly moves a moving member (such as an arm) of the robot to teach the work has been put into practical use (for example, see Patent Document 1). Direct teaching is simpler compared with a method in which a user creates a program of work to be performed by a robot on a computer, and has an effect of improving work efficiency in small lot work in a factory or the like.

JP 2006-346827 A

  However, when this teaching method is applied to a robot for a purpose such as substituting a simple task in a general home, not only does it take time to teach, but also the trouble of moving the robot arm bothersomely becomes a problem.

  The present invention has been made in view of the above problems, and provides a robot, a robot control method, and a robot control program capable of teaching work contents while reducing physical load in a shorter time. For the purpose.

  The robot according to the present invention is a robot that works on a work, and includes a camera that generates a picked-up image by photographing, a working unit that performs work on the work, and photographing that includes an image of the work in the work teaching process. Based on the image, it has a configuration including a control unit that recognizes the work content of the work by the user and controls the work unit so as to perform the work content in the work execution process.

  With this configuration, the work content for the work by the user is photographed by the camera, and the work content is taught to the robot using the photographed image, so that the physical load on the user can be reduced in a shorter time. The contents can be taught to the robot.

  In the work execution step, the control unit may recognize the work based on a photographed image including the image of the work, and may control the work unit so as to perform the work content corresponding to the recognized work.

  With this configuration, the work to be performed can be specified and the necessary work can be performed in the work execution process.

  The control unit further recognizes the workpiece in the workpiece teaching step, and compares the workpiece recognized in the workpiece teaching step with the workpiece image included in the photographed image in the operation performing step in the operation performing step. Thus, the workpiece may be recognized.

  With this configuration, even when the work is placed at an arbitrary position and posture in the work execution process, the work taught in the work teaching process can be recognized and the work necessary for the work can be performed.

  The working unit may be capable of moving a workpiece, and the work content may be to move the workpiece.

  With this configuration, the robot can be instructed to move the workpiece in a shorter time with less physical load on the user.

  The control unit controls the working unit to perform a work of moving a plurality of workpieces so that a relationship between the positions and postures of the workpieces is a predetermined position and posture in the work performing step, and the work teaching step. The position and orientation of the work destination after the user has moved the work may be recognized. The position and posture of the movement destination may be a position and posture with reference to another workpiece that has already been moved.

  For example, in the process of completing a combination by combining multiple workpieces, if the position and orientation of the combination that has been completed so far (combination in the process of completion) are changed, the completion after such change is completed. If the robot can recognize the position and posture of the intermediate combination with respect to the robot, this configuration allows the change of the position and posture of the combination in the middle of completion.

  The working unit may be able to lift and move the work, and the work content may be to lift and move the work.

  With this configuration, the work of lifting and moving the workpiece can be taught to the robot in a shorter time and with less physical load on the user.

  The working unit may be capable of gripping and moving a workpiece, and the work content may be gripping and moving the workpiece.

  With this configuration, it is possible to teach the robot the work of gripping and moving the workpiece in a shorter time and with less physical load on the user.

  The working unit may be capable of gripping a workpiece, and the control unit recognizes a gripping position of the workpiece by a user's finger in the work execution step, and in the work execution step, the work unit The working unit may be controlled to grip the workpiece at the gripping position.

  With this configuration, the working unit can be controlled so as to grip the workpiece at the gripping position when the user grips the workpiece.

  The working unit may include two finger members, and may be a finger unit that grips a workpiece by changing a distance between the two finger members. The control unit may be a user in the work teaching step. The gripping position of the workpiece by the two fingers may be recognized, and the working unit may be controlled so that the working unit grips the workpiece at the gripping position in the work execution step.

  With this configuration, the working unit can be controlled so as to grip the workpiece at the gripping position when the user grips the workpiece with two fingers.

  The control unit recognizes the gripping position of the workpiece by the user's finger in the work teaching step, and the position and orientation of the workpiece movement destination when the user releases the gripping of the workpiece. The work part may be controlled such that the part grips the work at the gripping position, moves the work to the position of the movement destination, and releases the gripping of the work at the position and posture of the movement destination.

  With this configuration, the user can teach the work of gripping the work and moving it to a desired destination, and the working unit can reproduce the work.

  The robot control method of the present invention is a robot control method for working on a workpiece, and includes a shooting process for generating a shot image by shooting, and a user's work on the workpiece based on the shot image including the image of the workpiece. It has a configuration including a work teaching process for recognizing contents and a work execution process for controlling the working unit to perform the work contents.

  With this configuration, since the work content of the work by the user is photographed, and the work content is taught to the robot using the photographed image, the physical burden on the user can be reduced in a shorter time, and the work content can be reduced. Can teach robots.

  A robot control program according to the present invention includes a camera that generates a captured image by photographing and a working unit that performs work on a work. The user recognizes the work content of the work on the work based on the captured image including the image, and functions as a control unit that controls the work unit so as to perform the work content in the work execution process.

  With this configuration, the work content for the work by the user is photographed by the camera, and the work content is taught to the robot using the photographed image, so that the physical load on the user can be reduced in a shorter time. The contents can be taught to the robot.

  According to the present invention, since the work content of the work by the user is photographed by the camera, and the work content is taught to the robot using the photographed image, the physical load on the user can be reduced in a shorter time. Can teach robots the work content.

FIG. 1 is an external view showing a configuration of an arm type robot in an embodiment of the present invention. The block diagram which shows the structure of the arm type robot in embodiment of this invention The flowchart of the work teaching process in embodiment of this invention The flowchart of the work implementation process in embodiment of this invention (A) The figure which shows an example of the workpiece | work which DB image was memorize | stored in the memory | storage part in embodiment of this invention, and its coordinate system (b) The example of the data memorize | stored in the memory | storage part in embodiment of this invention is shown Figure The block diagram which shows the structure of the attitude | position estimation apparatus in the 1st Embodiment of this invention (A) The figure which shows an example of the imaging | photography direction of the object in embodiment of this invention (b) The figure which shows an example of the imaging | photography direction of the object in embodiment of this invention (A) The figure which shows the example of the prediction image when it image | photographs from the candidate image when it image | photographs from the 1st imaging | photography direction and 2nd imaging | photography direction in embodiment of this invention. (B) In embodiment of this invention The figure which shows the example of the prediction image when it image | photographs from the candidate image when it image | photographs from the 1st imaging | photography direction and the 2nd imaging | photography direction. The figure which shows operation | movement of the attitude | position estimation apparatus in the 1st Embodiment of this invention. The block diagram which shows the structure of the attitude | position estimation apparatus in the 2nd Embodiment of this invention. (A) The figure which shows the example of the image of the object reflected in the image image | photographed with the camera (b) The figure which shows the combination possibility of the candidate image of the object which concerns on image a, and the candidate image of the object which concerns on image a The figure which shows the structure of the indication target object display apparatus of the 1st Embodiment of this invention. (A) The figure which shows the example of the object in which the pointing direction in embodiment of this invention crosses (b) The figure which shows the example in which the pointing direction in embodiment of this invention cross | intersects the expansion area | region of an object The figure for demonstrating the instruction | indication target object detection by an instruction | indication target object detection part in case there exists a some object in the pointing direction in embodiment of this invention The figure which shows the relationship between the position where the pointing direction in the embodiment of this invention cross | intersects, and the irradiation position by light emission The flowchart which shows operation | movement of the instruction | indication target object display apparatus of the 1st Embodiment of this invention. The flowchart which shows operation | movement of the pointing object display apparatus of the 2nd Embodiment of this invention. The flowchart which shows operation | movement of the instruction | indication target object display apparatus of the 3rd Embodiment of this invention.

  Hereinafter, a robot according to an embodiment of the present invention will be described with reference to the drawings. The embodiment described below shows an example when the present invention is implemented, and the present invention is not limited to the specific configuration described below. In carrying out the present invention, a specific configuration according to the embodiment may be adopted as appropriate.

  FIG. 1 is an external view showing a configuration of an arm type robot according to an embodiment of the present invention. An arm type robot (hereinafter simply referred to as “robot”) 10 of the present embodiment is a robot that performs work on a workpiece W. The robot 10 includes a base 11 installed on the installation surface, an arm unit 12 connected to the base, a hand unit 13 provided at an end of the arm unit 12 opposite to the base 11, and an arm of the hand unit 13. The finger part 14 provided in the end on the opposite side to the part 12 is provided. The arm unit 12 includes a first arm 12A and a second arm 12B. The finger portion 14 includes two finger members 14A and 14B whose distances are variable.

  An anti-slip member for gripping a workpiece is affixed to at least the surfaces of the two finger members 14A and 14B facing each other, or anti-slip processing is performed. The arm unit 12, the hand unit 13, and the finger unit 14 correspond to a “working unit”.

  The robot 10 includes a first axis 21 that rotates around an axis perpendicular to the installation surface, a second axis 22 that rotates the first arm 12A around a horizontal axis, and a third axis that rotates the first arm 12A around the axis. 23, fixed to the first arm 12A, fixed to the fourth shaft 24 that rotates the second arm 12B in the horizontal direction, the fifth shaft 25 that rotates around the axis of the second arm 12B, and the second arm 12B, This is a 6-axis control robot including a sixth axis 26 that rotates the hand unit 13 in the horizontal direction.

  A camera 15 is installed in the hand unit 13 so that the optical axis faces the direction from the hand unit 13 toward the finger unit 14. The camera 15 is a monocular camera. As described above, the robot 10 including the camera 15 in the hand unit 13 is also referred to as a hand-eye robot. The camera 15 may be provided on either the first arm 12A or the second arm 12B, or the arm unit 12 of the robot 10 at a position and posture where the relationship between the position and posture relative to the robot 10 is known. The camera 15 may be provided independently of the movement, or a plurality of such cameras 15 may be provided.

  Hereinafter, the robot 10 of the present embodiment will be further described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the robot according to the present embodiment. As shown in FIG. 2, the arm part 12 and the hand part 13 are provided with actuators 121 and 131 for driving the respective axes. In addition, the finger unit 14 is provided with an actuator 132 for driving the finger members 14A and 14B, and grips a workpiece (work target object) by changing the distance between the finger member 14A and the finger member 14B. And release the grip. The camera 15 can capture still images and can also capture moving images (continuous still images) under the control of the camera control unit 151.

  Each of the arm unit 12 and the hand unit 13 is provided with an arm sensor 122 and a hand sensor 132 for detecting a driving state (a rotation angle of each axis). The arm sensor 122 and the hand sensor 132 also include a torque sensor for detecting torque applied to the arm unit 12 and the hand unit 13.

  The robot 10 further includes a control unit 16 and a storage unit 17. The arm unit 12, the hand unit 13, the finger unit 14, and the camera 15 operate according to instructions from the control unit 16. That is, the control unit 16 operates the actuators 121, 131, and 142 to drive the arm unit 12, the hand unit 13, and the finger unit 14 by operating the actuators 121, 131, and 142. By sending a signal instructing photographing and image transmission to the control unit 151, the camera control unit 151 controls the camera 15 to cause the camera 15 to perform photographing and image transmission. An image captured by the camera 15 is sent to the control unit 16 via the camera control unit 151. Detection values of the arm sensor 122, hand sensor 132, and finger sensor 142 are also sent to the control unit 16.

  The control unit 16 includes a computer, and, based on a photographed image from the camera 15 and detection values of the arm sensor 122, hand sensor 132, and finger sensor 142, according to a predetermined control program, the arm unit 12, the hand unit 13, The finger unit 14 is controlled. The storage unit 17 stores information necessary for the operation described below. In particular, the storage unit 17 stores information on the work taught in the work teaching process described below and the contents of the work taught in the work teaching process.

  Hereinafter, a process of teaching a work to the robot 10 (work teaching process), a process of teaching work on the work (work teaching process), and a process of performing work on the work (work execution process) will be described. In the present embodiment, an example will be described in which a work having a specific shape is gripped and moved to a predetermined place.

(Work teaching process)
In the present embodiment, the teaching of a workpiece is to teach a three-dimensional shape of the workpiece as information on the workpiece to be worked. In the teaching of the workpiece, the workpiece surface texture and the workpiece surface color may be further taught as workpiece information.

  Various known methods can be adopted as a method for teaching the workpiece to the robot 10. For example, the shape information of the work to be taught may be taken into the robot 10 using 3D CAD data or the like. In this case, the robot 10 stores the three-dimensional CAD data in the storage unit 17 as work information. Alternatively, a solid shape may be measured on the workpiece, and the solid shape data obtained by the measurement may be taken into the robot 10. In this case, the robot 10 stores the solid shape data in the storage unit 17 as work information.

  Since the robot 10 according to the present embodiment includes the camera 15 in the hand unit 13, the robot 15 is used to teach a workpiece as follows. First, the user places a work to be taught at a position where the robot 10 can recognize. The robot 10 drives the arm unit 12, the hand unit 13, and the camera 15 under the control of the control unit 16, and causes the camera 15 to photograph the workpiece from a plurality of directions.

  The control unit 16 recognizes the workpiece from the captured image, and stores the image (DB image) of the workpiece in the storage unit 17 together with the posture of the arm unit 12 at that time. The control unit 16 may store the recognized workpiece image itself, but may store the recognized local feature amount of the workpiece as a DB image. As described above, the robot 10 obtains information on what kind of work image can be obtained from which direction the work is taken (that is, how the work looks when viewed from which direction). Will be. That is, the storage unit 17 stores captured images obtained by capturing the workpiece from each direction in association with the capturing direction and the rotation angle of the camera 15 at the time of capturing.

  A plurality of workpieces can be taught to the robot 10 as described above. When teaching a plurality of workpieces, the storage unit 17 stores a DB image for each workpiece.

(Work teaching process)
FIG. 3 is a flowchart of the work teaching process in the present embodiment. In order to teach a work on a work, the user first opens his finger toward the camera 15 and shakes his hand. In the robot 10, the camera 15 captures the movement of the user's hand, and the control unit 16 analyzes the captured image to recognize that the user has instructed the start of work teaching for the workpiece.

  When the robot 10 starts the work teaching process for the workpiece, the robot 10 first recognizes the workpiece and its position / posture by the camera 15 (step S31). For this purpose, the robot 10 photographs the workpiece from a plurality of directions with the camera 15 while driving the arm unit 12 and the hand unit 13. The camera 15 captures a still image of the work from each direction. The control unit 16 recognizes which work is photographed by comparing the photographed image with the DB image of the work stored in the storage unit 17 in advance. When the DB image is a local feature amount, the control unit 16 recognizes the work by extracting the local feature amount from the captured image and comparing it with the DB image.

  Further, the control unit 16 recognizes the position and posture of the workpiece based on the captured image and the posture of the arm unit 12 at that time. At this time, the control unit 16 recognizes the position and orientation of the workpiece in the coordinate system (robot coordinate system) of the space where the robot 10 is installed. A technique in which the control unit 16 recognizes (estimates) the position and orientation of a workpiece (hereinafter also referred to as “object”) will be described later.

  When the robot 10 recognizes the workpiece and recognizes its position and posture, the user grasps the workpiece using two fingers and moves the workpiece to a desired position. This work is a work to be taught to the robot 10. For this purpose, the control unit 16 of the robot 10 first recognizes the user's finger and tracks it (step S32). At this time, the control unit 16 recognizes the position of the finger in the robot coordinate system based on the captured image generated by the camera 15. A known method (for example, Leap Motion (registered trademark)) can be used for recognizing the user's finger.

  The control unit 16 assumes that the user's two fingers (hereinafter, the user's (human) finger is referred to as a “finger” and distinguished from the finger unit 14 or the finger members 14A and 14B of the robot 10) grips the workpiece. When recognized, the position of the finger in the work coordinate system is recognized (step S33). That is, the control unit 16 recognizes that the finger has gripped the workpiece from the finger position and the workpiece position, and converts the finger position at that time to the workpiece coordinate system. The control unit 16 stores the position in the workpiece coordinate system at this time in the storage unit 17 as gripping position information.

  Next, the user holding the workpiece with two fingers moves the workpiece to a desired location and releases the workpiece (releases the finger from the workpiece). The control unit 16 recognizes that the finger has moved away from the workpiece based on the image from the camera 15, recognizes the position and posture of the workpiece in the robot coordinate system at that time as the position and posture of the movement destination of the workpiece, Is stored (step S34). With the above operation, the control unit 16 stores information on the position of the workpiece in the workpiece coordinate system and the position and posture of the workpiece in the robot coordinate system. As a result, preparation for instructing the work to be performed on the work and having the robot 10 perform the work on the work is completed.

  Note that the position and orientation of the movement destination of the workpiece may be recognized based on the position and orientation of the workpiece when it is recognized that the movement of the workpiece has stopped for a predetermined time or more based on the captured image. In the above embodiment, the gripping position of the workpiece is determined based on the gripping position when gripping the workpiece. However, the gripping position of the finger after moving the workpiece to the movement destination may be used as the gripping position of the workpiece. In this case, the control unit 16 recognizes the gripping position of the workpiece in the workpiece coordinate system immediately before the user releases the workpiece from the position and posture of the workpiece at the movement destination and the position of the finger. If the workpiece is not released until it is placed on the destination after gripping the workpiece, the gripping position is recognized either when holding the workpiece or just before releasing the workpiece, but the user places the workpiece on the destination. When the work is changed before placing, the gripping position after the change is recognized. When the user carries the workpiece to the vicinity of the destination, the user does not need to pick up the workpiece with two fingers, and may lift it by an arbitrary hand movement. If the workpiece is held so as to be gripped with two fingers when placed at a desired location, it can be taught where to grip the workpiece to place it at the desired location.

  In step S34, the position and posture of the workpiece in the robot coordinate system are recognized as the position and posture of the movement destination of the workpiece. However, in the work execution process described later, a plurality of workpieces are related to each other in position and posture. When moving to a predetermined position and orientation, for example, when performing a task of completing a combination by combining a plurality of workpieces, the position and orientation of the destination are the same as those already moved. Coordinate system (combination product) with reference to the combination (finished product) that has been completed so far, for example, in the case of composing a product by combining multiple workpieces. In the coordinate system, it may be the position and posture of the workpiece to be moved. By doing in this way, for example, even if the position and posture of the combination in the middle of completion are changed, the combination in the middle of completion can be recognized if the position and posture of the combination in the middle of completion after the change can be recognized. It is possible to perform an operation of moving the workpiece to a predetermined position in a predetermined posture.

  Through the work teaching process described above, information on the gripping position of the work (work coordinate system) and information on the position and orientation of the movement destination are acquired, and these pieces of information are associated with each other and stored in the control unit 16 as work teaching information. Is done.

  In addition, as a work to teach the robot, when a predetermined work is taken out from a large number of work pieces contained in the material box and carried to a workbench, and a work is assembled on the workbench, In addition to work on the work, it is necessary to teach the robot which work is to be done in what order (assembly program).

(Work implementation process)
In the work execution process, the user instructs the robot 10 to perform a work to be worked on, and asks the robot 10 to perform the work. For this purpose, the user first instructs the robot 10 on a work to be performed. At this time, the user uses his / her one finger to first sway this finger toward the camera 15 to cause the robot 10 to recognize that the work instruction to the workpiece is to be performed. When the robot 10 recognizes that the start of the work instruction to the work has been instructed, the user indicates the work to be performed by pointing the work desired to be performed with one finger.

  FIG. 4 is a flowchart of the work execution process in the robot 10. The camera 15 captures the user's finger (one finger that extends straight) and sends the captured image to the control unit 16 via the camera control unit 151. Based on this photographed image, the control unit 16 uses the above-described finger recognition technology to indicate that the user is instructing a work to be performed on the workpiece, and the finger position and the finger are pointing in the robot coordinate system. Recognize direction. The control unit 16 controls the operation of the arm unit 12 so that the camera 15 is directed toward the tip of the finger, and causes the camera 15 to photograph the workpiece.

  The control unit 16 identifies a workpiece in the direction indicated by the finger by collating a captured image obtained by capturing the workpiece with the image DB, and determines the robot coordinates of the workpiece and the workpiece from the captured image and the posture of the arm unit 12 at that time. The position and orientation in the system are recognized (step S41). When a plurality of works are taught for one work, the work contents are also instructed after step S41. The work is specified and the work instruction is not limited to the user's pointing in the real space, but may be performed using, for example, a GUI using a virtual space in a smart phone or the like. May be selected by the user, or the user's voice may be recognized. A technique for recognizing the workpiece in the pointing direction will be described later.

  If the position and orientation of the workpiece can be recognized, the control unit 16 controls the arm unit 12, the hand unit 13, and the finger unit 14 to cause the finger unit 14 to grip the workpiece (step S42). At this time, the control unit 16 controls the arm unit 12, the hand unit 13, and the finger unit 14 so as to grip the workpiece at the stored workpiece gripping position.

  When the finger unit 14 grips the workpiece, the control unit 16 controls the arm unit 12 and the hand unit 13 so that the workpiece moves to the movement destination taught in advance and assumes the posture taught in advance. The arm unit 12 and the hand unit 13 are driven by the control of the control unit 16 and move the workpiece to the taught destination so that the posture is taught in advance (step S43). When the workpiece reaches a predetermined position and posture, the control unit 16 drives the finger unit 14 so that the distance between the finger members 14A and 14B is increased, and releases the workpiece (releases the workpiece). (Step S44).

  In the case of performing an operation of assembling a plurality of types of workpieces, the user does not instruct the operation for each workpiece as described above, but the robot 10 automatically and sequentially performs the workpieces according to the assembly program. Recognize and do the necessary work.

  As described above, according to the robot and the robot control method of the present embodiment, the user can teach the robot 10 the work on the work by actually performing the work on the work using the finger. Does not need to be programmed, does not need to input to a computer using any GUI, and does not require the user to directly operate the arm unit 12, the hand unit 13, and the finger unit 14 of the robot 10. The robot 10 can be instructed to work on the workpiece by a simple and intuitive method.

  Furthermore, in the present embodiment, when teaching the gripping work, the gripping position in the work system coordinates is stored, so when actually instructing the work, the position and posture of the work to be worked are: If the position and posture of the workpiece can be recognized even if the position and posture are different from those taught, the same workpiece position as that taught can be gripped by the finger members 14A and 14B.

  Also, since there are two finger members 14A and 14B and the user grips the workpiece using two fingers when teaching work, it is compared with the case of using three or more finger members and fingers. Thus, the processing load for avoiding interference and the like can be reduced.

  In the above embodiment, the example of performing the work of moving the desired workpiece to the desired position and posture has been described. However, the work content of the robot and the robot control method of the present invention is not limited to the above example, In the work teaching step, the user may teach the robot only the workpiece gripping position, and in the work execution step, the user may determine the movement destination by pointing the movement destination of the workpiece with a finger. In this case, if necessary, the workpiece may be moved so as to be in a posture according to the user's instruction in some instruction process to release the grip.

  The work content may be a work that recognizes and grips a desired workpiece, moves to a desired position, and keeps it held in a desired posture (does not release), and recognizes the desired workpiece. The workpiece may be gripped, and the workpiece may be repeatedly given positive and negative speeds in a predetermined space, or the positive and negative accelerations may be repeatedly applied (shake). .

  Further, in the above-described embodiment, the workpiece is gripped by the finger portion 14 including the finger members 14A and 14B having two variable distances at the tip of the hand portion 13. However, the present invention is not limited to this, and a platform is provided at the tip of the hand portion 13. The work content provided may be a work of placing a work on the table, or a hook may be provided at the tip of the hand unit 13, and the work content may be a work of hooking the work on the hook. In any of these cases, the work content may involve a work of moving a work placed on a table or a work hung on a hook.

  Furthermore, in the above embodiment, the robot 10 lifts and moves the workpiece. However, the robot of the present invention is not limited to this. For example, the workpiece is placed by pressing a predetermined portion of the workpiece in a predetermined direction. It is also possible to move the work while sliding it on the platform.

  In the above embodiment, the user teaches the work on the work by actually picking up the work with a finger, but when this is reproduced by the robot 10, the finger member in the finger portion 14 of the robot 10 is taught. The torque in the direction in which 14A and 14B are brought close to each other is insufficient, and / or the friction coefficient between the finger members 14A and 14B and the workpiece is small, and the workpiece having a large gravity may not be lifted. In this case, the control unit 16 detects an error when instructing.

  Specifically, when the finger members 14 </ b> A and 14 </ b> B are smaller than the distance between the gripping positions of the workpiece, the control unit 16 receives information from the finger sensor 133 and the finger unit 14 moves the workpiece. It can be determined that it is not gripped. Further, immediately after gripping and lifting the workpiece, the control unit 16 takes an image of the vicinity of the finger portion 14 with the camera 15 and drives the arm unit 12 and the hand unit 13 and moves the workpiece after the movement after gripping is started. Can be determined as not being able to grip the workpiece.

  In the above embodiment, the robot 10 stores only the gripping position of the workpiece and the position and posture of the workpiece movement destination as teaching information. However, when teaching the work content for the workpiece, the robot 10 When the position and the stretching direction are further memorized and an instruction is received, the arm unit 12 sets the work held at the tip by the approach that matches the position and the stretching direction of the user's arm when teaching to the destination. You may move it. Thereby, the workpiece can be moved to a desired position in a desired posture while avoiding interference between the other parts around the workpiece and the arm portion 12.

  In the above embodiment, when teaching the grip position of the workpiece to the robot 10, the grip position when the user grips the workpiece with two fingers is detected and used as teaching information. Not limited to this. After the user picks up the workpiece with two fingers, the camera 15 captures the position and posture of the workpiece and the finger holding the workpiece, and based on the captured image, the gripping position of the workpiece held by the user You may ask for.

  The robot of the present invention can be applied to various embodiments. The robot of the present invention may be, for example, a robot for moving a desired article to the user's hand. In this case, the control unit 16 determines that the user has caught the article when detected by any of the torque sensors of the arm sensor 122, the hand sensor 132, and the finger sensor 142, and holds the grip by the finger unit 14. May be canceled. The robot of the present invention may be a robot that completes a product by assembling desired parts, or may be a robot that holds a tablet terminal and holds it toward the user.

(Position and orientation estimation device)
Next, first and second embodiments will be described for a technique in which the control unit 16 recognizes (estimates) the position and orientation of a workpiece (hereinafter also referred to as “object”). FIG. 5A shows an example of a workpiece whose DB image is stored in the storage unit 17 and its coordinate system, and FIG. 5B is a diagram showing an example of workpiece information stored in the storage unit 17. An object having a triangular prism shape as shown in FIG. 5A will be described as an example of a workpiece. For this object, a coordinate system unique to the object is set. The posture of the object can be defined depending on which direction and how much the object is tilted with respect to a reference coordinate system (for example, a coordinate system defined by the robot apparatus).

  As seen in FIG. 5A, when an object is photographed in the (1, 0, 0) direction, a rectangular image is obtained, and when an object is photographed in the (-1, 0, 0) direction, a rectangular image is obtained. Become. When shooting in the (0, 1, 0) direction, a right-angled triangle image is formed with a right vertex at the lower right. When shooting in the (0, -1, 0) direction, a right vertex at the lower left. When the image is taken in the (0, 0, 1) direction and (0, 0, -1) direction, the image becomes a rectangular image.

  As shown in FIG. 5B, the storage unit 17 stores a captured image of an object and data of a shooting direction in which the captured image is captured in association with each other. In FIG. 5B, for simplicity, images from six shooting directions are shown, but in reality, a large number of image data shot from all directions is stored. In addition, the rotation angle of the camera 15 at the time of shooting is shown only in the case of 0 degrees, but actually, a captured image when the rotation angle of the camera 15 is changed is stored for each shooting direction. Yes.

  The control unit 16 has a function as a position / orientation estimation device 160 that collates an object image captured by the camera 15 with a DB image stored in the storage unit 17 to obtain the position and orientation of the workpiece. . FIG. 6 is a block diagram illustrating a configuration of the control unit 16 that functions as the position / orientation estimation apparatus 160. As shown in FIG. 6, the position / orientation estimation apparatus 160 has functions of a captured image acquisition unit 161, a matching unit 162, a predicted image extraction unit 163, an imaging direction determination unit 164, and a position / orientation calculation unit 165. ing.

  The shot image acquisition unit 161 has a function of issuing a shooting instruction to the camera 15 and receiving and acquiring an image shot by the camera 15. Note that the shooting instruction issued from the captured image acquisition unit 161 to the camera 15 includes an instruction for a shooting direction and a rotation angle of the camera. The matching unit 162 has a function of obtaining the similarity between the image captured by the camera 15 and the image stored in the storage unit 17 and extracting an image having a similarity greater than or equal to a predetermined threshold. As a method for obtaining the similarity between images, a known method such as template matching using a luminance gradient direction can be employed.

  In addition, the matching unit 162 is based on the distance from the camera 15 to the object at the time of shooting so that the shooting conditions of both images when matching the shot image and the image stored in the storage unit 17 are the same. It is desirable to normalize the size of the captured image. For example, when the image stored in the storage unit 17 is photographed at a distance of 10 cm from the camera 15 to the object, and when the posture is estimated and the distance from the camera 15 to the object is photographed at 20 cm, the size of the object in the photographed image is set. Enlarging to 2x to ensure proper matching. As a method for obtaining the distance from the camera 15 to the object, a known distance measuring technique using light waves or radio waves can be employed. Further, for example, when the object is placed on a work table whose positional relationship with the camera 15 is known, the distance to the center position of the work table can be set as the distance to the object. At this time, it is possible to determine where the object is on the workbench and correct the distance according to the position of the object on the workbench, thereby obtaining a more accurate distance.

  The predicted image extraction unit 163 has a function of extracting, from the storage unit 17, a predicted image that will be obtained when an object is shot from the next shooting direction (second shooting direction). Specifically, the matching unit 162 obtains, as candidate images, a plurality of images whose similarity to an image taken from the first shooting direction is a predetermined threshold value or more. The predicted image extraction unit 163 obtains a shooting direction obtained by adding a relative movement amount from the first shooting direction to the second shooting direction to the shooting direction associated with each of the candidate images. The associated image is read from the storage unit 17.

  Here, the function of the predicted image extraction unit 163 will be described with a specific example. Since the image of the object is stored in the storage unit 17 in association with the shooting direction, the moving direction from the current shooting direction (first shooting direction) to the next shooting direction (second shooting direction) and If the moving angle is determined, the shooting direction of the predicted image with respect to the candidate image can be specified.

  FIGS. 7A and 7B are diagrams illustrating an example of the shooting direction of an object. It is assumed that a vertically long rectangular image is captured when an object is captured from the first capturing direction. In this case, when matching with the image data stored in the storage unit 17 shown in FIG. 5B is performed by the matching unit 162, an image with a high degree of similarity is displayed in the (1, 0, 0) direction or (−1. , 0, 0) directions are extracted, and these are candidate images. That is, a vertically long rectangular image is expected to be an image taken from either direction shown in FIGS. 7 (a) and 7 (b).

  Next, assuming that the second shooting direction is from the right side in the first shooting direction (that is, the shooting direction is rotated 90 degrees horizontally), the first shooting direction is as shown in FIG. (1, 0, 0) direction, the second shooting direction is the (1, 0, 0) direction, and according to the data stored in the storage unit 17, the lower right is displayed as the predicted image. A right triangle having a right vertex is extracted (see FIG. 8A). When the first shooting direction is the (-1, 0, 0) direction as shown in FIG. 7B, the second shooting direction is the (0, -1, 0) direction, and the storage unit According to the data stored in 17, a right triangle having a lower left corner as a predicted image is extracted as a predicted image (see FIG. 8B). In this manner, the predicted image extraction unit 163 extracts a predicted image when the shooting direction of the candidate image is correct.

  The shooting direction determination unit 164 determines the shooting direction in which the similarity between predicted images for each of the plurality of candidate images is low as the second shooting direction. In the example shown above, since the predicted images shown in FIGS. 8A and 8B have the opposite directions of the right triangle and the degree of similarity is low, such a shooting direction is set as the second shooting direction. To do.

  If there are candidate images as shown in FIGS. 7A and 7B, for example, if the second shooting direction is a direction from top to bottom, the predicted image for any candidate image is rectangular. It becomes a shape image and an image with high similarity. The shooting direction determination unit 164 does not determine the shooting direction in which the similarity of the predicted image is increased as described above as the second shooting direction. This is because if the similarity between the predicted images is high, the matching result between the image captured from the second imaging direction and the predicted image has a similar value, and the candidates for the imaging direction cannot be narrowed down.

  Returning to FIG. 6, the configuration of the position / orientation estimation apparatus 160 will be described. The position / orientation calculation unit 165 has a function of obtaining the object photographing direction from the photographed image and calculating the object posture (the inclination of the object coordinate system) from the obtained photographing direction and the direction of the camera 15.

  FIG. 9 is a flowchart illustrating the operation of the position / orientation estimation apparatus 160 according to the first embodiment. First, the position / orientation estimation apparatus 160 captures an object to be estimated with the camera 15 (step S91). Specifically, the captured image acquisition unit 161 of the control unit 16 instructs the camera 15 to acquire a captured image. Note that the shooting direction at this time is the first shooting direction.

  Next, the matching unit 162 of the position / orientation estimation device 160 performs matching between the captured image and the image stored in the storage unit 17, an image whose similarity is equal to or greater than a predetermined threshold, the shooting direction, and the camera 15. The rotation angle is extracted from the storage unit 17 (step S92). Here, matching with all images stored in the storage unit 17 is performed, and all images having a similarity equal to or higher than a predetermined threshold are extracted. Among the many images extracted here, an image associated with the shooting direction corresponding to the true shooting direction is included, but at this point, usually one image (and its shooting direction) is included. I cannot narrow down. The image extracted here is called a candidate image.

  Subsequently, the shooting direction determination unit 164 of the position / orientation estimation apparatus 160 determines the next shooting direction (second shooting direction) (step S93). At this time, the shooting direction determination unit 164 determines the shooting direction in cooperation with the predicted image extraction unit 163. That is, the predicted image extraction unit 163 obtains a predicted image for each candidate image based on the moving direction and the moving angle from the first shooting direction to the second shooting direction, and the shooting direction determination unit 164 A direction in which the similarity between images is equal to or less than a predetermined threshold is determined as the second imaging direction.

  The position / orientation estimation apparatus 160 images the object to be estimated from the second imaging direction (step S94). Specifically, the captured image acquisition unit 161 of the control unit 16 instructs the camera 15 about the moving direction and the moving angle of the camera 15 and also instructs the shooting to acquire a captured image.

  Next, the position / orientation estimation apparatus 160 performs matching between the captured image and the predicted image (step S95). The predicted image corresponds to each of the candidate images obtained in step S2. Accordingly, in this step, matching with all images stored in the storage unit 17 is not performed. As a result of obtaining the similarity between the captured image and the plurality of predicted images, whether or not there is a predicted image whose similarity is higher than the similarity of the other predicted images by a predetermined threshold or more, that is, the similarity is prominent than others. It is determined whether or not there is a predicted image with a high (step S96). When there is a predicted image whose similarity is higher than the similarity of another predicted image by a predetermined threshold or more (YES in step S96), the position / orientation calculation unit 165 of the position / orientation estimation apparatus 160 associates the predicted image with the predicted image. Based on the obtained shooting direction and the orientation of the camera 15, the posture of the object is calculated, and data of the posture of the object obtained by calculation is output (step S98).

  As a result of matching between the image captured from the second image capturing direction and the predicted image, if there is no predicted image whose similarity is higher than the similarity of other predicted images by a predetermined threshold or more (NO in step S96), similar A predicted image having a degree equal to or greater than a predetermined threshold is set as a candidate image (step S97), another shooting direction (another second shooting direction) is determined, an object is shot, and matching with the predicted image is performed to obtain a candidate. The image is narrowed down (steps S93 to S96). Then, it is determined whether or not there is a predicted image whose similarity is higher than the similarity of other predicted images by a predetermined threshold or more (step S96), and photographing is performed until a predicted image having a higher similarity than the other predicted images is found. Repeat the process by changing the direction. The configuration and operation of the posture estimation apparatus 1 according to the first embodiment have been described above.

  The position / orientation estimation apparatus 160 according to the first embodiment extracts a plurality of candidate images by matching an image photographed from the first photographing direction with an image stored in the storage unit 17, and then extracts each candidate image. Since a predicted image obtained by shooting from the second shooting direction obtained by changing the shooting direction by a predetermined moving angle in the predetermined moving direction is obtained and matching between the predicted image and the shot image is performed, the candidate image is efficiently It is possible to narrow down and determine the shooting direction.

  Further, in the present embodiment, when determining the second shooting direction, since the similarity between the predicted images is determined to be low, a significant difference occurs in the matching result between the predicted image and the shot image. The shooting direction can be narrowed down with a small number of shots.

[Second Embodiment]
FIG. 10 is a block diagram illustrating a configuration of the position / orientation estimation apparatus 260 according to the second embodiment. The basic configuration of the position / orientation estimation apparatus 260 of the second embodiment is the same as that of the first embodiment, but the control unit 16 includes a control candidate image narrowing unit 266 that narrows down candidate images. Is different.

  When a plurality of objects appear in the captured image, the control candidate image narrowing-down unit 266 determines whether the candidate images are a combination of objects that can exist at the same time based on the positional relationship between them. Has a function to narrow down. To give a simple example, for example, when two objects are shown in a photographed image, if each image is viewed separately, a cube whose length of each side is 100 mm is listed as a candidate. Suppose. However, if the distance between the images of the two objects is less than 100 mm, judging from the image being shown, both cannot be a cube with a side length of 100 mm. Or it turns out that both candidates are wrong. The control candidate image narrowing-down unit 266 has a function of narrowing down candidate images using such constraint conditions.

  FIG. 11A is a diagram illustrating an example of an image of an object shown in an image captured by a camera. In the example shown in FIG. 11A, an object related to image A and an object related to image A are shown. FIG. 11B is a diagram illustrating the possibility of combining the candidate image of the object related to image a and the candidate image of the object related to image a. In the table shown in FIG. 11B, in the vertical direction, an object related to a candidate image having a high degree of similarity to the image A and its shooting direction are described. Here, for the sake of simplicity, the shooting direction is described as “downward”, “rightward” or the like, but actually, as described above, the shooting direction is defined by a vector or the like.

In the example shown in FIG. 11B, (i) “object A” is “downward” as a candidate for image a.
(Ii) An image of “object A” viewed “upward”, (iii) An image of “object B” viewed “rightward”, and (iv) “object F” “lower right” There are five possibilities: an image viewed in the “direction” and (v) an image viewed in the “left direction” of the “object H”. As candidates for image a, (i) an image of “object A” viewed “right”, (ii) an image of “object C” viewed “right”, and (iii) “object D” “ There are four possibilities: an image viewed in the “upper left direction”, and (iv) an image viewed in the “upward direction” of “object G”. As shown in FIG. 11B, when image a and image a are individually analyzed, there are five candidates for image a and four candidates for image a.

  The candidate image narrowing-down unit 266 determines whether the candidate images obtained from the images a and a can exist simultaneously. For example, in the case where the image A is obtained by photographing the “object A” from the “downward direction” and the image a is obtained by photographing the “object A” from the “rightward direction”, it is illustrated in FIG. Whether or not both of them can exist at the same time in view of the positional relationship between images a and a, that is, as described above, the object A on image a and the object A on image a interfere with each other. In such a case, it is determined that they cannot exist at the same time. In the example shown in FIG. 11B, “x” is described as such a combination does not exist. In addition, “◯” is described in the possible combinations. The candidate image narrowing-down unit 266 obtains the possibility of existence of each combination of candidate images as shown in FIG. Note that the possibility of the combination being present may be determined when a candidate image is obtained by first photographing. When matching the predicted image by changing the shooting direction and shooting from the second shooting direction, matching is performed within the range of the candidate image obtained first, so no new candidate image appears. It is.

  In the example shown in FIG. 11B, “object B”, which is one of the candidates for image a, has no possibility of being combined with any candidate image of image a. In such a case, if the image a is “object B”, the object corresponding to the image a does not exist. Therefore, the candidate image narrowing unit 266 determines that the image a is “object B”. Is not determined, and is excluded from the candidate images. Similarly, “object G” which is one of the candidates related to the image a is also excluded from the candidate images.

  Further, for example, as in the first embodiment, the object related to the image A is acquired by matching the image taken from the second direction, and the object related to the image A is “object F”. If it is determined that the object related to the image a that can be compatible with the “object F” is only “object C”, the image candidate narrowing-down unit 19 determines that the object related to the image a is “object C”. . In this case, with respect to the image a, it is not necessary to perform matching of images taken from different directions.

  As described above, the posture detection apparatus according to the second embodiment reduces the number of matching operations by narrowing down the candidate images by the control candidate image narrowing unit 266, thereby reducing the calculation processing load necessary for image matching. be able to.

  In the second embodiment, the possibility of the simultaneous existence of two images has been described as an example. However, when images of three or more objects are shown in the captured image, these images are included in the images. The candidate images may be narrowed down based on the possibility of simultaneous existence of such objects.

  The posture estimation apparatus and the posture estimation method according to the present invention have been described in detail with reference to the embodiments. However, the present invention is not limited to the above-described embodiments.

  In the above-described embodiment, the example in which the shooting direction determination unit 164 determines the second shooting direction in consideration of the similarity between predicted images for a plurality of candidate images has been described. It may be determined at random. Further, it may be determined according to the feature of the shape of the object to be estimated. For example, in the case of an object to be rotated by 90 degrees, the object is moved by an angle smaller than 90 degrees.

  In the above-described embodiment, as a result of obtaining the similarity between the captured image from the second imaging direction and the predicted image, the imaging direction is changed until a predicted image having a higher similarity than the other is found. Although the example of performing the narrowing down has been described, the shooting direction may be determined based on the predicted image having the highest degree of similarity with the image from the second shooting direction without performing the determination process in step S96. Also in this case, the accuracy of the estimation of the posture of the object can be improved based on the captured images from the first shooting direction and the second shooting direction.

  In the above-described embodiment, the example in which the storage unit 17 stores the captured image of the object in association with the data of the imaging direction in which the image is captured has been described. Even if the object is not actually photographed, data obtained by rotating the object may be calculated and stored.

  In the above-described embodiment, the storage unit 17 may store image data of a plurality of objects. With this configuration, the position / orientation estimation apparatus 160 can estimate the orientations of a plurality of objects.

(Indicator object display device)
Next, a technique for recognizing a work in the pointing direction in step S41 of the work execution process will be described in detail. Below, 1st-3rd embodiment is described about the indication target object display apparatus which can recognize where the user himself is pointing by irradiating the laser beam to the object to which the user is pointing. . The pointing object display device 100 may be configured in the robot 10 described above or may be configured as a device different from the robot 10. Hereinafter, an example in which the pointing object display device 100 is configured in the robot 10 will be described.

[First Embodiment]
FIG. 12 is a diagram illustrating a configuration of the pointing object display device 100 according to the embodiment. The instruction target display device 100 includes a camera 15, a control unit 16, a storage unit 17, and a light emitting unit 18. That is, the pointing object display device 100 is configured by adding the light emitting unit 18 to the robot 10.

  The storage unit 17 stores data such as an environment in which the user is present, that is, an object around the user, a position and posture of the object, and the like. The data stored in the storage unit 17 may be acquired and stored in advance if the objects around the user are static (for example, walls, ceilings, kitchens, cupboards, air conditioners, etc.). Good. When the object around the user is movable (remote controller, book, glass, etc.), the camera 15 captures the environment around the user when displaying the target object, and the captured image It is good also as acquiring and memorize | storing the data of an object by analyzing. When the objects around the user are a combination of a static object and a dynamic object, the static object is stored in the storage unit 17 in advance, and the dynamic object is used for displaying the pointing object. It is good also as memorizing.

  The camera 15 captures an object from a plurality of directions in order to detect the pointing direction of the user from the captured image. In order to photograph an object from a plurality of directions, a plurality of cameras 15 may be provided, or the camera 15 may be mounted on a moving body, and the camera 15 may be moved to photograph from a plurality of directions.

  The light emitting unit 18 has a function of irradiating light. The light emitting unit 18 is, for example, a laser diode, and has a configuration capable of changing the irradiation direction. Moreover, the light emission part 18 is mounted in the moving body, and can change the starting point of the light emission.

  The control unit 16 obtains a captured image acquisition unit 166 that acquires an image captured by the camera 15, and obtains the pointing direction of the user from the captured image, and an object in the pointing direction of the user (that is, on an extension line in the pointing direction). A target object detection unit 167 that detects the target object) as a target object, and a light emission control unit 168 that controls the light emitting unit 18 to irradiate the target target with light.

  The instruction target detection unit 167 first analyzes the captured image to obtain the pointing direction of the user. Specifically, it is calculated where and where the user's fingertip is in the spatial coordinates. As a method for obtaining the pointing direction of the user from the photographed image, for example, a method described in Japanese Patent Laid-Open No. 2009-151419, Kaoning Hu, et al. “Hand Pointing Estimation for Human Computer Interaction Based on Two Orthogonal-Views” A known method such as the method described in ICPR '10 Proceeding of the 2010 20th International Conference on Pattern Recognition can be adopted.

  The instruction target detection unit 167 reads data related to the object from the storage unit 17, obtains an object that intersects the user's pointing direction, and detects the object as the instruction target. FIG. 13A is a diagram illustrating an example of an object in which the pointing directions intersect. In this way, when the pointing direction intersects an object, the object becomes an instruction target.

  The pointing object detection unit 167 has a function of detecting the object as the pointing object even when the pointing direction passes near the object. The instruction target detection unit 167 sets an extended region for the object that is larger than the actual size of the object. The size of the extended region can be set to a size obtained by multiplying the actual size of the object by a predetermined magnification (for example, 1.5 times), for example. A region obtained by combining the extension region and the existence region of the object corresponds to the “predetermined region including the object” of the present invention.

  FIG. 13B is a diagram illustrating an example in which the pointing direction intersects the extended region of the object. As shown in FIG. 13B, the pointing object detection unit 167 also detects the object as the pointing object even when the pointing direction intersects the extended region of the object.

  In addition, when there are a plurality of objects in the pointing direction, the pointing object detection unit 167 detects an object whose intersection with the pointing direction is closest to the center as the pointing object. As shown in FIG. 14, it is assumed that there are an object A and an object B in the pointing direction, and an extension line in the pointing direction of the user intersects with the extension area of the object A and intersects with the existence area of the object B. Referring to FIG. 14, when the distance between each intersection position and the object center is viewed, the intersection position with the object B is closer to the object center than the intersection position with the object A. In such a case, the pointing object detection unit 167 detects the object B as the pointing object. Thereby, when there are a plurality of objects, an object suitable for the user's intention can be detected as an instruction target.

  The light emission control unit 168 has a function of controlling the light emitting direction of the light emitting unit 18 so as to irradiate the object detected as the pointing object. The light emission control unit 168 irradiates light at a position where the surface of the object that is the pointing target is easily visible. In the present embodiment, the light emission control unit 168 does not irradiate the intersection position between the pointing direction and the object (or its extended region), but emits light to a position near the center of the object from the intersection position. Irradiate. Thereby, the user can easily determine whether or not light is irradiated when the object is viewed.

  FIG. 15 is a diagram illustrating a relationship between a position where the pointing directions intersect and an irradiation position by light emission. The diagram on the left side of FIG. 15 shows the coordinate system of a predetermined area including the extension area, and the diagram on the right side shows the coordinate system of the object. In FIG. 15, for the sake of simplicity, the description is made with two-dimensional coordinates. The light emission control unit 168 calculates the coordinates of the irradiation position by multiplying the coordinate value of the intersection position in the pointing direction by 2/3 in order to bring the irradiation position to the center of the object (the origin of the coordinate system). The storage unit 17 of the pointing object display device 100 according to the present embodiment stores a table for converting the coordinates of the intersection position into the coordinates of the irradiation position.

  FIG. 16 is a flowchart illustrating the operation of the pointing object display device 100 according to the embodiment. First, the indication object display device 100 images the user with the camera 15 (step S161), and the indication object detection unit 167 detects the pointing direction of the user from the captured image (step S162). Subsequently, the instruction target detection unit 167 reads out data related to an object around the user from the storage unit 17, and uses an object whose predetermined area defined for the object intersects the user's pointing direction as the instruction target. Obtained (step S163). Next, the light emission control unit 168 of the pointing object display device 100 calculates the irradiation position on the object from the crossing position of the pointing direction and the object (step S164), and controls the light emitting unit 18 to face the irradiation position. To emit light (step S165). The instruction target object display device 100 repeatedly performs the above processing.

  Thereby, when the user moves the pointing direction, the pointing object display device 100 detects an object based on the pointing direction of the user, and performs irradiation by obtaining an irradiation position on the object. As the user's pointing direction changes, the irradiation position on the object also changes. Therefore, the user can appropriately indicate a desired object by changing the pointing direction while viewing the irradiation position.

  Further, as can be seen from the content described with reference to FIG. 14, in this embodiment, light is irradiated on the surface of the object by pointing to the area near the object. As the pointing direction is moved, the irradiation position moves toward the object when the pointing direction approaches the object. Since the user does not have to point the pointing direction strictly toward the object, the user can easily indicate a desired object.

[Second Embodiment]
Next, the pointing object display device according to the second embodiment will be described. The basic configuration of the pointing object display device of the second embodiment is the same as that of the pointing object display device 100 of the first embodiment. However, in the second embodiment, the fingertip of the user's fingertip is the same. The difference is that the irradiation position is not shaken by the tremor.

  FIG. 17 is a flowchart illustrating the operation of the pointing object display device according to the second embodiment. First, the pointing object display device captures the user with the camera 15 (step S171), and the pointing object detection unit 167 detects the pointing direction of the user from the captured image (step S172). Data in the pointing direction is stored in the storage unit 17 (step S173).

  Next, the pointing object display device according to the second embodiment reads the data in the pointing direction within the predetermined time immediately before (for example, 0.3 seconds) from the storage unit 17, and the amount of movement in the pointing direction Is greater than a predetermined threshold (for example, 1 degree) (step S174). As a result, when the amount of pointing motion is within a predetermined threshold (NO in step S174), the user is photographed (step S171), and the process of detecting the pointing direction of the user (step S172) is performed. Return.

  When the amount of pointing motion is greater than the predetermined threshold (YES in step S174), the pointing object detection unit 167 reads data related to the object around the user from the storage unit 17, and determines the predetermined value determined for the object. An object whose area intersects with the user's pointing direction is obtained as an instruction target (step S175). Next, the light emission control unit 168 of the pointing object display device calculates the irradiation position on the object from the crossing position of the pointing direction and the object (step S176), and controls the light emitting unit 18 toward the irradiation position. Light is emitted (step S177). The instruction object display device repeats the above processing.

  The trembling of the user's fingertip is a phenomenon that occurs unconsciously, and this phenomenon is not intended to indicate an object by the user. In the second embodiment, it is possible to realize an instruction of an object that matches the user's intention by not reflecting in the irradiation position a fine movement smaller than a predetermined threshold within a predetermined time.

[Third Embodiment]
Next, an indication object display device according to a third embodiment will be described. The basic configuration of the pointing object display device according to the third embodiment is the same as that of the pointing object display device according to the first embodiment. However, in the third embodiment, the user can quickly move the fingertip. When moving and changing the pointing direction, the difference is that the crossing position between the pointing direction and the extended area of the object is not detected in the movement process.

  FIG. 18 is a flowchart illustrating the operation of the pointing object display device according to the third embodiment. First, the pointing object display device captures the user with the camera 15 (step S181), and the pointing object detection unit 167 detects the pointing direction of the user from the captured image (step S182). Subsequently, the pointing object detection unit 167 calculates the moving speed in the pointing direction (step S183). As in the case of the second embodiment, the moving speed is calculated by storing the obtained pointing direction data in the storage unit 17 and using the time series data stored in the storage unit 17. By calculating the speed.

  The instruction target display device determines whether or not the moving speed is equal to or less than a predetermined threshold (step S184). As a result, when the moving speed is equal to or lower than the predetermined threshold (YES in step S184), the same processing as in the first embodiment is performed. That is, the indication target object display device reads out data related to an object around the user from the storage unit 17 and obtains an object whose predetermined area defined for the object intersects the user's pointing direction as the indication object ( Step S185). Next, the light emission control unit 168 of the pointing object display device calculates the irradiation position on the object from the crossing position of the pointing direction and the object (step S186), and controls the light emitting unit 18 toward the irradiation position. Light is emitted (step S187).

  On the other hand, when the moving speed of the user's fingertip is greater than the predetermined threshold (NO in step S184), the pointing object display device obtains a position where the pointing direction and the object intersect (step S188). In this step, an extended region is not set for the object, and a position where the object and the pointing direction intersect is obtained. And the light emission part 18 light-emits toward the calculated | required crossing position (step S189).

  According to the present embodiment, when the pointing direction is moving at a speed equal to or lower than a predetermined threshold, the irradiation position during movement sticks to the object as described in the first embodiment. Since it moves like this, it is easy for the user to point to an object on the way. When the pointing direction is moving at a speed greater than a predetermined threshold, the irradiation position indicates the pointing direction as it is and moves smoothly. When moving the pointing direction quickly, the pointing direction is moving toward the desired object, and it is considered that there is no intention to point to an object in between. It is suitable for the user's intention not to do this.

  In the present embodiment, an example in which the method of obtaining the irradiation position is changed depending on whether or not the moving speed in the pointing direction is equal to or less than a predetermined threshold has been described, but the determination may be made based on the moving acceleration in the pointing direction. Alternatively, both the moving speed and the moving acceleration may be used.

  Further, in this embodiment, an example in which the pointing direction is irradiated as it is when the moving speed in the pointing direction is larger than the predetermined threshold has been described. However, the moving speed in the pointing direction is larger than the predetermined threshold. Alternatively, the irradiation in the pointing direction may be stopped. As mentioned above, it is thought that there is no intention to indicate the object when moving quickly in the pointing direction, so there is no problem even if it does not irradiate, and the process of obtaining the intersection position with the object is unnecessary. , The process becomes simple.

  In the first to third embodiments described above, the example in which the extension area is set for the object so that the object can be detected even when the vicinity of the object is pointed is described. Other objects may be provided. For example, an extension area may be set for an object having a size equal to or smaller than a predetermined threshold, and no extension area may be set for an object larger than the predetermined threshold. Since it is difficult to point a small object accurately, it is possible to easily indicate a small object by setting an extended area. Conversely, for large objects, it is easy to point without an extended area. Conversely, when an extended area is set for a large object, a large object is in the way even though you want to point to another object. Is also possible.

  Further, an extension area may be set for a movable object, and an extension area may not be set for a static object. For example, when an instruction is given to a robot, it often involves moving an object, such as bringing an object or putting it away. It is considered that there is little indication of static things such as the ceiling. Therefore, it is practical to set an extension area for a dynamic object to facilitate the instruction of the object.

  Further, when setting the extension area, it is not necessary to be an area that is uniformly enlarged over the entire circumference of the object. For example, an extension area that is greatly enlarged in the horizontal direction from the vertical direction may be set. Since humans often move the pointing direction horizontally, it is possible to appropriately capture an object in the pointing direction by enlarging the extended area in the horizontal direction. Further, the shape of the extended region may be a shape suitable for the user by learning the habit of pointing at the user. For example, for a user who tends to point to the right side of the actual position of the object, it is possible to easily indicate the object by setting an extended area on the right side.

  In the first to third embodiments described above, an example has been described in which when a plurality of objects are detected, an object that is instructed to be the closest to the center among the plurality of objects is used as an instruction target ( 14), the object closest to the user among the plurality of objects may be used as the instruction target.

  Further, as another aspect, the elbow angle of the user who is pointing from the captured image is obtained, and when the elbow is stretched, the distant object is used as the pointing object, and the elbow is bent In some cases, a nearby object may be used as the instruction target. Thereby, a user's intention can be estimated and an indication subject can be irradiated appropriately.

  Further, when the movement in the pointing direction is detected, the intent of the user may be estimated based on the radius of curvature of the movement locus of the fingertip, and the pointing object may be determined. For example, when the radius of curvature of the movement trajectory of the fingertip is large, it is considered that the pointing direction is changed by turning the shoulder instead of turning the elbow or wrist. It can be presumed that this is the intention. In this case, among the objects in the pointing direction after movement, an object far away is detected as the pointing object.

  In the first to third embodiments described above, when obtaining the irradiation position from the intersection position, the coordinate value of the irradiation position is calculated by multiplying the coordinate value of the intersection position in the pointing direction by 2/3. As described above, the conversion method (conversion table) from the intersection position to the irradiation position does not necessarily have to be the same magnification, and various variations are conceivable. For example, a predetermined area including an extended area is divided into two areas depending on whether or not the object is in the vicinity of the center of the object. When entering the area, coordinate conversion may be performed so that the irradiation position enters the area near the center. Further, the center of the object may be set as the irradiation position regardless of the intersection position.

  INDUSTRIAL APPLICABILITY The present invention has an effect that a robot can be instructed to work on a work by a simple and intuitive method, and is useful as a robot that performs work on a work, a robot control method, and the like.

DESCRIPTION OF SYMBOLS 10 Arm type robot 11 Base 12 Arm part 12A 1st arm 12B 2nd arm 13 Hand part 14 Finger part 14A, 14B Finger member 15 Camera 16 Control part 17 Storage part 18 Light emitting part 21 1st axis 22 2nd axis 23 3rd Axis 24 Fourth Axis 25 Fifth Axis 26 Sixth Axis 121 Actuator 122 Arm Sensor 131 Actuator 132 Hand Sensor 141 Actuator 142 Finger Sensor 151 Camera Control Unit 160 Position / Orientation Estimation Device 161 Captured Image Acquisition Unit 162 Matching Unit 163 Predictive Image Extraction Unit 164 Shooting direction determination unit 165 Position and orientation calculation unit 166 Shooting image acquisition unit 167 Pointed object detection unit 168 Light emission control unit 266 Control candidate image narrowing unit

Claims (11)

A robot that works on workpieces,
A camera that generates a shot image by shooting;
A working unit that performs work on the workpiece;
A control unit for recognizing a work content for a work by a user based on a photographed image including an image of the work in the work teaching step, and controlling the work unit so as to perform the work content in the work execution step;
Equipped with a,
The working unit is capable of gripping a workpiece,
The control unit recognizes a work gripping position by a user's finger in the work teaching process, and controls the work part so that the work part grips the work at the gripping position in the work execution process. Robot characterized by.   In the work execution step, the control unit recognizes a work based on a photographed image including an image of the work, and controls the work unit so as to perform the work content corresponding to the recognized work. The robot according to claim 1.   The control unit further recognizes the workpiece in the workpiece teaching step, and compares the workpiece recognized in the workpiece teaching step with the workpiece image included in the photographed image in the operation performing step in the operation performing step. The robot according to claim 2, wherein the robot recognizes a workpiece. The working unit is capable of moving a workpiece,
The robot according to claim 1, wherein the work content is to move a workpiece.   The control unit controls the working unit to perform a work of moving a plurality of workpieces so that a relationship between the positions and postures of the workpieces is a predetermined position and posture in the work performing step, and the work teaching step. The position and orientation of the work destination after the user has moved the work, and the position and orientation of the move destination are positions and orientations with reference to another work that has already been moved. The robot according to claim 4. The working unit can lift and move the workpiece,
The robot according to claim 1, wherein the work content is lifting and moving a workpiece. Before SL work, the robot according to claim 1, characterized in that by moving grips the workpiece. The working portion includes two finger members, and is a finger portion that grips a workpiece by changing a distance between the two finger members.
The control unit recognizes a gripping position of a workpiece by a user's two fingers in the work teaching step, and in the work execution step, the control unit controls the working unit to grip the workpiece at the gripping position. The robot according to claim 1, wherein the robot is controlled.   The control unit recognizes the gripping position of the workpiece by the user's finger in the work teaching step, and the position and orientation of the workpiece movement destination when the user releases the gripping of the workpiece. A part grips the work at the gripping position, moves the work to the position of the movement destination, and controls the working unit to release the gripping of the work at the position and posture of the movement destination. Item 8. The robot according to Item 7. A robot control method for controlling a robot having a working unit for working on a workpiece,
A shooting process for generating a shot image by shooting;
A work teaching step for recognizing the work content for the work by the user based on a photographed image including the image of the work;
A work execution step of controlling the working unit to perform the work content;
Only including,
The working unit is capable of gripping a workpiece,
The work teaching step recognizes a gripping position of a work by a user's finger,
The robot control method characterized in that the work execution step controls the work part so that the work part grips a workpiece at the gripping position . A robot computer that has a camera that generates a photographed image by photographing and a working unit that performs work on the work,
In the work teaching step, based on a photographed image including the image of the work, the user recognizes the work content for the work, and in the work execution step, functions as a control unit that controls the work unit to perform the work content. ,
The working unit is capable of gripping a workpiece,
Wherein, in the work teaching process, to recognize the gripping position of the workpiece by the user's finger, in the work carried out step, that controls the working unit so that the working unit grips the workpiece by the gripping position A robot control program characterized by that.