JP2015221485A - Robot, robot system, control unit and control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot capable of acquiring information of teaching contents without causing a considerable error.SOLUTION: A robot includes an arm and a control unit activating the arm. The control unit receives a plurality of first images obtained by imaging a work object in time series, extracts a second image from the plurality of first images when a motion state of the work object is changed and activates the arm by a visual servo by using the second image extracted.

Description

  The present invention relates to a robot, a robot system, a control device, and a control method.

  A technique for operating a robot by visual servoing based on a captured image captured by an imaging unit has been researched and developed. In order to operate the robot, it is necessary to instruct the robot in advance of the operation.

  As a teaching method, for example, the method described in Patent Document 1 can be considered. In Patent Document 1, a teacher directly operates a work through a three-dimensional pointing device and sequentially inputs teaching data obtained by performing the work.

JP 11-85240 A

  However, in this teaching method, since the position and posture of the workpiece are calculated using the three-dimensional pointing device, it is necessary to accurately grasp the position and posture of the three-dimensional pointing device and the position and posture of the workpiece. For this reason, if the position and orientation of the robot cannot be accurately grasped, a large error may occur in the teaching content of the operation to the robot.

  Therefore, the present invention has been made in view of the above-described problems of the prior art, and provides a robot, a robot system, a control apparatus, and a control method that can acquire information on teaching contents without a large error.

One aspect of the present invention includes an arm and a control device that operates the arm, and the control device receives a plurality of first images obtained by imaging a movement of a work object in time series, and a plurality of the first images. A robot that extracts a second image when the motion state of the work object changes from one image, and operates the arm by visual servoing using the extracted second image.
With this configuration, the robot receives a plurality of first images obtained by imaging the movement of the work object in time series, and extracts a second image when the movement state of the work object changes from the plurality of first images. The arm is operated by visual servoing using the extracted second image. Thereby, the robot can acquire the information of the teaching content without a large error. As a result, the robot can perform the taught motion by the visual servo.

According to another aspect of the present invention, in the robot, a configuration in which the movement state of the work object is a movement state of the work object may be used.
With this configuration, the robot extracts the second image when the movement state of the work object changes, and uses the extracted second image to operate the arm by visual servo. Thereby, the robot can acquire the teaching content based on the second image when the movement state of the work object changes.

According to another aspect of the present invention, in the robot, the second image when the movement state of the work object is changed is an image when the speed of the work object is changed. May be.
With this configuration, the robot extracts an image when the speed of the work target changes, and uses the extracted image to operate the arm by visual servo. Thereby, the robot can acquire the teaching content based on the second image when the speed of the work object changes.

According to another aspect of the present invention, in the robot, a configuration in which the motion state of the work object is a contact state of the work object may be used.
With this configuration, the robot extracts the second image when the contact state of the work object changes, and uses the extracted second image to operate the arm by visual servo. Thereby, the robot can acquire the teaching content based on the second image when the contact state of the work object changes.

According to another aspect of the present invention, in the robot, the second image may be an image selected from a plurality of the first images input from an input receiving unit.
With this configuration, the robot operates the arm by visual servo using the plurality of first images input from the input receiving unit as the second image. Thereby, the robot can easily acquire the teaching content desired by the user.

According to another aspect of the present invention, an imaging unit, a robot including an arm, and a control device that operates the arm are provided. The control device performs time series movement of a work object by the imaging unit. Receiving a plurality of first images picked up, extracting a second image when the movement state of the work object changes from the plurality of first images, and using the extracted second images, The robot system operates the arm by a servo.
With this configuration, the robot system receives a plurality of first images captured in time series by the imaging unit for the movement of the work object, and the first motion when the motion state of the work object changes from the plurality of first images. Two images are extracted, and the arm is operated by visual servoing using the extracted second image. Thereby, the robot system can acquire the information of the teaching content without a large error.

According to another aspect of the present invention, a plurality of first images obtained by chronologically capturing the movement of the work object are received, and the movement state of the work object changes from the plurality of first images. The control apparatus extracts two images and uses the extracted second image to operate a robot arm by visual servo.
With this configuration, the control device receives a plurality of first images obtained by capturing the movement of the work object in time series, and extracts a second image when the movement state of the work object changes from the plurality of first images. Then, using the extracted second image, the robot arm is operated by visual servo. Thereby, the control apparatus can acquire the information on the teaching content without a large error.

According to another aspect of the present invention, a plurality of first images obtained by chronologically capturing the movement of the work object are received, and the movement state of the work object changes from the plurality of first images. In this control method, two images are extracted and the robot arm is operated by visual servoing using the extracted second image.
With this configuration, the control method receives a plurality of first images obtained by capturing the movement of the work object in time series, and extracts a second image when the motion state of the work object changes from the plurality of first images. Then, using the extracted second image, the robot arm is operated by visual servo. Thereby, the control method can acquire the information of the teaching content without a large error.

  As described above, the robot, the robot system, the control device, and the control method receive a plurality of first images obtained by capturing the movement of the work object in time series, and the motion state of the work object changes from the plurality of first images. The second image is extracted, and the arm is operated by visual servo using the extracted second image. Thereby, the robot, the robot system, the control device, and the control method can acquire the information on the teaching content without a large error.

It is a lineblock diagram showing an example of robot system 1 concerning a 1st embodiment. 2 is a diagram illustrating an example of a hardware configuration of a control device 30. FIG. 3 is a diagram illustrating an example of a functional configuration of a control device 30. FIG. 7 is a flowchart illustrating an example of a flow of processing for extracting a target image by a control unit. It is a figure which illustrates the frame in the time from which a 1st captured image differs. It is a figure which illustrates the flame | frame extracted from the 1st captured image as a target image. 2 is a diagram illustrating a coordinate system used in the robot system 1. FIG. It is a flowchart which shows an example of the flow of the process in which the control part 36 operates the robot 20 by the visual servo based on a target image. 4 is a flowchart illustrating an example of a flow of robot control processing based on a target image performed by a robot control unit of a control unit. It is a block diagram which shows an example of the robot system 2 which concerns on 2nd Embodiment.

<First Embodiment>
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a configuration diagram illustrating an example of a robot system 1 according to the first embodiment. The robot system 1 includes a first imaging unit 11, a second imaging unit 12, a robot 20, and a control device 30.

  In the robot system 1, the first imaging unit 11 captures a plurality of images (a plurality of first images) captured in time series as the work target M <b> 1 is assembled to the assembly target O <b> 1. The control device 30 is instructed to perform an operation related to the work of assembling the work object M2 to the assembling object O2. The work target M1 is a member used by being assembled to the assembly target O1 in a predetermined state, and is, for example, a plug. The work target M2 is a member used by being assembled to the assembly target O2 in a predetermined state, and is, for example, a plug. The work target M1 and the work target M2 are members having the same shape, but the sizes may be the same or different. The work target M1 and the work target M2 are examples of work objects, respectively. In the present embodiment, a description will be given using a moving image composed of a plurality of still images as an example of a plurality of images captured in time series.

  The assembly target O1 is a member used when the work target M1 is assembled in a predetermined state, and is, for example, an outlet. The assembly target O2 is a member used when the work target M2 is assembled in a predetermined state, and is, for example, an outlet. The assembly target O1 and the assembly target O2 are members having the same shape, but the sizes may be the same or different. When the size of the work target M1 and the work target M2 are different, the ratio of the size of the assembly target O1 and the assembly target O2 is the same as the ratio of the size of the work target M1 and the work target M2. Is preferred.

  Hereinafter, as an example of the above teaching, a case where the user U uses the hand to capture the work target M1 to the assembly target O1 will be described with respect to the first imaging unit 11, but the user U uses the hand. Instead of assembling, the first imaging unit 11 may image the state of assembling by some other method.

  The robot system 1 causes the robot 20 to perform the work of assembling the work object M2 to the assembly object O2 based on the operation taught to the control device 30. At this time, the robot system 1 controls the robot 20 by visual servoing based on the captured image captured by the second imaging unit 12. In the following description, it is assumed that the work target M2 is gripped in advance by the gripping unit HND (end effector) of the robot 20, but the work target M2 is not gripped by the gripping unit HND in advance. There may be. In this case, the robot system 1 detects the position and orientation of the work target M2 based on the captured image captured by the second imaging unit 12, and holds the work target M2 based on the detected position and orientation.

  The first imaging unit 11 is a camera including, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, which is an imaging element that converts collected light into an electrical signal. The first imaging unit 11 is a stereo camera configured by two cameras. For example, the first imaging unit 11 may be configured by three or more cameras, and configured to capture a two-dimensional image by one camera. There may be.

  The 1st imaging part 11 is connected with the control apparatus 30 via the cable so that communication is possible. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB (Universal Serial Bus), for example. The first imaging unit 11 and the control device 30 may be configured to be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). The first imaging unit 11 includes a work target M1 gripped by the user U and an assembly target O1 that is the assembly target O1 gripped by the user U and to which the work target M1 is assembled by the user U. It is installed at a position where the range can be imaged. Hereinafter, for convenience of explanation, the moving image captured by the first imaging unit 11 will be referred to as a first captured image.

  The second imaging unit 12 is, for example, a camera that includes a CCD, a CMOS, or the like that is an imaging device that converts collected light into an electrical signal. The second imaging unit 12 is a stereo camera configured by two cameras. For example, the second imaging unit 12 may be configured by three or more cameras, and configured to capture a two-dimensional image by one camera. There may be.

  The second imaging unit 12 is connected to the control device 30 via a cable so as to be communicable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The second imaging unit 12 and the control device 30 may be configured to be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The second imaging unit 12 is located at a position where the range including the robot 20, the work target M2 gripped by the gripping unit HND of the robot 20, and the assembly target O2 to which the work target M2 is assembled by the robot 20 can be captured. is set up. As shown in FIG. 1, the assembly target O2 is set in advance by a jig or the like at a position away from the robot 20 within a range in which the work target M2 can be assembled by the robot 20. Hereinafter, for convenience of explanation, the captured image captured by the second imaging unit 12 will be referred to as a second captured image. Note that the second captured image may be a moving image or a still image.

  The robot 20 is, for example, a single-arm robot including a gripping unit HND, a force sensor 22, a manipulator MNP, and a plurality of actuators (not shown). The manipulator MNP is an example of an arm. Further, the robot system 1 may be configured to include a dual-arm robot instead of the configuration including the single-arm robot.

  The arm of the robot 20 is a 6-axis vertical articulated type, and can operate with 6 degrees of freedom by an operation in which the support base, the manipulator MNP, and the gripper HND are linked by an actuator. The arm of the robot 20 may operate with 5 degrees of freedom (5 axes) or less, or may operate with 7 degrees of freedom (7 axes) or more. Below, the operation | movement of the robot 20 performed with the arm provided with the holding part HND and the manipulator MNP is demonstrated.

  The robot 20 is communicably connected to the control device 30 by a cable, for example. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The robot 20 and the control device 30 may be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). Further, in the robot system 1, the robot 20 is configured to be connected to the control device 30 installed outside the robot 20 as shown in FIG. The structure built in 20 may be sufficient.

The grip part HND of the robot 20 includes a claw part that can grip an object.
The force sensor 22 is provided between the gripping part HND of the robot 20 and the manipulator MNP, and detects a force and a moment that act on the gripping part HND (or the work target M2 gripped by the gripping part HND). The force sensor 22 outputs information indicating the detected force and moment to the control device 30 by communication. Information indicating the force and moment detected by the force sensor 22 is used for compliant motion control of the robot 20 by the control device 30, for example.

  The robot 20 acquires a control signal from the control device 30, and based on the acquired control signal, moves the work target M2 gripped by the gripping unit HND of the robot 20 from the current position by the manipulator MNP, and sets the assembly target O2. Performs operations related to assembly work. The operation related to the work for assembling the work object M2 to the assembly object O2 is, for example, translational movement or rotation. Further, the work of assembling the work object M2 to the assembly object O2 includes at least one operation.

  The control device 30 controls the robot 20 to assemble the work target M2 to the assembly target O2 by the operation taught based on the first captured image. More specifically, the control device 30 extracts a target image (second image) of at least one frame from the first captured image (moving image) captured by the first imaging unit 11. The target image is a template image in visual servoing, for example, a position and posture to be realized when the work target M2 is moved by the robot 20, and is a relative between the work target M2 and the assembly target O2. It is an image showing a correct position and posture (hereinafter referred to as a state of the work target M2 and the assembly target O2). The control device 30 reads target images in the order of extraction, and performs visual servoing based on the read target images.

  Then, based on the read target image, the control device 30 is configured so that the state of the work target M2 and the assembly target O2 is the state represented by the target image and the state of the work target M2 and the assembly target O2. The work target M2 is moved to the robot 20. For example, when three target images are extracted from the first captured image (hereinafter, referred to as first to third target images in time series order), the control device 30 first reads the first target image as a template image. Then, by the visual servo based on the read first target image, the work target M2 is moved to the robot 20 so that the state represented by the target image and the state of the work target M2 and the assembly target O2 is realized.

  Thereafter, the control device 30 reads the second target image as the next template image, and the state represented by the target image by the visual servo based on the read second target image, and the states of the work target M2 and the assembly target O2 The work target M2 is moved to the robot 20 so that is realized. Then, the control device 30 finally reads the third target image as a template image, and is a state represented by the target image by visual servoing based on the read third target image, and is a state of the work target M2 and the assembly target O2. The work target M2 is moved to the robot 20 so that is realized.

  In this way, the control device 30 extracts the target image of at least one frame from the first captured image, and completes the operation of assembling the work target M2 to the assembly target O2 based on the extracted target image.

  Next, the hardware configuration of the control device 30 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the control device 30. The control device 30 includes, for example, a CPU (Central Processing Unit) 31, a storage unit 32, an input reception unit 33, and a communication unit 34, and the first imaging unit 11 and the second imaging unit 12 via the communication unit 34. Communicate with the robot 20. These components are connected to each other via a bus Bus so that they can communicate with each other. The CPU 31 executes various programs stored in the storage unit 32.

  The storage unit 32 includes, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a ROM (Read-Only Memory), a RAM (Random Access Memory), and the like. Various information, images, and programs processed by the control device 30 are stored. Note that the storage unit 32 may be an external storage device connected by a digital input / output port such as a USB instead of the one built in the control device 30.

The input receiving unit 33 is, for example, a keyboard, a mouse, a touch pad, or other input device. The input receiving unit 33 may function as a display unit, and may be configured as a touch panel.
The communication unit 34 includes, for example, a digital input / output port such as a USB, an Ethernet port, and the like.

  Next, the functional configuration of the control device 30 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a functional configuration of the control device 30. The control device 30 includes a storage unit 32, an input reception unit 33, an image acquisition unit 35, and a control unit 36. Some or all of these functional units are realized, for example, by the CPU 31 executing various programs stored in the storage unit 32. Some or all of these functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

  The control device 30 incorporates the compliant motion control into the visual servo loop based on the target image of at least one frame, so that the work object M2 and the assembly object O2 can be assembled without destroying the work object M2 and the assembly object O2. Move relative to O2. Further, the control device 30 includes a timing unit (not shown), acquires the second captured image from the image acquisition unit 35 at the timing timed by the timing unit, and from the force sensor 22 of the robot 20 at the same timing, Information indicating the force and moment detected by the force sensor 22 is acquired.

The image acquisition unit 35 acquires the first captured image captured by the first imaging unit 11. Then, the image acquisition unit 35 causes the storage unit 32 to store the acquired first captured image. The image acquisition unit 35 acquires the second captured image captured by the second imaging unit 12. The image acquisition unit 35 outputs the acquired second captured image to the control unit 36.
The control unit 36 includes a target image extraction unit 41 and a robot control unit 42.

  The target image extraction unit 41 reads the first captured image stored in the storage unit 32 and extracts a frame that satisfies a predetermined condition as a target image based on the read first captured image. The predetermined condition is that the motion state of the work target M1 has changed. Specifically, the movement state of the work target M1 has changed, for example, the speed has changed. Examples of the speed include a translational movement speed and a rotational angular speed.

  That is, the target image extraction unit 41 extracts, for example, a frame at the time when the translational movement speed of the work target M1 becomes zero as a target image. Since the movement state of the work object M1 only needs to be changed, the predetermined condition is that the movement state of the work object M2 has changed, and the contact state between the work object M1 and the assembly object O1. May have changed (for example, change from point contact to surface contact, change from non-contact state to contact state, etc.), the speed of the work object M1 deviates from within a predetermined range, It may be some other condition such that the absolute value of the speed of the target M1 is greater than or equal to a predetermined threshold value or less than a predetermined threshold value. Further, the predetermined condition may be that the absolute value of the speed of the work target M1 is changed from zero to non-zero, or from non-zero to zero. Further, the predetermined condition may be that the manner of movement of the work target M1 changes. The target image extraction unit 41 stores the extracted target images in the storage unit 32 in chronological order.

  The robot control unit 42 acquires the second captured image from the image acquisition unit 35. The robot control unit 42 reads target images from the storage unit 32 in chronological order. The robot control unit 42 acquires information indicating the force and moment that are applied to the gripping unit HND (or the work target M2) from the force sensor 22. The robot control unit 42 controls the robot 20 by visual servo combined with compliant motion control based on the second captured image, the target image, and information indicating the force and moment applied to the gripping unit HND. The robot 20 is controlled by generating a control signal for output and outputting the generated control signal to the robot 20.

  Hereinafter, the process of extracting the target image by the control unit 36 will be described with reference to FIG. FIG. 4 is a flowchart illustrating an example of a process flow for extracting a target image by the control unit 36. First, the control unit 36 controls the first imaging unit 11 so that the first imaging unit 11 captures an image of the work of the user assembling the work target M1 and the assembly target O1 (step S100). Although the first captured image captured by the first imaging unit 11 is a moving image, it may be a semi-moving image instead.

  Next, the image acquisition unit 35 acquires the first captured image from the first imaging unit 11, and stores the acquired first captured image in the storage unit 32 (step S110). Next, the target image extraction unit 41 reads the first captured image stored in the storage unit 32 (step S120).

  Here, the first captured image will be described with reference to FIG. FIG. 5 is a diagram illustrating frames at different times of the first captured image. In FIG. 5, the user's hand holding the work object M1 and the assembly object O1 is omitted.

  FIG. 5A shows an example of a frame in which a state in which the work target M1 is approaching the assembly target O1 by the user is captured. In FIG. 5A, the work target M1 is translated by the user in the direction of the arrow and is moving so as to approach the assembly target O1. FIG. 5B shows an example of a frame in which the states of the work target M1 and the assembly target O1 after a lapse of time from the frame shown in FIG. In FIG. 5B, the work target M1 is in a state immediately before being translated by the user in the direction of the arrow and fitted into the assembly target O1.

  FIG. 5C shows an example of a frame in which the states of the work target M1 and the assembly target O1 after a lapse of time from the frame shown in FIG. In FIG. 5C, the work target M1 is in a state immediately before being fitted into the assembly target O1 by the user and rotated in the direction of the arrow. FIG. 5D shows an example of a frame in which the states of the work target M1 and the assembly target O1 after a lapse of time from the frame shown in FIG. In FIG. 5D, the work target M1 is assembled to the assembly target O1 by rotation, and shows a state where the work of assembling the work target M1 to the assembly target O1 is completed.

  Next, the target image extraction unit 41 extracts a target image of at least one frame from the read first captured image (step S130).

  Here, with reference to FIG. 6, the process in which the target image extraction unit 41 extracts a target image of at least one frame from the first captured image will be described. FIG. 6 is a diagram illustrating a frame extracted as a target image from the first captured image. The target image extraction unit 41 extracts (captures) a frame that satisfies a predetermined condition from the first captured image as a target image. That is, in this example, the target image extraction unit 41 extracts a frame when the speed of the work target M1 becomes zero as a target image.

  More specifically, for example, the target image extraction unit 41 detects the work target M1 from each frame arranged in time series by pattern matching or the like, and the detected work target M1 has a position and orientation in a frame for a predetermined time. It is determined whether or not it does not change (does not change within a predetermined error range), and when it is determined that it has not changed, it is determined that the speed of the work target M1 has become zero, and one of the frames used for the determination One frame (for example, the last frame) is extracted as a target image.

  In the case of the first captured image shown in FIG. 5, the frame in which the movement state of the work target M1 has changed is that the work target M1 is rotated with the frame when the work target M1 is fitted into the assembly target O1. These are the frame when the assembly is completed on the assembly target O1. FIG. 6A shows an example of a frame when the work target M1 is fitted into the assembly target O1. When the work object M1 is fitted into the assembly object O1, the translational movement speed of the work object M1 becomes zero. When the target image extraction unit 41 determines that the translational movement speed of the work target M1 is zero in a frame for a predetermined time, the target image extraction unit 41 extracts the last frame used for the determination as a target image.

  FIG. 6B shows an example of a frame when the assembly is completed on the assembly target O1 by rotating the work target M1. When the assembly is completed on the assembly target O1 by rotating the work target M1, the rotation angular velocity of the work target M1 becomes zero. When the target image extraction unit 41 determines that the rotation angular velocity of the work target M1 is zero in a frame for a predetermined time, the target image extraction unit 41 extracts the last frame of the frame used for the determination as a target image.

  The target image extraction unit 41 is configured to extract a frame when the speed of the work target M1 becomes zero as a target image. Instead, the speed of the work target M1 becomes zero. A configuration may be used in which a frame at every predetermined time is extracted as a target image from the first captured image together with the time frame. In addition to the configuration in which the target image extraction unit 41 extracts the frame when the speed becomes zero as the target image, the target image extraction unit 41 sets the frame when the physical quantity related to some work target M1 different from the speed becomes zero. It may be configured to extract as an image.

  Next, the target image extraction unit 41 stores the extracted target images in the storage unit 32 in chronological order (step S140). That is, in this example, the target image extraction unit 41 causes the storage unit 32 to store the target image illustrated in FIG. 6A and the target image illustrated in FIG.

  Here, a specific example of the robot control performed by the control device 30 based on the target image as shown in FIGS. 6A and 6B will be described. Hereinafter, for convenience of description, the state represented by the target image and the state of the work target M1 and the assembly target O1 is simply referred to as a state represented by the target image. For example, the control device 30 controls the robot 20 by visual servo based on the target image. The control device 30 performs the second imaging in which a range including the robot 20, the work target M2 gripped by the gripping unit HND of the robot 20, and the assembly target O2 to which the work target M2 is assembled by the robot 20 is captured. Based on the image, the relative positional relationship among the work object M2, the assembly object O2, and the force sensor 22 is derived.

  Therefore, the control device 30 controls the robot 20 to realize the state represented by the target image based on the derived relative positional relationship and the extracted target image of at least one frame. More specifically, the control device 30 controls the robot 20 to realize the state represented by the target image by moving the work target M2 relative to the assembly target O2 by the gripper HND and the manipulator MNP. .

  In addition, when the control device 30 moves the work target M2 relative to the assembly target O2, the control device 30 may control the robot 20 so that the robot 20 rotates the work target M2 held by the holding unit HND. is there. In this case, the control device 30 controls the robot 20 to rotate the work target M2 around a predetermined position (hereinafter referred to as a rotation center position) set to the work target M2. Hereinafter, the posture of the work object M2 at the rotation center position is referred to as a rotation center posture.

  In this control, the control device 30 has a relative positional relationship from the second captured image even when the relative position and posture (relative positional relationship) of the work target M2 and the force sensor 22 change. The rotation center position and the rotation center posture after the change are detected. Then, when the control device 30 causes the robot 20 to perform the operation of rotating the work target M2, the control device 30 controls the robot 20 so as to rotate the work target M2 based on the rotation center position with the detected rotation center position as the center. To do. In addition, when the robot 20 translates the work target M2, the control device 30 controls the robot 20 to translate it based on the detected rotation center posture.

The rotation center position is set at an arbitrary position on the work target M2 (stored in the storage unit 32 of the control device 30) by the user (for example, the user U in FIG. 1). Further, the rotation center posture is the posture of the work target M2 at the rotation center position, but may not coincide with the posture of the work target M2 as long as the posture is associated with the posture of the work target M2.
As described above, the control device 30 in the present embodiment is configured to operate the robot 20 by the visual servo based on the target image, but instead, operates the robot 20 by some other robot control based on the target image. The structure to be made may be sufficient.

  Here, a coordinate system used when the control device 30 performs robot control by the visual servo will be described with reference to FIG. FIG. 7 is a diagram illustrating a coordinate system used in the robot system 1. In the following description, it is assumed that the character described after “_” represents the subscript of the character before “_”.

  As shown in FIG. 7, the control device 30 includes seven three-dimensional orthogonal coordinate systems, that is, an imaging unit coordinate system Σ_c, a work coordinate system Σ_w, a tool coordinate system Σ_t, a centroid coordinate system Σ_g, and a work target. Using the coordinate system Σ_m, the external force coordinate system Σ_e, and the assembly target coordinate system Σ_o, a process for controlling the robot 20 to perform a predetermined work is performed. The origin and coordinate axis directions of these seven coordinate systems are set (stored or registered) in the control device 30 by the user.

  The origins of these seven coordinate systems are all the object X (in this case, the second imaging unit 12, the support base of the robot 20, the force sensor 22, the center of gravity of the work object M2, the rotation center of the work object M2, the work object M2). , It is set to move along with the object X so as to represent the position of the assembly object O2). In addition, the directions of the coordinate axes of these seven coordinate systems are set so as to move along with the posture change of the target X so as to represent the posture of the target X. Note that the user can associate the position of the target X with the position of the origin of the coordinate system, and further, the coordinate system set for the target X on the premise that the inclination of the target X can be associated with the direction of the coordinate system. The position of the origin and the direction of the coordinate axes may be set to arbitrary positions and directions.

The imaging unit coordinate system Σ_c is a coordinate system that represents the position of the second imaging unit 12 (for example, a predetermined position on the imaging element is the origin) and the posture.
The work coordinate system Σ_w is a coordinate system that represents the position and orientation of the support base of the robot 20.
The tool coordinate system Σ_t is a coordinate system set to the position of the force sensor 22 (for example, the center of gravity of the force sensor 22 or the position of a marker indicating the position of the force sensor 22 as an origin) and the posture. In the present embodiment, the tool coordinate system Σ_t represents the position and orientation of the force sensor 22 and the position and orientation of the gripper HND in order to simplify the description.

  In general, the sensor coordinate system that represents the position and orientation of the force sensor 22 and the hand (gripping unit) coordinate system that represents the position and orientation of the gripper HND do not match. Using the relative positional relationship between the sensor coordinate system and the relative positional relationship between the sensor coordinate system and the hand coordinate system, the relative positional relationship between the work coordinate system and the hand coordinate system is calculated and calculated. The robot 20 is controlled based on the relative positional relationship between the work coordinate system and the hand coordinate system.

The centroid coordinate system Σ_g is a coordinate system that represents the position and orientation of the centroid of the work target M2.
The work target coordinate system Σ_m is a coordinate system that represents the position and orientation of the work target M2 (for example, a position on the work target M2 farthest from the gripper HND in the initial state).
The external force coordinate system Σ_e is a coordinate system that defines an external force and an external moment acting on an object. In this embodiment, the coordinate system that defines the motion by the compliant motion control is made to coincide with the external force coordinate system Σ_e. That is, the rotational motion by the compliant motion control is represented by the rotation around the origin of the external force coordinate system Σ_e with the posture of the external force coordinate system Σ_e as a reference. However, these do not need to be matched and can be arbitrarily arranged by the user.

  In the following, unless there is a need to distinguish, an external force acting on an object (that is, an external force detected by a force sensor) is simply referred to as a force, and an external moment (that is, a moment detected by a force sensor) is referred to as a moment. Called. As described above, the rotation center position can be set to an arbitrary position by the user, but in the present embodiment, the rotation center position is described as being set at a predetermined position on the work target M2.

The assembly target coordinate system Σ_o is a coordinate system representing the position and orientation of the assembly target O2 (for example, the position on the assembly target O2 closest to the work target M2).
In the following description, since the coordinate system moves along with the target X, the position of the origin of the coordinate system b in the coordinate system a will be described as the position of the target X for which the coordinate system b is set. For example, the position of the origin of the work target coordinate system Σ_m in the work coordinate system Σ_w is referred to as the position of the work target M2 in the work coordinate system Σ_w. Similarly, the orientation of the coordinate system b in the coordinate system a will be described as the orientation of the target X for which the coordinate system b is set. For example, the posture of the work target coordinate system Σ_m in the work coordinate system Σ_w is referred to as the posture of the work target M2 in the work coordinate system Σ_w.

  Here, before describing a more specific description, a notation method of mathematical expressions used for describing the processing performed by the control device 30 will be described. First, in the following description, it is assumed that the character described after “^” represents the superscript of the character before “^”. In addition, a description will be given assuming that the foremost character with “(→)” added is a vector. The frontmost character with “(^)” added will be described as a matrix.

  Under these notations, a vector representing the position of the target X for which the coordinate system b is set in the coordinate system a is represented as a position vector p_b ^ a (→). The position vector p_b ^ a (→) is defined by the x-coordinate x ^ b, y-coordinate y ^ b, and z-coordinate z ^ b of the object X in the coordinate system b as shown in the following equation (1). .

  Similar to the notation of the position vector, a vector representing the posture of the target X for which the coordinate system b is set in the coordinate system a is represented as a posture vector o_b ^ a (→). The posture vector o_b ^ a (→) is expressed by the following expression (2) using Euler angles (α_b ^ a, β_b ^ a, γ_b ^ a) as shown in the following expression (2). It expresses like this.

  Here, the Euler angles are the z-axis, y-axis and x-axis of the coordinate system a in order to match the x-axis, y-axis and z-axis of the coordinate system a with the x-axis, y-axis and z-axis of the coordinate system b, respectively. It is defined as an angle to rotate around the axis, and is expressed as γ_b ^ a, β_b ^ a, and α_b ^ a, respectively.

  A rotation matrix for rotating the position and orientation of the object X represented by the orientation of the coordinate system b to the position and orientation of the object X represented by the orientation of the coordinate system a is represented as a rotation matrix R_b ^ a (^). Represent. There is a relationship of the following formula (3) between the Euler angle and the rotation matrix.

  Here, in the vectors shown in the above equations (1) to (3), the subscripts can be interchanged in the following conversion equations (4) to (6).

  Here, [R_b ^ a (^)] ^ T represents a transposed matrix of R_a ^ b. That is, the rotation matrix subscripts can be interchanged by transposing the rotation matrix, and the vector subscripts can be interchanged by the rotation matrix. In the following, the position vector p_b ^ a (→) indicating the position of the target X for which the coordinate system b is set in the coordinate system a is simply referred to as the position of the target X in the coordinate system a, unless necessary. Called. For example, the position vector p_o ^ c (→) indicating the position of the origin of the assembly target coordinate system Σ_o set to the assembly target O2 represented in the imaging unit coordinate system Σ_c is simply assembled in the imaging unit coordinate system Σ_c. This is called the position of the object O2.

  As in the case of the position, in the following, the orientation vector o_b ^ a (→) indicating the orientation of the target X in which the coordinate system b is set in the coordinate system a is simply used in the coordinate system a, unless necessary. This is called the posture of the target X. For example, the orientation vector o_o ^ c (→) indicating the orientation of the assembly target coordinate system Σ_o set to the assembly target O2 represented in the imaging unit coordinate system Σ_c is simply set as the assembly target O2 in the imaging unit coordinate system Σ_c. Called the posture.

  Hereinafter, with reference to FIG. 8, a process in which the control unit 36 operates the robot 20 by visual servo based on the target image will be described. FIG. 8 is a flowchart illustrating an example of a flow of processing in which the control unit 36 operates the robot 20 by visual servo based on the target image. First, the robot control unit 42 is the position and orientation input from the user via the input receiving unit 33, and the assembly target O2 in the work target coordinate system Σ_m in the installed state as shown in FIG. A position p_o ^ m (→) and an attitude o_o ^ m (→) are set (step S200).

  Next, the robot control unit 42 is the rotation center position and rotation center posture input from the user via the input receiving unit 33, and the rotation center position p_e ^ m (→) and the rotation center in the work target coordinate system Σ_m. The posture o_e ^ m (→) is set (step S210). Steps S200 to S210 are initial settings to the control device 30 performed by the user. Next, the robot control unit 42 reads the target images from the storage unit 32 in chronological order (step S220).

  Next, the robot control unit 42 selects the target images read in step S220 in chronological order, and repeats the process of step S240 based on the selected target images (step S230). Each functional unit of the control unit 36 includes the selected target image, the set position p_o ^ m (→) and posture o_o ^ m (→) of the assembly target O2, and the rotation center position in the work target coordinate system Σ_m. Robot control processing for operating the robot 20 by visual servoing based on p_e ^ m (→) and the rotation center posture o_e ^ m (→) is performed (step S240).

  Here, with reference to FIG. 9, the robot control process based on the target image performed by the robot control unit 42 of the control unit 36 will be described. FIG. 9 is a flowchart illustrating an example of the flow of the robot control process based on the target image performed by the robot control unit 42 of the control unit 36. From here, it demonstrates that the imaging by the 2nd imaging part 12 was started after step S230, and the control part 36 acquired the 2nd captured image from the image acquisition part 35. FIG. After the control unit 36 acquires the second captured image, the robot control unit 42, based on the second captured image acquired by the image acquisition unit 35, the position p_o ^ c of the assembly target O2 in the imaging unit coordinate system Σ_c. (→) and posture o_o ^ c (→) are detected (step S300).

  Next, the robot control unit 42 determines the position p_o ^ m (→) and posture o_o ^ m (→) of the assembly target O2 in the work target coordinate system Σ_m set in the process of step S200 in the flowchart shown in FIG. And the position p_o ^ c (→) and orientation o_o ^ c (→) of the assembly target O2 in the imaging unit coordinate system Σ_c detected in step S300 and the selected target image, as shown below. From Expression (7), the target position and orientation of the work target M2 in the state of the work target M2 and the assembly target O2 represented by the selected target image, and the target position p_m ^ of the work target M2 in the imaging unit coordinate system Σ_c c (d) (→) and posture o_m ^ c (d) (→) are calculated (step S310). The target position is the position of the work target M2 in the imaging unit coordinate system Σ_c when the operation derived by the robot control unit 42 is completed. Hereinafter, the posture of the work target M2 in the imaging unit coordinate system Σ_c when the operation derived by the robot control unit 42 is completed is referred to as a target posture.

  More specifically, the robot control unit 42 detects the relative position and orientation in the imaging unit coordinate system Σ_c between the work target M2 and the assembly target O2 based on the target image. Note that the imaging unit coordinate system of the first imaging unit 11 and the imaging unit coordinate system Σ_c of the second imaging unit 12 are associated by calibration or the like. Then, the robot control unit 42 assembles the relative position and orientation in the imaging unit coordinate system Σ_c between the detected work target M2 and the assembly target O2, and the work target coordinate system Σ_m set in the process of step S200. The position p_o ^ m (→) and orientation o_o ^ m (→) of the target O2, and the position p_o ^ c (→) and orientation o_o ^ c () of the assembly target O2 in the imaging unit coordinate system Σ_c detected in step S300. →), from the above equation (7), the target position and orientation of the work target M2 represented by the selected target image in the state of the work target M2 and the assembly target O2, and the imaging unit coordinate system Σ_c The target position p_m ^ c (d) (→) and the posture o_m ^ c (d) (→) of the work target M2 are calculated.

  Here, “(d)” of the target position p_m ^ c (d) (→) and the posture o_m ^ c (d) (→) of the work target M2 in the imaging unit coordinate system Σ_c is determined in step S320 by the robot control unit 42. This is a label added to distinguish the position p_m ^ c (→) and the posture o_m ^ c (→) of the work target M2 in the imaging unit coordinate system Σ_c detected by the above.

  The processing from step S300 to step S310 is a state in which the control unit 36 represents the relative position and posture of the second imaging unit 12 and the assembly target O2, and the selected target image, and the work target M2 Based on the relative positional relationship (position and orientation) between the work target M2 and the assembly target O2 in the state of the assembly target O2, the work target M2 in the state represented by the selected target image, and the second imaging unit 12 Is a process for indirectly calculating the relative position and orientation.

  Next, the robot control unit 42 detects the position p_m ^ c (→) and the posture o_m ^ c (→) of the work target M2 in the imaging unit coordinate system Σ_c from the acquired second captured image (step S320). . Next, the robot control unit 42 detects the position p_c ^ t (→) and the posture o_c ^ t (→) of the second imaging unit 12 in the tool coordinate system Σ_t from the acquired second captured image (step S330). ).

  Next, the robot control unit 42 determines the position p_m ^ c (→) and posture o_m ^ c (→) of the work target M2 in the imaging unit coordinate system Σ_c calculated in step S320, and the tool coordinates calculated in step S330. Based on the position p_c ^ t (→) and the posture o_c ^ t (→) of the second imaging unit 12 in the system Σ_t, the position p_e ^ t of the rotation center in the tool coordinate system Σ_t is obtained from the following equation (8). (→) and posture o_e ^ (→) are calculated (step S340).

  Here, the process of step S340 is a process of detecting a change in the relative positional relationship between the force sensor 22 and the above-described rotation center position and rotation center posture. The difference between the rotation center position p_e ^ t (→) and the posture o_e ^ t (→) in the tool coordinate system Σ_t calculated in step S340 from the values calculated in the previous routine is that the force sensor 22 and the rotation center It shows that the relative positional relationship between the position and the rotation center posture is changed by an external force. Even if the rotation center position and the rotation center posture with respect to the force sensor 22 are changed by an external force, the control unit 36 detects the change to change the rotation center position and the rotation center posture with respect to the force sensor 22 after the change. Based on this, it is possible to control the robot 20 to always rotate the work target M2 with the rotation center position as the center and the rotation center posture as a reference.

  Next, the robot control unit 42 calculates the position p_t ^ w (→) and the posture o_t ^ w (→) of the force sensor 22 in the work coordinate system Σ_w based on forward kinematics (step S350). Next, the robot control unit 42 determines the rotation center position p_e ^ t (→) and the rotation center posture o_e ^ t (→) in the tool coordinate system Σ_t calculated in step S340, and the work coordinate system calculated in step S350. Based on the position p_t ^ w (→) and the posture o_t ^ w (→) of the force sensor 22 in Σ_w, the position p_w ^ e of the support base of the robot 20 in the external force coordinate system Σ_e is obtained from the following equation (9). (→) and posture o_we (→) are calculated (step S360).

  Here, the control unit 36 calculates the position p_w ^ e (→) and the posture o_w ^ e (→) of the support base of the robot 20 in the external force coordinate system Σ_e by the process of step S360, so that the force sensor 22 The force and moment applied to the rotation center position by the applied force and moment can be calculated in the next step S370.

  Next, the robot control unit 42 shows the force and moment detected by the force sensor 22 based on the force f ^ t (→) and moment m ^ t (→) in the tool coordinate system Σ_t. From the Newton-Euler equation (10), the force in the external force coordinate system Σ_e (that is, the force acting on the rotation center position) f ^ e (→) and the moment (that is, the moment generated at the rotation center position) m ^ e ( →) is calculated (step S370).

  Here, the upper equation in the above equation (10) indicates that the external force component f ^ e (→) applied to the rotation center position of the work target M2 is the force component f ^ t (→) detected by the force sensor 22. ) From the component mE (^) g (→) due to gravity and the component mE (^) p_c ^ w (→) (··) due to the inertial motion by the manipulator MNP of the robot 20 ing. The foremost character with “(••)” represents a variable that has been differentiated twice with respect to time. The matrix E (^) is a unit matrix, the vector g (→) is a vector representing gravitational acceleration, and the scalar m represents the mass of the work target M2.

  Similar to the upper equation, the lower equation represents that the moment m ^ e generated at the center of rotation due to the external force applied to the work target M2 is the torsional moment component I (^) o_c ^ w (→) (··). , Moment component o_c ^ w (→) (•) × I (^) o_c ^ w (→) (•) of Coriolis force, moment m ^ t (→) detected by the force sensor 22, and force From the sum of the moment component p_e ^ t (→) × f ^ t (→) due to the force component f ^ t (→) detected by the sensor 22, the moment component p_g ^ t (→) × due to gravity. mR_w ^ t (^) g (→) represents a component obtained by subtracting.

  Next, based on the force f ^ e (→) and moment m ^ e (→) in the external force coordinate system Σ_e calculated in step S370, the robot control unit 42 calculates the external force from the following equation (11). A small relaxation movement amount Δp ^ e (→) and a small relaxation rotation amount Δo ^ (→) of the relaxation operation amount of the work target M2 in the coordinate system Σ_e are calculated (step S380).

  Here, ΔP ^ e (→) and ΔO ^ e (→) in equation (11) used in step S380 are Laplace transforms of Δp ^ e (→) and Δo ^ e (→), respectively. Equation (11) determines the amount of movement (a small amount of movement and a small amount of rotation) that should move in a direction to relieve the external force when the rotation center position set for the work object M2 receives an external force. It depends on the exercise model.

  In this embodiment, as shown in the above equation (11), the inertial mass matrix M_p (^), the damper coefficient matrix D_p (^), the spring multiplier matrix K_p (^) related to the minute movement amount, and the minute rotation amount. A motion model represented by an inertial mass matrix M_o (^), a damper coefficient matrix D_o (^), and a spring multiplier matrix K_o (^) is employed.

  Control performed by such a motion model is called impedance control in compliant motion control. The scalar s is a variable used for Laplace transform. The compliant motion control is not limited to impedance control, and for example, stiffness control or damping control may be applied.

  Next, the robot control unit 42 calculates the force sensor 22 in the work coordinate system Σ_w from the following equation (12) based on the relaxation operation amount of the work target M2 in the external force coordinate system Σ_e calculated in step S380. A small relaxation movement amount Δp_t ^ w (i) (→) and a small relaxation rotation amount Δo_t ^ w (i) (→) are calculated as the relaxation operation amount (step S390).

  Here, “(i)” in the minute relaxation movement amount Δp_t ^ w (i) (→) and the minute relaxation rotation amount Δo_t ^ w (i) (→), which are relaxation movement amounts, represents the relaxation movement amount and the target movement amount. It is a label for distinguishing. The robot control unit 42 of the control unit 36 relaxes the force and moment applied to the rotation center position based on the relaxation operation amount in the work coordinate system Σ_w calculated by the robot control unit 42 through the processing from step S380 to step S390. The robot 20 can be controlled to move the position and posture of the force sensor 22 to the position where the force sensor 22 is moved.

  Next, the robot control unit 42 sets the target position p_m ^ c (d) (→) of the work target M2 in the imaging unit coordinate system Σ_c in the state of the work target M2 and the assembly target O2 represented by the target image calculated in step S310. ) And posture o_m ^ c (d) (→), and the position p_m ^ c (→) and posture o_m ^ c (→) of the work target M2 in the imaging unit coordinate system Σ_c detected in step S320. From the equation (13) shown in FIG. 5, the target motion amount for moving the work target M2 to the target position and posture, and the minute target movement amount Δp_m ^ c (→) as the target motion amount in the imaging unit coordinate system Σ_c The minute target rotation amount Δo_m ^ c (→) is calculated (step S400).

  Next, the robot control unit 42 calculates the force sensor 22 in the work coordinate system Σ_w from the following equation (14) based on the target motion amount of the work target M2 in the imaging unit coordinate system Σ_c calculated in step S400. As the target motion amount, a small target movement amount Δp_t ^ w (v) (→) and a small target rotation amount Δo_t ^ w (v) (→) are calculated (step S410).

  Here, “(v)” in the minute target movement amount Δp_t ^ w (v) (→) and the minute target rotation amount Δo_t ^ w (v) (→), which are target movement amounts, represents the target movement amount and the relaxation movement amount. It is a label for distinguishing.

  Next, the robot control unit 42 adds the small movement amount Δp_t ^ obtained by adding the mitigation movement amount calculated in step S390 and the target movement amount calculated in step S410 as shown in the following equation (15). w (→) and the additional minute rotation amount Δo_t ^ w (→) are calculated (step S420).

  Next, the robot control unit 42 determines whether or not both the addition minute movement amount Δp_t ^ w (→) and the addition minute rotation amount Δo_t ^ w (→) calculated in step S420 are less than a predetermined threshold value for each. Determination is made (step S430). Each of the predetermined thresholds for each of the above is set separately. When the robot control unit 42 determines that both the addition minute movement amount Δp_t ^ w (→) and the addition minute rotation amount Δo_t ^ w (→) are less than a predetermined threshold value for each of them (step S430-Yes). It is determined that there is no need to move the work object M2 any more, and the process is terminated.

  On the other hand, when the robot control unit 42 determines that one or both of the additional minute movement amount Δp_t ^ w (→) and the additional minute rotation amount Δo_t ^ w (→) is not less than a predetermined threshold for each ( In step S430-No), the robot control unit 42 calculates the work coordinate system from the following equation (16) based on the mitigation motion amount calculated in step S390 and the target motion amount calculated in step S410. The target position p_t ^ w (d) (→) and posture o_t ^ w (d) (→) of the force sensor 22 at Σ_w are calculated (step S440).

  Note that the robot control unit 42 determines whether or not both the added minute movement amount Δp_t ^ w (→) and the added minute rotation amount Δo_t ^ w (→) calculated in step S420 are less than a predetermined threshold value for each. Instead of the configuration to be performed, both the minute target movement amount Δp_t ^ w (v) (→) and the minute target rotation amount Δo_t ^ w (v) (→) calculated in step S410 are less than a predetermined threshold value for each. The structure which determines whether there exists may be sufficient. In addition, the robot control unit 42 determines Δp_t ^ w (→) (i), Δo_t ^ w (i), Δp_t ^ w (→) (v), and Δo_t ^ depending on the property of each operation that the robot 20 performs. Using an arbitrary function configured by w (v), it is determined whether or not the value obtained from the function is less than a predetermined threshold (whether or not the work target M2 needs to be moved further). May be.

  Next, the robot control unit 42 determines that the position and posture of the force sensor 22 are the target position p_t ^ w (d) (→) and the posture o_t ^ w () of the force sensor 22 in the work coordinate system Σ_w calculated in step S440. d) The robot 20 is controlled to move the force sensor 22 so as to coincide with (→) (step S450). The control unit 36 repeats the processes in steps S300 to S450 until it is determined that it is not necessary to move the work object M2 any more by the determination in step S430, thereby performing an operation for realizing the state represented by the target image. Control is performed by the robot 20.

  In the process from step S330 to step S360, for example, the robot control unit 42 is configured such that the force sensor 22 is provided inside the manipulator MNP and cannot be seen from the outside, and the second imaging unit 12 images the force sensor 22. If it is not possible, it may be configured to include a marker indicating the position of the force sensor 22 described above, based on the relative positional relationship in the initial state of the force sensor 22 and the rotation center position and rotation center posture set by the user, From the following equations (17) to (19), the position p_t ^ e (→) and the posture o_t ^ e (→) of the force sensor 22 in the external force coordinate system Σ_e may be sequentially calculated.

  The above equations (17) to (19) are obtained by subtracting the amount of movement of the robot 20 itself from the amount of movement of the work target M2 detected from the second captured image, so that the relative force sensor 22 and the work target M2 are relative to each other. This is an expression for calculating the amount of deviation of the general positional relationship. Note that the amount of movement of the robot 20 itself is calculated by the control unit 36 based on the initial state of the robot 20. In the configuration using the force sensor 22 set by the user and the relative positional relationship between the rotation center position and the rotation center posture in the initial state and the above equations (17) to (19), the work object M2 and the force sensor 22 2 Even when the position of the force sensor 22 cannot be detected even if a marker indicating the position of the force sensor 22 is used, the position, posture, and force of the work target M2 are not included in the imaging range of the imaging unit 12. This is useful because the position and orientation of the sensor 22 can be tracked.

  The robot control unit 42 performs the sequential calculation using the above equations (17) to (19), so that even if the force sensor 22 cannot be imaged by the second imaging unit 12, the second Similar to the case where the force sensor 22 can be imaged by the imaging unit 12, the processing from step S330 to step S360 can be performed. As a result, the control unit 36 can control the robot 20 so that the robot 20 operates based on the relative positional relationship between the force sensor 22 and a predetermined position set for the work target M2. Note that the case where the force sensor 22 cannot capture an image by the second imaging unit 12 includes, for example, a case where the force sensor 22 is covered by a member of the manipulator MNP, or a force sensor based on the angle of view of the second imaging unit 12. For example, the manipulator MNP including 22 may come off.

<Variation 1 of the first embodiment>
Hereinafter, the modification 1 of 1st Embodiment of this invention is demonstrated. Instead of the configuration in which the control unit 36 of the control device 30 according to the first modification of the first embodiment selects a target image extracted from the first captured image and performs visual servoing based on the selected target image, A still image received from the input receiving unit 33 by the user is selected as a target image together with the target image extracted from the first captured image, and visual servoing is performed based on the selected target image.

  Thereby, the control part 36 can perform visual servo based on the target image showing the state of the work target M2 and the assembly target O2 that cannot be extracted from the first captured image according to a predetermined condition. As a result, the user can more easily teach the controller 36 more detailed operations of the robot 20.

<Modification 2 of the first embodiment>
Hereinafter, Modification 2 of the first embodiment of the present invention will be described. The control unit 36 of the control device 30 according to the second modification of the first embodiment extracts a frame that satisfies a predetermined condition from the first captured image as a target image. The predetermined condition is, for example, that the work target M1 is stationary for a predetermined time.

  Thereby, when the user captures the first captured image by the first imaging unit 11, the user stops the work target M2 for at least a predetermined time for each state related to the operation to be taught to the control unit 36. It is possible to cause the control unit 36 to extract a desired target image. As a result, the user can more easily teach the controller 36 more detailed operations of the robot 20.

  As described above, the robot system 1 according to the first embodiment receives a plurality of first captured images obtained by capturing the movement of the work target M1 in time series, and the movement of the work target M1 from the plurality of first captured images. A frame when the state changes is extracted as a target image, and the manipulator MNP is operated by visual servoing using the extracted target image. Thereby, the robot system 1 can acquire the information of the teaching content without a large error. As a result, the robot can perform the taught motion by the visual servo. Further, the robot system 1 can easily teach the operation.

  Further, the robot system 1 extracts a frame when the movement state of the work target M1 is changed as a target image, and operates the manipulator MNP by visual servo using the extracted target image. Thereby, the robot system 1 can acquire the teaching content based on the target image when the movement state of the work target M1 changes.

  Further, the robot system 1 extracts a frame when the speed of the work target M1 changes as a target image, and operates the manipulator MNP by visual servo using the extracted target image. Thereby, the robot system 1 can acquire the teaching content based on the target image when the speed of the work target M1 changes.

  Further, the robot system 1 extracts a frame when the contact state of the work target M1 is changed as a target image, and operates the manipulator MNP by visual servo using the extracted target image. Thereby, the robot system 1 can acquire the teaching content based on the target image when the contact state of the work target M1 changes.

  Further, the robot system 1 assembles the work target M2 to the assembly target O2 by moving the manipulator MNP by visual servo using the frame when the speed of the work target M1 becomes zero. Thereby, the robot system 1 can use the frame when the speed of the work target M2 becomes zero as a target image, and can set the time when the speed becomes zero as a segment of one operation.

  Also, the robot system 1 uses the extracted image obtained by extracting a desired scene from the first captured image as a target image, and moves the manipulator MNP by visual servo to assemble the work target M2 to the assembly target O2. . Thereby, the robot system 1 can acquire more detailed teaching content.

Second Embodiment
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. The robot system 2 according to the second embodiment has a configuration in which a double-arm robot is provided as the robot 25 instead of a configuration in which the robot 20 is provided with a single-arm robot. Note that in the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  FIG. 10 is a configuration diagram illustrating an example of the robot system 2 according to the second embodiment. The robot system 2 includes a first imaging unit 11, a second imaging unit 12, a robot 25, and a control device 30. In the second embodiment, the assembly target O2 is installed on a table such as a table with a jig or the like, and the one of the arms of the robot 25 that is a double-arm robot is described in the first embodiment. The work object M2 is assembled by visual servoing based on the target image. The assembly target O2 is gripped by the gripping unit HND2 of the robot 25, and the gripper HND1 is an operation related to the assembly of the work target M2 to the assembly target O2 by the visual servo based on the target image described in the first embodiment. May be performed. In this case, the roles of the gripping part HND1 and the gripping part HND2 may be reversed.

  The robot 25 includes, for example, a gripper HND1, a gripper HND2, a force sensor 22, a manipulator MNP1, a manipulator MNP2, and a plurality of actuators (not shown) on each arm as shown in FIG. A double-armed robot.

  Each arm of the robot 25 is a six-axis vertical articulated type, and one arm can perform a six-axis freedom movement by a support base, a manipulator MNP1, and a gripper HND1 that are coordinated by an actuator. The other arm can be operated with six axes of freedom by the operation in which the support base, the manipulator MNP2, and the gripper HND2 are linked by the actuator. Each arm of the robot 20 may operate with 5 degrees of freedom (5 axes) or less, or may operate with 7 degrees of freedom (7 axes) or more.

  The robot 25 performs the operation controlled by the control device 30 described in the first embodiment with the arm including the gripping unit HND1 and the manipulator MNP1, but the same operation is performed with the arm including the gripping unit HND2 and the manipulator MNP2. An operation may be performed. The manipulator MNP1 (or the manipulator MNP1 and the grip portion HND1) and the manipulator MNP2 (or the manipulator MNP2 and the grip portion HND2) are examples of arms, respectively. The robot 25 is communicably connected to the control device 30 by a cable, for example. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The robot 25 and the control device 30 may be connected by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  Further, the robot 25 is configured such that the robot 25 is controlled by a control device 30 mounted inside the robot 25 as shown in FIG. The structure installed in may be sufficient.

  As described above, the robot 25 of the robot system 2 according to the second embodiment is a dual-arm robot, and extracts a target image of at least one frame from the first captured image captured by the first imaging unit 11. Since the task of assembling the work target M2 to the assembly target O2 is performed by one or both of the two arms of the double-arm robot based on the extracted target image, the same effect as in the first embodiment can be obtained. Can do.

  In the above-described embodiment, the first imaging unit 11 captures the first captured image, and the second imaging unit 12 captures the second captured image. Instead, the first imaging unit 11 Alternatively, a configuration may be adopted in which either the first captured image or the second captured image is captured by either one of the second imaging units 12. In that case, the timing for capturing the first captured image is always before the timing for capturing the second captured image.

  In addition, a program for realizing the functions of arbitrary components in the devices described above (for example, robot systems 1 and 2) is recorded on a computer-readable recording medium, and the program is read into the computer system. You may make it perform. Here, the “computer system” includes hardware such as an OS (Operating System) and peripheral devices. “Computer-readable recording medium” means a portable disk such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a CD (Compact Disk) -ROM, or a hard disk built in a computer system. Refers to the device. Further, the “computer-readable recording medium” means a volatile memory (RAM: Random Access) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Memory that holds a program for a certain period of time, such as Memory).

In addition, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the functions described above. Further, the program may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

DESCRIPTION OF SYMBOLS 1, 2 Robot system, 11 1st imaging part, 12 2nd imaging part, 20, 25 Robot, 22 Force sensor, 30 Control apparatus, 31 CPU, 32 Storage part, 33 Input reception part, 34 Communication part, 35 Image acquisition Unit, 36 control unit, 41 target image extraction unit, 42 robot control unit

Claims (8)

Arm,
A control device for operating the arm;
Including
The controller is
Receiving a plurality of first images obtained by imaging the movement of the work object in time series;
Extracting a second image when the movement state of the work object is changed from the plurality of first images;
Using the extracted second image, the arm is operated by visual servoing.
robot. The robot according to claim 1,
The movement state of the work object is a movement state of the work object.
robot. The robot according to claim 2,
The second image when the movement state of the work object is changed is an image when the speed of the work object is changed.
robot. The robot according to claim 1,
The movement state of the work object is a contact state of the work object.
robot. The robot according to any one of claims 1 to 4,
The second image is an image selected from the plurality of first images input from the input receiving unit.
robot. An imaging unit;
A robot with an arm;
A control device for operating the arm;
Comprising
The controller is
Receiving a plurality of first images captured in time series by the imaging unit for the movement of the work object,
Extracting a second image when the movement state of the work object is changed from the plurality of first images;
Using the extracted second image, the arm is operated by visual servoing.
Robot system. Receiving a plurality of first images obtained by imaging the movement of the work object in time series;
Extracting a second image when the movement state of the work object is changed from the plurality of first images;
Using the extracted second image, the robot arm is operated by visual servoing.
Control device. Receiving a plurality of first images obtained by imaging the movement of the work object in time series;
Extracting a second image when the movement state of the work object is changed from the plurality of first images;
Using the extracted second image, the robot arm is operated by visual servoing.
Control method.

Images