JP2018034245A - Robot, robot control device, and robot system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot which can carry out work relating to a first object accurately.SOLUTION: A robot is provided with an arm, and moves the arm based on a position of the first object detected by using a first image which images a first range without the first object and a second image which images a second range overlapping with at least a part of the first range and including the first object.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a robot, a robot control device, and a robot system.

  Research and development of a technique for controlling a robot based on a captured image captured by an imaging unit has been performed.

  In this regard, the difference between the first captured image obtained by imaging the object by the first imaging device and the model image of the object in the first virtual space corresponding to the space imaged by the first imaging device. A first degree of difference indicating a degree is calculated for each of a plurality of different postures of the object in the first virtual space, and the second imaging device arranged at a position different from the first imaging device selects the object. A second dissimilarity indicating a dissimilarity between the second captured image obtained by imaging and the model image of the object in the second virtual space corresponding to the space captured by the second imaging device is expressed as the second difference. The orientation of the object in the first imaging coordinate system is calculated for each of the plurality of postures of the object in the virtual space, and based on the calculated plurality of first differences and the calculated plurality of second differences. There is known an attitude detection device that estimates the angle (see Patent Document 1).

JP 2014-161937 A

  However, in such a posture detection apparatus, a background other than the target object (for example, an object to which the target object is assembled) included in the captured image obtained by capturing the target object may be erroneously detected as the target object. It was. As a result, the posture detection device may not be able to accurately detect the position of the object.

In order to solve at least one of the above problems, one embodiment of the present invention includes a first image that includes an arm and includes a first range that does not include a first object, and at least a part of the first range A robot that moves the arm based on a position of the first object detected using a second image obtained by imaging a second range that is an overlapping range and includes the first object.
With this configuration, the robot captures the first image in which the first range that does not include the first object is captured, and the second range that includes at least a part of the first range and includes the first object. The arm is moved based on the position of the first object detected using the second image. Thereby, the robot can perform the work regarding the first object with high accuracy.

According to another aspect of the present invention, in the robot, a difference image based on a difference between the first image and the second image is generated, and the first object detected using the generated difference image. A configuration in which the arm is moved based on the position may be used.
With this configuration, the robot generates a difference image based on the difference between the first image and the second image, and moves the arm based on the position of the first object detected using the generated difference image. Thereby, the robot can perform the work regarding the first object with high accuracy based on the position of the first object detected using the difference image.

According to another aspect of the present invention, there is provided a gain relating to a position of the first object detected using the first image and the second image and a movement amount for moving the arm in the robot. A configuration in which the arm is moved based on the gain of the servo may be used.
With this configuration, the robot moves the arm based on the position of the first object detected using the first image and the second image, and a gain related to the amount of movement for moving the arm and the gain of the visual servo. . Thereby, the robot can perform the work regarding the first object with high accuracy based on the position and gain of the first object detected using the first image and the second image.

According to another aspect of the present invention, in the robot, the first object is gripped by the arm, and the first object detected using the first image and the second image is detected. A configuration in which the arm is moved based on a difference between a position and a target position that is a target for changing the position of the first object may be used.
With this configuration, the robot can move the arm based on the difference between the position of the first object detected using the first image and the second image and the target position that is the target for changing the position of the first object. Move. Thereby, the robot can perform the work regarding the first object with high accuracy based on the difference between the position of the first object detected using the first image and the second image and the target position.

According to another aspect of the present invention, in the robot, the first object is an object assembled to the second object by the arm, and a third range that does not include the second object is imaged. A position of the second object detected using a fourth image in which an image and a fourth range including the second object are imaged and overlapping with at least a part of the third range; A configuration may be used in which the first object and the second object are assembled based on target position information indicating a relative position between the position and the target position.
With this configuration, the robot assembles the first object and the second object based on the position of the second object detected using the third image and the fourth image and the target position information. Thereby, the robot performs an operation of assembling the first object and the second object based on the position of the second object detected using the third image and the fourth image and the target position information. It can be performed with high accuracy.

According to another aspect of the present invention, in the robot, a configuration in which the third image is the second image may be used.
With this configuration, in the robot, the third image is the second image. Thereby, the robot can shorten the time required for the work based on the position of the first object.

In another aspect of the present invention, the robot may be configured to move the arm based on the posture of the first object detected using the first image and the second image. Good.
With this configuration, the robot moves the arm based on the posture of the first object detected using the first image and the second image. Thereby, the robot can perform the work regarding the first object with high accuracy based on the position and orientation of the first object.

According to another aspect of the present invention, in the robot, the posture of the first object detected using the first image and the second image, and a target for changing the posture of the first object. A configuration may be used in which the arm is moved based on a difference from a target posture.
With this configuration, the robot can move the arm based on the difference between the posture of the first object detected using the first image and the second image and the target posture that is a target for changing the posture of the first object. Move. As a result, the robot can accurately perform the work on the first object based on the difference between the posture of the first object detected using the first image and the second image and the target posture. .

According to another aspect of the present invention, in the robot, the first object is an object to which the second object held by the arm is assembled, and the first image and the second image are used. Target position information indicating a detected position of the first object, a relative position between the position and a target position to be a target for changing the position of the second object, and the position of the second object A configuration may be used that moves the arm based on
With this configuration, the robot moves the arm based on the position of the first object detected using the first image and the second image, target position information, and the position of the second object. As a result, the robot can work on the first object based on the position of the first object detected using the first image and the second image, the target position information, and the position of the second object. Can be performed with high accuracy.

According to another aspect of the present invention, in the robot, the position of the first object detected using the first image and the second image, the target position information, and the second object Detected using a third image in which a third range not included is captured, and a fourth image in which a fourth range including at least a part of the third range and including the second object is captured. In addition, a configuration in which the first object and the second object are assembled based on the position of the second object may be used.
With this configuration, the robot detects the position of the first object detected using the first image and the second image, target position information, the second image detected using the third image, and the fourth image. The first object and the second object are assembled based on the position of the object. Thereby, the robot detects the position of the first object detected using the first image and the second image, the target position information, the third image, and the second object detected using the fourth image. Based on the position of the object, the work related to the first object can be accurately performed.

According to another aspect of the present invention, in the robot, the posture of the first object detected using the first image and the second image, and the posture and the posture of the second object are changed. The structure which moves the said arm based on the target attitude | position information which shows a relative attitude | position with the target attitude | position used as the target to be made may be used.
With this configuration, the robot moves the arm based on the posture of the first object detected using the first image and the second image and the target posture information. Thereby, the robot can perform the work regarding the first object with high accuracy based on the posture of the first object detected using the first image and the second image and the target posture information.

Another aspect of the present invention is a robot control apparatus that controls the robot described above.
With this configuration, the robot control device includes a first image in which a first range that does not include the first object is captured, and a second range that includes at least a part of the first range and includes the first object. The arm is moved based on the position of the first object detected using the second image obtained by capturing. Thereby, the robot control apparatus can cause the robot to accurately perform the work related to the first object.

Another aspect of the present invention is a robot system including the robot described above and the robot control device described above.
With this configuration, the robot system includes a first image in which the first range that does not include the first object is captured, and a second range that includes at least a part of the first range and includes the first object. The arm is moved based on the position of the first object detected using the captured second image. Accordingly, the robot system can cause the robot to perform the work related to the first object with high accuracy.

As described above, the robot detects the first object detected using the first image in which the range including the first object is captured and the second image in which the range not including the first object is captured. Move the arm based on the position. Thereby, the robot can perform the work performed based on the position of the first object with high accuracy.
In addition, the robot control device and the robot system include a first image in which a first range that does not include the first target is captured, and a first range that overlaps at least a part of the first range and includes the first target. The arm is moved based on the position of the first object detected using the second image obtained by imaging the two ranges. Thereby, the robot control device and the robot system can cause the robot to accurately perform the work related to the first object.

It is a figure showing an example of composition of robot system 1 concerning an embodiment. 2 is a diagram illustrating an example of a hardware configuration of a robot control device 30. FIG. 2 is a diagram illustrating an example of a functional configuration of a robot control device 30. FIG. 4 is a flowchart illustrating an example of a flow of processing in which the robot control device 30 causes the robot 20 to perform a predetermined operation. It is an example of a 1st image. It is an example of a 2nd image. It is an example of a 4th image. It is an example of the 1st difference image which the position and orientation calculation part 46 produced | generated. It is an example of the 2nd difference image which the position and orientation calculation part 46 produced | generated. It is a figure which shows the other example of the flow of the process in which the robot control apparatus 30 makes the robot 20 perform predetermined work. It is a figure which shows an example of a structure of the robot system 2 which concerns on the modification 2 of embodiment. It is a figure which shows the other example of a structure of the robot system 2 which concerns on the modification 2 of embodiment.

<Embodiment>
Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Robot system configuration>
First, the configuration of the robot system 1 will be described.
FIG. 1 is a diagram illustrating an example of a configuration of a robot system 1 according to the embodiment. The robot system 1 includes an imaging unit 10, a robot 20, and a robot control device 30.

  The imaging unit 10 is a camera including, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, which is an imaging element that converts collected light into an electrical signal. In this example, the imaging unit 10 is installed at a position where an imaging range that is a range including a work area in which the robot 20 can work is taken.

  The imaging unit 10 is connected to the robot control device 30 through a cable so as to be communicable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB (Universal Serial Bus), for example. The imaging unit 10 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The robot 20 is a single-arm robot including an arm A and a support base B that supports the arm A. The single arm robot is a robot having one arm (arm) like the arm A in this example. The robot 20 may be a multi-arm robot instead of the single-arm robot. The multi-arm robot is a robot having two or more arms (for example, two or more arms A). Of the multi-arm robot, a robot having two arms is also referred to as a double-arm robot. That is, the robot 20 may be a double-arm robot having two arms or a multi-arm robot having three or more arms (for example, three or more arms A). The robot 20 may be another robot such as a SCARA robot, a Cartesian coordinate robot, or a cylindrical robot. The orthogonal coordinate robot is, for example, a gantry robot.

The arm A includes an end effector E and a manipulator M.
In this example, the end effector E is an end effector including a finger portion that can grip an object. The end effector E may be an end effector capable of lifting an object by air suction, a magnetic force, a jig or the like, or another end effector, instead of the end effector including the finger portion.

  The end effector E is communicably connected to the robot control device 30 via a cable. Thereby, the end effector E performs an operation based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The end effector E may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The manipulator M includes six joints. Each of the six joints includes an actuator (not shown). That is, the arm A including the manipulator M is a 6-axis vertical articulated arm. The arm A performs an operation with six degrees of freedom by a coordinated operation by the support base B, the end effector E, the manipulator M, and the actuators of each of the six joints included in the manipulator M. The arm A may be configured to operate with a degree of freedom of 5 axes or less, or may be configured to operate with a degree of freedom of 7 axes or more.

  Each of the six actuators (provided at the joints) included in the manipulator M is communicably connected to the robot controller 30 via a cable. Thereby, the actuator operates the manipulator M based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. In addition, a part or all of the six actuators included in the manipulator M may be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  In this example, the robot control device 30 is a controller that controls (operates) the robot 20. The robot control device 30 causes the imaging unit 10 to image a range including the above-described work area. The robot control device 30 acquires a captured image captured by the imaging unit 10 from the imaging unit 10. The robot control device 30 causes the robot 20 to perform a predetermined operation based on the acquired captured image.

<Outline of predetermined work performed by robot>
Hereinafter, an outline of predetermined work performed by the robot 20 will be described.

  As shown in FIG. 1, the robot 20 holds the operation target N by the end effector E in advance. Note that the robot 20 may be configured to hold the operation target N arranged at a predetermined arrangement place by the end effector E in a predetermined operation.

  The operation target N is an object to be assembled to the assembly target O supported by the jig F installed on the floor surface of the work area. The operation target N is, for example, an industrial part or member such as a plate, screw, or bolt that is assembled to a product. In FIG. 1, for simplification of the drawing, the operation target N is represented as a hexagonal prism-shaped object having a height that can be gripped by the finger portion of the end effector E. The operation target N may be other objects such as daily necessities and living bodies instead of industrial parts and members. Further, the shape of the operation target N may be another shape instead of the hexagonal prism shape. The operation object N is an example of the first object when the assembly object O is an example of the second object, and the second object when the assembly object O is an example of the first object It is an example of a thing.

  The assembly target object O is an object having a recess in which the operation target object N is assembled. The assembly object O is, for example, an industrial part or member such as a plate, screw, or bolt that is assembled to a product. The assembly object O may be other objects such as daily necessities and living bodies instead of industrial parts and members. Further, the shape of the assembly object O may be another shape instead of the shape shown in FIG. Moreover, the assembly | attachment target object O is supported by the jig | tool F installed in the floor surface of a work area as mentioned above. Note that the assembly object O may be configured to be installed in another place such as a table or some table instead. Further, the jig F may be configured to be installed in another place such as a table or some table in the work area, instead of the configuration installed on the floor surface. The assembly object O is an example of the second object when the operation object N is an example of the first object, and the first object when the operation object N is an example of the second object It is an example.

  In this example, the robot 20 performs a work of assembling the operation target N gripped by the end effector E on the assembly target O as a predetermined work. The contour VN illustrated in FIG. 1 represents the contour of the operation target N when the operation target N is assembled to the assembly target O. Instead of this, the robot 20 may be configured to perform other work as a predetermined work.

  Here, in order to simplify the description, a case will be described in which the robot 20 moves the operation target N along a plane perpendicular to the optical axis of the imaging unit 10 while the robot 20 performs a predetermined operation. . That is, in this example, the operation target N performs a two-dimensional movement along the plane until it is assembled to the assembly target O. The robot 20 may be configured to cause the operation target N to perform a three-dimensional motion in the work area while the robot 20 performs a predetermined work. In this case, the imaging unit 10 is a camera that can capture an image capable of acquiring three-dimensional information, such as a stereo camera or a light field camera.

<Outline of the process in which the robot control device causes the robot to perform a predetermined work>
Hereinafter, an outline of processing in which the robot control apparatus 30 causes the robot 20 to perform a predetermined operation will be described.

  The robot control device 30 sets a control point T that moves together with the end effector E at a position previously associated with the end effector E. The position previously associated with the end effector E is a position in the robot coordinate system RC. The position associated with the end effector E in advance is, for example, the position of the center of gravity of the operation target N gripped by the end effector E.

  The control point T is, for example, a TCP (Tool Center Point). The control point T may be another virtual point such as a virtual point associated with a part of the arm A instead of TCP. That is, the control point T may be configured to be set at the position of another part of the end effector E instead of the position associated with the end effector E, and at a certain position associated with the manipulator M. The configuration may be set.

  The control point T is associated with control point position information, which is information indicating the position of the control point T, and control point attitude information, which is information indicating the attitude of the control point T. The position is a position in the robot coordinate system RC. The posture is a posture in the robot coordinate system RC. The control point T may be configured to be associated with other information in addition to these. When the robot control device 30 designates (determines) the control point position information and the control point posture information, the position and posture of the control point T are determined. The said position and the said attitude | position are the position and attitude | position in the robot coordinate system RC. The robot control device 30 operates the arm A to match the position of the control point T with the position indicated by the control point position information designated by the robot control device 30, and the control point posture information designated by the robot control device 30 indicates. The posture of the control point T is matched with the posture. Hereinafter, for convenience of explanation, the position indicated by the control point position information designated by the robot control device 30 will be referred to as a target position, and the posture indicated by the control point posture information designated by the robot control device 30 will be referred to as a target posture. That is, the robot control device 30 operates the robot 20 by designating the control point position information and the control point posture information, and matches the position and posture of the control point T with the target position and target posture.

  In this example, the position of the control point T is represented by the position of the origin of the control point coordinate system TC in the robot coordinate system RC. The attitude of the control point T is represented by the direction in the robot coordinate system RC of each coordinate axis of the control point coordinate system TC. The control point coordinate system TC is a three-dimensional local coordinate system associated with the control point T so as to move with the control point T.

  The robot control device 30 sets the control point T based on the control point setting information input in advance by the user. The control point setting information is, for example, information indicating a relative position and posture between the position and posture of the center of gravity of the end effector E and the position and posture of the control point T. Instead of this, the control point setting information may be information indicating a relative position and posture between any position and posture associated with the end effector E and the position and posture of the control point T. It may be information indicating a relative position and posture between any position and posture associated with the manipulator M and the position and posture of the control point T, and any position and posture associated with other parts of the robot 20 It may be information indicating a relative position and posture between the posture and the position and posture of the control point T.

  Here, the robot control device 30 includes a first image in which a first range that does not include the first object is captured, and a second range that includes at least a part of the first range and includes the first object. The arm A is moved based on the position of the operation target N detected using the second image obtained by capturing. In this example, since the range in which the imaging unit 10 can capture images does not change, the second range is the same range as the first range. Moreover, below, the case where the 1st target object is the operation target object N is demonstrated as an example. Thereby, the robot control apparatus 30 can perform the work (in this example, the above-described predetermined work) related to the first object (in this example, the operation object N) with high accuracy. The first object may be an assembly object O instead of the operation object N.

<Hardware configuration of robot controller>
Hereinafter, the hardware configuration of the robot control device 30 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the robot control device 30.

  The robot control device 30 includes, for example, a CPU (Central Processing Unit) 31, a storage unit 32, an input receiving unit 33, a communication unit 34, and a display unit 35. These components are connected to each other via a bus Bus so that they can communicate with each other. Further, the robot control device 30 communicates with the robot 20 via the communication unit 34.

The CPU 31 executes various programs stored in the storage unit 32.
The storage unit 32 includes, for example, a hard disk drive (HDD), a solid state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), and a random access memory (RAM). The storage unit 32 may be an external storage device connected via a digital input / output port such as a USB instead of the one built in the robot control device 30. The storage unit 32 stores various information processed by the robot control device 30, various programs, various images, and the like.

The input receiving unit 33 is, for example, a touch panel configured integrally with the display unit 35. The input receiving unit 33 may be a keyboard, a mouse, a touch pad, or other input device.
The communication unit 34 includes, for example, a digital input / output port such as USB, an Ethernet (registered trademark) port, and the like.
The display unit 35 is, for example, a liquid crystal display panel or an organic EL (ElectroLuminescence) display panel.

<Functional configuration of robot controller>
Hereinafter, the functional configuration of the robot control device 30 will be described with reference to FIG. 3. FIG. 3 is a diagram illustrating an example of a functional configuration of the robot control device 30.

  The robot control device 30 includes a storage unit 32 and a control unit 36.

  The control unit 36 controls the entire robot control device 30. The control unit 36 includes an imaging control unit 40, an image acquisition unit 42, a position / orientation calculation unit 46, and a robot control unit 50. These functional units included in the control unit 36 are realized, for example, when the CPU 31 executes various programs stored in the storage unit 32. Further, some or all of the functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

The imaging control unit 40 causes the imaging unit 10 to capture a range that can be captured by the imaging unit 10.
The image acquisition unit 42 acquires a captured image captured by the imaging unit 10 from the imaging unit 10.
The position / orientation calculation unit 46 calculates the position and orientation of the operation target N based on the captured image acquired by the image acquisition unit 42 from the imaging unit 10. Further, the position / orientation calculation unit 46 calculates the position and orientation of the assembly target object O based on the captured image acquired by the image acquisition unit 42 from the imaging unit 10. Note that the position and orientation calculation unit 46 may be configured to calculate the position of the operation target N and the position of the assembly target O based on the captured image. In this case, the posture of the operation target N and the posture of the assembly target O are fixed and do not change.
Based on the position and orientation of the operation target N calculated by the position / orientation calculation unit 46 and the position and orientation of the assembly target object O calculated by the position / orientation calculation unit 46, the robot control unit 50 applies predetermined information to the robot 20. Let's do the work.

<Process in which the robot controller causes the robot to perform a predetermined work>
Hereinafter, with reference to FIG. 4, a process in which the robot control device 30 causes the robot 20 to perform a predetermined operation will be described. FIG. 4 is a flowchart illustrating an example of a process flow in which the robot control device 30 causes the robot 20 to perform a predetermined operation.

  In the following, in the time zone before the process of step S130 in the flowchart shown in FIG. 4 is performed, the assembly object O is not attached to the jig F described above (that is, the assembly object O). Is not supported by the jig F), and the case where the assembly target object O is not included in the range in which the imaging unit 10 can capture an image will be described as an example. In the following, the end effector E, the operation target N gripped by the end effector E, and the manipulator M are included in the range in the time zone before the process of step S160 in the flowchart is performed. The case where there is no case is described as an example. That is, in this time zone, the end effector E, the operation target N gripped by the end effector E, and the manipulator M are located outside the range of the angle of view of the imaging unit 10 in this example.

  The imaging control unit 40 causes the imaging unit 10 to capture a range that can be captured by the imaging unit 10 (step S110). Hereinafter, for convenience of explanation, the range captured by the imaging unit 10 in step S110 will be referred to as a first range. Next, the image acquisition unit 42 acquires the captured image captured by the imaging unit 10 in step S110 as a first image (step S120). Here, the first image will be described with reference to FIG.

  FIG. 5 is an example of the first image. The first image P1 illustrated in FIG. 5 is an example of the first image. As described above, the first range in this example does not include the assembly target object O, the operation target object N gripped by the end effector E, the end effector E, and the manipulator M. For this reason, as shown in FIG. 5, the first image P <b> 1 includes a jig F and other objects (not shown) (for example, a floor surface or a work table).

  After the processing of step S120 is performed, the imaging control unit 40 continues until the assembly target object O is attached to the jig F by the user or another device (until the assembly target object O is supported by the jig F). ) Wait (step S130). The state of the assembly target object O and the jig F shown in FIG. 1 is determined after the assembly target object O is attached to the jig F by the user or another device in step S130. It is in a state after being supported by F.

  After the assembly target object O is attached to the jig F in step S130, the imaging control unit 40 causes the imaging unit 10 to image a range that can be imaged by the imaging unit 10 (step S140). Hereinafter, for convenience of description, the range captured by the imaging unit 10 in step S140 will be referred to as a second range. As described above, the second range is the same as the first range because the imageable range of the imaging unit 10 in this example does not change. Next, the image acquisition unit 42 acquires the captured image captured by the imaging unit 10 in step S140 from the imaging unit 10 as a second image (step S150). Here, the second image will be described with reference to FIG.

  FIG. 6 is an example of the second image. The second image P2 illustrated in FIG. 6 is an example of a second image. In the second range in this example, the assembly object O is attached to the jig F, and each of the operation object N, the end effector E, and the manipulator M gripped by the end effector E is not included. Therefore, in the second image P2, as shown in FIG. 6, the jig F, the assembly object O supported by the jig F, and other objects not shown (floor surface, work table, etc.) It is included. The contour VN shown in FIG. 6 represents the contour of the operation target N when the operation target N is assembled to the assembly target O.

  Here, since the second image P2 is an image that does not include the operation target N, it can be used as a third image that is captured by the imaging unit 10 in a third range that does not include the operation target N. . For this reason, below, the case where the robot control apparatus 30 utilizes the 2nd image P2 also as a 3rd image is demonstrated as an example. Note that the robot control device 30 again sets the range that can be imaged by the imaging unit 10 as the third range in the imaging unit 10 after the process of step S150 and before the process of step S160 is performed. It may be configured to take an image. In this case, the robot control apparatus 30 acquires the image captured by the imaging unit 10 from the imaging unit 10 as a third image. In this example, the imageable range of the imaging unit 10 does not change, so the third range is the same range as the first range.

  After the process of step S150 is performed, the robot control unit 50 reads out the initial position and orientation information stored in advance in the storage unit 32 from the storage unit 32. The initial position / posture information is information indicating a predetermined initial position and initial posture. The initial position and the initial posture are positions and postures in which the entire operation target N is included within the imageable range of the imaging unit 10 when the position and posture of the control point T are matched with the initial position and initial posture. Any position and orientation may be used. The robot control unit 50 operates the robot 20 based on the read initial position and orientation information, and matches the position and orientation of the control point T with the initial position and initial orientation, so that the inside of the range that can be imaged by the imaging unit 10. The operation target N is moved to (step S160).

  Next, the imaging control unit 40 causes the imaging unit 10 to capture a range that can be captured by the imaging unit 10 (step S170). Hereinafter, for convenience of explanation, the range captured by the imaging unit 10 in step S170 will be referred to as a fourth range. Next, the image acquisition unit 42 acquires the captured image captured by the imaging unit 10 in step S170 as a fourth image (step S180). Here, the fourth image will be described with reference to FIG.

  FIG. 7 is an example of the fourth image. The fourth image P4 illustrated in FIG. 7 is an example of a fourth image. In the fourth range in this example, the assembly object O is attached to the jig F, and the operation object N gripped by the end effector E is included. Therefore, in the fourth image P4, as shown in FIG. 7, the jig F, the assembly object O supported by the jig F, the operation object N gripped by the end effector E, An end effector E, a part of a manipulator M (not shown), and other objects (a floor surface, a work table, etc.) not shown are included. The contour VN shown in FIG. 7 represents the contour of the operation target N when the operation target N is assembled to the assembly target O.

  After the process of step S180 is performed, the position / orientation calculation unit 46 generates a first difference image based on the first image P1 and the second image P2 acquired by the image acquisition unit 42. Further, the position / orientation calculation unit 46 generates a second difference image based on the third image (in this example, the second image P2) and the fourth image P4 (step S190). Here, the process of step S190 will be described.

  First, the first difference image will be described. Hereinafter, the calculation method of the difference will be specifically described by representing coordinates indicating a certain pixel on the captured image captured by the imaging unit 10 by (x1, y1). The position / orientation calculation unit 46 generates an image having the same number of pixels on the captured image captured by the imaging unit 10 and the same arrangement (position) of the pixels as the first difference image. In addition, the position / orientation calculation unit 46 displays the pixel value p1 (x1, y1) of the pixel indicated by the coordinates (x1, y1) on the first image P1 and the pixel indicated by the coordinates (x1, y1) on the second image P2. The difference from the pixel value p2 (x1, y1) is calculated as the first difference. The pixel value of a certain pixel is, for example, the luminance value of the pixel. Note that the pixel value of a certain pixel may be another value associated with the pixel, such as the brightness value of the pixel or the saturation value of the pixel. The position / orientation calculation unit 46 performs such calculation of the first difference for each of all the pixels of the first image P1 and the second image P2.

  The position / orientation calculation unit 46 calculates an average value of pixel values corresponding to each of the first differences that are equal to or greater than a predetermined first threshold value pt among the plurality of calculated first differences. The first threshold value pt is about 5, for example. The first threshold value may be a value smaller than 5 or a value larger than 5 instead of 5. The position / orientation calculation unit 46 determines whether or not the calculated average value is greater than or equal to an intermediate value between the maximum value of the pixel value in the captured image captured by the imaging unit 10 and the minimum value of the pixel value. Below, the case where the said maximum value is 255 and the said minimum value is 0 is demonstrated as an example. When the average value is equal to or greater than the intermediate value, the position / orientation calculation unit 46 sets the predetermined pixel value px used in the process described below to 0. On the other hand, when the average value is less than the intermediate value, the position / orientation calculation unit 46 sets the predetermined pixel value px used in the processing described below as 255. Below, the case where the said average value is more than the said intermediate value is demonstrated as an example. Note that one or both of 0 and 255, which are the values of px described above, are merely examples, and may be other values.

  Further, the position / orientation calculation unit 46 determines whether or not the first difference is less than a predetermined first threshold value pt for each of the calculated first differences. Here, the pixel indicated by the coordinates (x1, y1) on the first image P1 and the pixel indicated by the coordinates (x1, y1) on the second image P2 will be described as an example again. When the position / orientation calculation unit 46 determines that the first difference based on these pixels is less than the first threshold value pt, the pixel value ps () of the pixel indicated by the coordinates (x1, y1) on the generated first difference image. x1, y1) is changed to the predetermined pixel value px described above (in this example, 0). On the other hand, when the position / orientation calculation unit 46 determines that the first difference is greater than or equal to the first threshold value pt, the position / orientation calculation unit 46 changes the pixel value ps (x1, y1) to the pixel value p2 (x1, y1). The position / orientation calculation unit 46 changes the pixel value on the first difference image based on the first difference for each of all the pixels of the first image P1, the second image P2, and the first difference image. Do it. That is, when a coordinate indicating an arbitrary pixel on the captured image captured by the imaging unit 10 is represented as (x, y), the pixel value ps (px) of the pixel indicated by the coordinate (x, y) on the first difference image. x, y) is expressed as the following formula (1).

  That is, the first difference image generated by the position / orientation calculation unit 46 uses the first image P1 as a second image of the background that is an image portion other than the image portion of the assembly target object O in the second image P2. It is an image removed from P2. FIG. 8 is an example of a first difference image generated by the position / orientation calculation unit 46. The first difference image P12 illustrated in FIG. 8 is an example of the first difference image. Here, the area in which dots are drawn in FIG. 8 represents an area filled in black. As shown in FIG. 8, the first difference image P12 includes the assembly target object O. In this example, the image portion other than the image portion of the assembly target object O in the first difference image P12 is blacked out because the predetermined pixel value px is 0. Thereby, the robot control apparatus 30 can extract only the image portion of the assembly target object O from the captured image captured by the imaging unit 10.

  The position / orientation calculation unit 46 uses the same method as the first difference image generation method, based on the third image (in this example, the second image P2) and the fourth image P4. Is generated. That is, in the description of the generation method, “first image P1” is “third image”, “second image P2” is “fourth image P4”, and “pixel value p1 (x1, y1)” is “pixel value p3”. (X1, y1) ”,“ pixel value p2 (x1, y1) ”is“ pixel value p4 (x1, y1) ”, and“ pixel value p1 (x, y) ”is“ pixel value p3 (x, y) ”. The description of the case where “pixel value p2 (x, y)” is read as “pixel value p4 (x, y)” is a method for generating the second difference image. For this reason, description of the method is omitted below.

  Similar to the first difference image, the second difference image generated by the position / orientation calculation unit 46 is an image of each of the operation target N, the end effector E, and the manipulator M held by the end effector E in the fourth image P4. This is an image obtained by removing the background, which is an image portion other than the portion, from the fourth image P4 using the third image. FIG. 9 is an example of the second difference image generated by the position / orientation calculation unit 46. Here, the area in which dots are drawn in FIG. 9 represents an area filled in black. The second difference image P34 illustrated in FIG. 9 is an example of a second difference image. As shown in FIG. 9, the second difference image P34 includes each of the operation target N, the end effector E, and the manipulator M (not shown in FIG. 9) held by the end effector E. In this second example, the image portions other than the image portions of the operation target N, the end effector E, and the manipulator M in the second difference image P34 are blacked out because the predetermined pixel value px is 0 in this example. Thereby, the robot control device 30 can extract only the image portions of the operation target N, the end effector E, and the manipulator M from the captured image captured by the imaging unit 10.

  After the process of step S190 is performed, the position / orientation calculation unit 46 calculates the position and orientation of the assembly target object O based on the first difference image P12 generated in step S190. Further, the position / orientation calculation unit 46 calculates the position and orientation of the operation target N based on the second difference image P34 generated in step S190 (step S200). Here, the process of step S200 will be described.

  First, processing for calculating the position and orientation of the assembly target object O will be described. The position / orientation calculation unit 46 reads the assembly target model stored in advance in the storage unit 32 from the storage unit 32. The assembly target model is an image (for example, CG (Computer Graphics) or the like) representing the outline of the assembly target O. In this example, the position / orientation calculation unit 46 performs edge detection on the first difference image P12 generated in step S190. The position / orientation calculation unit 46 calculates the position and orientation of the assembly target object O by pattern matching based on the detected edge and the read assembly target object model. The position and orientation are the position and orientation in the imaging unit coordinate system CC which is a coordinate system representing the position on the first difference image P12 (that is, the coordinate system representing the position on the captured image captured by the imaging unit 10). is there. The position / orientation calculation unit 46 may be configured to calculate the position and orientation of the assembly target object O in the imaging unit coordinate system CC by another method instead of edge detection and pattern matching.

  Further, the position / orientation calculation unit 46 reads out the operation object model stored in advance in the storage unit 32 from the storage unit 32. The operation target object model is an image (for example, CG or the like) representing the contour of the operation target object N. In this example, the position / orientation calculation unit 46 performs edge detection on the second difference image P34 generated in step S190. The position / orientation calculation unit 46 calculates the position and orientation of the operation target N by pattern matching based on the detected edge and the read operation target object model. The position and orientation are the position and orientation in the imaging unit coordinate system CC which is a coordinate system representing the position on the second difference image P34 (that is, the coordinate system representing the position on the captured image captured by the imaging unit 10). is there. The position / orientation calculation unit 46 may be configured to calculate the position and orientation of the operation target N in the imaging unit coordinate system CC by another method instead of edge detection and pattern matching.

  After the process of step S200 is performed, the robot control unit 50 reads out target position information and target posture information stored in advance in the storage unit 32. The target position information is information indicating a relative position between the position of the assembly target object O and the position of the operation target object N in a state where the operation target object N is assembled to the assembly target object O. The target posture information is information indicating a relative posture between the posture of the assembly target object O and the posture of the operation target object N in a state where the operation target object N is assembled to the assembly target object O. Based on the read target position information and the position of the assembly target object O calculated by the position / orientation calculation unit 46 in step S200, the robot control unit 50 has assembled the operation target object N into the assembly target object O. The position of the operation target N in the state in the imaging unit coordinate system CC is calculated as a target position that is a target for changing the position of the operation target N. Further, the robot control unit 50 assembles the operation target N to the assembly target object O based on the read target posture information and the posture of the assembly target object O calculated by the position / posture calculation unit 46 in step S200. The posture of the operation target N in the imaging unit coordinate system CC in the obtained state is calculated as a target posture that is a target for changing the posture of the operation target N. Then, the robot control unit 50 changes the position of the operation target N based on the vector representing the difference between the calculated target position and the position of the operation target N calculated by the position / orientation calculation unit 46 in step S200. Calculated as the amount of movement. Further, the robot control device 30 changes the posture of the operation target N based on the vector representing the difference between the calculated target posture and the posture of the operation target N calculated by the position / posture calculation unit 46 in step S200. The amount of movement is calculated (step S210). In the following, for convenience of explanation, unless there is a need to distinguish between the first movement amount and the second movement amount, they will be collectively referred to as a movement amount.

  Next, the robot control unit 50 converts the movement amount calculated in step S210 to the movement amount in the robot coordinate system RC based on the relative posture between the posture of the imaging unit coordinate system CC and the posture of the robot coordinate system RC. Conversion is performed (step S220). More specifically, the robot control unit 50 reads information indicating the posture stored in advance in the storage unit 32, and based on the read information, the movement amount calculated in step S210 in the robot coordinate system RC. Convert to travel. Next, the robot controller 50 operates the robot 20 based on the movement amount converted in step S220, assembles the operation target N to the assembly target O (step S230), and ends the process.

  As described above, the robot control device 30 overlaps at least a part of the first range with the first image obtained by capturing the first range that does not include the first target (in this example, the operation target N). The arm A is moved based on the position of the operation target N detected using the second image in which the second range including the first target is captured. Thereby, the robot control apparatus 30 can accurately perform the work relating to the first object (in this example, the predetermined work described above).

  Note that the robot control device 30 replaces the first difference image P12 with a configuration that calculates the position and orientation of the assembly target object O and the position and orientation of the operation target object N from the second difference image P34. The structure which calculates the position of the assembly | attachment target object O from the difference image P12 and the position of the operation target object N from the 2nd difference image P34 may be sufficient. In this case, the robot control device 30 calculates the first movement amount in step S210 based on the calculated position of the assembly target object O, the position of the operation target object N, and the target position information in step S210. Calculated as the amount of movement in CC. Then, the robot control device 30 converts the calculated movement amount into a movement amount in the robot coordinate system RC, operates the robot 20 based on the converted movement amount, and changes the operation target N to the assembly target O. Assemble. In this case, as described above, the posture of the operation target N and the posture of the assembly target O are fixed and do not change.

  In addition, the robot control device 30 replaces the first difference image P12 with the configuration for calculating the position and orientation of the assembly target object O and the second difference image P34 with the position and orientation of the operation target object N. The structure which calculates the position and attitude | position of the assembly target object O from the difference image P12 may be sufficient. In this case, the processes in steps S160 to S180 are omitted. In this case, the robot control device 30 stores information indicating the position and orientation of the operation target N in advance or acquires the information from another device.

  In addition, the robot control device 30 replaces the first difference image P12 with the configuration for calculating the position and orientation of the assembly target object O and the second difference image P34 with the position and orientation of the operation target object N. The structure which calculates the position of the assembly | attachment target object O from the difference image P12 may be sufficient. In this case, the processes in steps S160 to S180 are omitted. In this case, the robot control device 30 stores information indicating the position of the operation target N in advance, or acquires the information from another device. In this case, the posture of the operation target N and the posture of the assembly target O are fixed and do not change.

  Further, the robot control device 30 replaces the configuration in which the position and orientation of the assembly object O are calculated from the first difference image P12 and the position and orientation of the operation object N from the second difference image P34. The structure which calculates the position and attitude | position of the operation target N from the difference image P34 may be sufficient. In this case, the processes in steps S110 to S120 are omitted. In this case, the robot control device 30 stores information indicating the position and orientation of the assembly target object O in advance or acquires the information from another device.

  Further, the robot control device 30 replaces the configuration in which the position and orientation of the assembly object O are calculated from the first difference image P12 and the position and orientation of the operation object N from the second difference image P34. The structure which calculates the position of the operation target N from the difference image P34 may be sufficient. In this case, the processes in steps S110 to S120 are omitted. In this case, the robot control device 30 stores information indicating the position of the assembly target object O in advance, or acquires the information from another device. In this case, the posture of the operation target N and the posture of the assembly target O are fixed and do not change.

  Further, the assembly object O may be configured to be held by a robot other than the robot 20 instead of the configuration supported by the jig F. In this case, in step S <b> 130, the assembly target object O is moved by the robot to the inside of the range that can be imaged by the imaging unit 10.

<Modification 1 of Embodiment>
Hereinafter, a modification of the embodiment will be described with reference to FIG. In addition, in the modification 1 of embodiment, the same code | symbol is attached | subjected with respect to the component similar to embodiment, and description is abbreviate | omitted.

  In the first modification of the embodiment, when causing the robot control unit 50 to perform a predetermined operation, the robot control device 30 performs the process of the flowchart shown in FIG. 10 instead of the process of the flowchart shown in FIG. Specifically, the robot control device 30 in the embodiment operates the robot 20 by position control based on each of the first image P1, the second image P2, the third image, and the fourth image P4 captured by the imaging unit 10. The operation object N is assembled to the assembly object O. On the other hand, the robot control device 30 according to the first modification of the embodiment controls the robot 20 by visual servoing based on each of the first image P1, the second image P2, the third image, and the fourth image P4 captured by the imaging unit 10. The operation target N is assembled to the assembly target O by operating.

  FIG. 10 is a diagram illustrating another example of a process flow in which the robot control device 30 causes the robot 20 to perform a predetermined operation. Note that the processing in steps S110 to S210 illustrated in FIG. 10 is the same as the processing in steps S110 to S210 illustrated in FIG.

  After the process of step S210 is performed, the robot control unit 50 determines whether or not the amount of movement calculated by the position / orientation calculation unit 46 in step S210 is less than a predetermined threshold (step S240). In this example, the magnitude of the moving amount is the square root of the norm of the vector representing the moving amount. Instead of this, the amount of movement may be another amount based on the vector such as the norm. Here, the process of step S240 will be described.

  The robot control unit 50 calculates the square root of the norm of the vector representing the first movement amount among the movement amounts calculated by the position / orientation calculation unit 46 in step S210 as the first residual. In the following, as an example, the component of the vector includes an X-direction translation amount Δx for translating the operation target N in the X-axis direction in the imaging unit coordinate system CC, and an operation target N in the Y-axis direction in the imaging unit coordinate system CC. A Y direction translation amount Δy for translating the Z direction and a Z direction translation amount Δz for translating the operation target N in the Z axis direction in the imaging unit coordinate system CC will be described. In this case, the square root is expressed as the following formula (2).

  Further, the robot control unit 50 calculates the square root of the norm of the vector representing the second movement amount among the movement amounts calculated by the position / orientation calculation unit 46 in step S210 as the second residual. In the following, as an example, the components of the vector include the U-direction rotation amount Δu that rotates the operation target N in the U-axis direction in the imaging unit coordinate system CC, and the operation target N in the V-axis direction in the imaging unit coordinate system CC. A description will be given of a case where each has a V-direction rotation amount Δv for rotating the rotation direction and a W-direction translation amount Δw for rotating the operation target N in the W-axis direction in the imaging unit coordinate system CC. In this case, the square root is expressed as the following formula (3). The U axis is a coordinate axis that represents the amount of rotation about the X axis in the imaging unit coordinate system CC, and the V axis is a coordinate axis that represents the amount of rotation about the Y axis in the imaging unit coordinate system CC. The axis is a coordinate axis representing a rotation amount around the Z axis in the imaging unit coordinate system CC.

  Then, in step S240, the robot control unit 50 determines whether or not the calculated first residual is less than a predetermined second threshold th1 and the calculated second residual is less than a predetermined third threshold th2. By determining, it is determined whether or not the amount of movement calculated by the position / orientation calculating unit 46 in step S210 is less than a predetermined threshold. In step S240, the robot controller 50 determines whether the calculated first residual is less than a predetermined second threshold th1, or the calculated second residual is a predetermined third threshold th2. Even if it is the structure which determines whether the magnitude | size of the movement amount which the position / orientation calculation part 46 calculated in step S210 is less than a predetermined | prescribed threshold value by any one of determining whether it is less than. Good.

  If it is determined in step S240 that the amount of movement calculated by the position / orientation calculation unit 46 is less than a predetermined threshold (YES in step S240), the robot control unit 50 sets the operation target object as the assembly target object O. It is determined that N is assembled, and the process is terminated. On the other hand, when it is determined that the magnitude of the movement amount calculated by the position / orientation calculation unit 46 is greater than or equal to a predetermined threshold (step S240—NO), the robot control unit 50 determines the orientation of the imaging unit coordinate system CC and the robot coordinate system. Based on the posture relative to the posture of RC, the first movement amount and the second movement amount calculated in step S210 are converted into the first movement amount and the second movement amount in the robot coordinate system RC (step S250). More specifically, the robot control unit 50 reads information indicating the posture stored in advance in the storage unit 32, and based on the read information, the movement amount calculated in step S210 in the robot coordinate system RC. Convert to travel.

  Next, the robot control unit 50 has a predetermined gain related to the amount of movement for moving the operation target N (that is, the arm A) with respect to each of the first movement amount and the second movement amount converted in step S250, and is visual. The servo gain is multiplied (step S260). The gain may be any value as long as it is a value larger than 0 and smaller than 1. The gain multiplied by the first movement amount and the gain multiplied by the second movement amount may be the same value or different values.

  Next, the robot control unit 50 operates the robot 20 based on the first movement amount and the second movement amount multiplied by the gain in step S260 to bring the operation target N closer to the assembly target O (step S270). ). Then, the imaging control unit 40 proceeds to step S <b> 170, and causes the imaging unit 10 to capture the range that can be captured by the imaging unit 10 again.

  As described above, the robot control device 30 according to the first modification of the embodiment includes the position of the first object (the operation object N in this example) detected using the first image P1 and the second image P2. Then, the arm A is moved based on the gain relating to the movement amount for moving the arm A (that is, the movement amount for moving the operation target N). Thereby, the robot control device 30 performs an operation related to the first object (in this example, based on the position and gain of the first object detected using the first image P1 and the second image P2). It is possible to cause the robot 20 to perform the above-described predetermined operation) with high accuracy. That is, the robot control apparatus 30 can operate the robot 20 by visual servo and cause the robot 20 to perform the work with high accuracy. Further, even if a conversion error that cannot be ignored occurs in the conversion of the movement amount in step S250, the robot control device 30 causes the assembly object O to be attached by such visual servo (ie, feedback control). The operation target N can be assembled with high accuracy. As a result, when the operation object N is assembled to the assembly object O, it is guaranteed that the magnitude of the movement amount is less than the predetermined threshold value in step S240.

<Modification 2 of Embodiment>
Hereinafter, a second modification of the embodiment will be described with reference to FIG. In the second modification of the embodiment, the same reference numerals are given to the same components as those in the embodiment, and the description thereof is omitted.

  FIG. 11 is a diagram illustrating an example of a configuration of the robot system 2 according to the second modification of the embodiment. The robot system 2 includes a robot 60 that incorporates a robot control device 30.

  The robot 60 is a double-arm robot including a first arm, a second arm, a support base that supports the first arm and the second arm, and a robot control device 30 inside the support base. The robot 60 may be a multi-arm robot having three or more arms instead of the double-arm robot.

  The first arm includes a first end effector E1 and a first manipulator M1.

  In this example, the first end effector E1 is an end effector including a finger portion that can grip an object. The first end effector E1 may be an end effector capable of lifting an object by air suction, a magnetic force, a jig or the like, or another end effector, instead of the end effector provided with the finger portion. .

  The first end effector E1 is communicably connected to the robot controller 30 via a cable. Thus, the first end effector E1 performs an operation based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The first end effector E1 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The first manipulator M1 includes seven joints and a first imaging unit 61. Each of the seven joints includes an actuator (not shown). That is, the first arm including the first manipulator M1 is a 7-axis vertical articulated arm. The first arm performs a motion with seven degrees of freedom by a coordinated operation of the support, the first end effector E1, the first manipulator M1, and the actuators of the seven joints included in the first manipulator M1. The first arm may be configured to operate with a degree of freedom of 6 axes or less, or may be configured to operate with a degree of freedom of 8 axes or more.

  When the first arm operates with a degree of freedom of seven axes, the posture that the first arm can take is increased as compared with a case where the first arm operates with a degree of freedom of six axes or less. Thereby, for example, the first arm can be smoothly operated, and interference with an object existing around the first arm can be easily avoided. In addition, when the first arm operates with a degree of freedom of 7 axes, the control of the first arm is easy and requires a smaller amount of calculation than when the first arm operates with a degree of freedom of 8 axes or more.

  Each of the seven actuators (provided at the joints) included in the first manipulator M1 is connected to the robot controller 30 via a cable so as to be communicable. Accordingly, the actuator operates the first manipulator M1 based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. Further, some or all of the seven actuators included in the first manipulator M1 may be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). Good.

  The first imaging unit 61 is, for example, a camera including a CCD, a CMOS, or the like that is an imaging element that converts collected light into an electrical signal. In this example, the first imaging unit 61 is provided in a part of the first manipulator M1. Therefore, the first imaging unit 61 moves according to the movement of the first arm. In addition, the range in which the first imaging unit 61 can image changes according to the movement of the first arm. The first imaging unit 61 may be configured to capture a still image in the range, or may be configured to capture a moving image in the range.

  The first imaging unit 61 is communicably connected to the robot control device 30 via a cable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The first imaging unit 61 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The second arm includes a second end effector E2 and a second manipulator M2.

  In this example, the second end effector E2 is an end effector including a finger portion that can grip an object. The second end effector E2 may be an end effector capable of lifting an object by air suction, a magnetic force, a jig, or the like, instead of the end effector provided with the finger portion, or other end effector. .

  The second end effector E2 is communicably connected to the robot controller 30 via a cable. Accordingly, the second end effector E2 performs an operation based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The second end effector E2 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The second manipulator M <b> 2 includes seven joints and a second imaging unit 62. Each of the seven joints includes an actuator (not shown). That is, the second arm provided with the second manipulator M2 is a 7-axis vertical articulated arm. The second arm performs an operation with seven degrees of freedom by a coordinated operation of the support, the second end effector E2, the second manipulator M2, and the actuators of the seven joints included in the second manipulator M2. The second arm may be configured to operate with a degree of freedom of 6 axes or less, or may be configured to operate with a degree of freedom of 8 axes or more.

  When the second arm operates with a degree of freedom of 7 axes, the posture that the second arm can take is increased as compared with a case where the second arm operates with a degree of freedom of 6 axes or less. Thereby, for example, the operation of the second arm becomes smooth, and interference with an object existing around the second arm can be easily avoided. Further, when the second arm operates with a degree of freedom of 7 axes, the control of the second arm is easy and requires a smaller amount of calculation than when the second arm operates with a degree of freedom of 8 axes or more.

  Each of the seven actuators (provided at the joints) included in the second manipulator M2 is communicably connected to the robot controller 30 via a cable. Thus, the actuator operates the second manipulator M2 based on the control signal acquired from the robot control device 30. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. Further, some or all of the seven actuators included in the second manipulator M2 may be configured to be connected to the robot controller 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). Good.

  The second imaging unit 62 is, for example, a camera that includes a CCD, a CMOS, or the like that is an imaging element that converts collected light into an electrical signal. In this example, the second imaging unit 62 is provided in a part of the second manipulator M2. Therefore, the second imaging unit 62 moves according to the movement of the second arm. In addition, the range that can be imaged by the second imaging unit 62 changes according to the movement of the second arm. The second imaging unit 62 may be configured to capture a still image within the range, or may be configured to capture a moving image within the range.

  The second imaging unit 62 is communicably connected to the robot control device 30 via a cable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. Note that the second imaging unit 62 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

The robot 60 includes a third imaging unit 63 and a fourth imaging unit 64.
The third imaging unit 63 is, for example, a camera provided with a CCD, a CMOS, or the like that is an imaging element that converts collected light into an electrical signal. The third imaging unit 63 is provided in a region that includes the above-described work area and that can be imaged by the fourth imaging unit 64 together with the fourth imaging unit 64. The third imaging unit 63 is communicably connected to the robot control device 30 via a cable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The third imaging unit 63 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The fourth imaging unit 64 is, for example, a camera that includes a CCD, a CMOS, or the like that is an imaging element that converts collected light into an electrical signal. The fourth imaging unit 64 is provided in a region that includes the work area and can be imaged by the third imaging unit 63 in a portion that can be imaged in stereo with the third imaging unit 63. The fourth imaging unit 64 is connected to the robot control device 30 via a cable so as to be able to communicate. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. The fourth imaging unit 64 may be configured to be connected to the robot control device 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  In this example, each functional unit included in the robot 60 described above acquires a control signal from the robot control device 30 built in the robot 60. Then, each functional unit performs an operation based on the acquired control signal. The robot 60 may be configured to be controlled by the robot control device 30 installed outside, instead of the configuration incorporating the robot control device 30. In this case, the robot 60 and the robot control device 30 constitute a robot system. Further, the robot 60 may be configured not to include one or both of the first imaging unit 61 and the second imaging unit 62.

  Here, the operation target N in this example is gripped in advance by the first end effector E1 of the first arm, as shown in FIG. Accordingly, the robot control device 30 controls the first arm in the same manner as the arm A in the embodiment. Further, the robot control apparatus 30 in this example causes the third imaging unit 63 and the fourth imaging unit 64 to perform stereo imaging of the work area instead of the imaging unit 10 in the embodiment, and the third imaging unit 63 and the fourth imaging unit. The first image P1, the second image P2, the third image, and the fourth image P4 are obtained from each of 64. In this case, the robot control device 30 generates the first difference image and the second difference image based on the first image P1, the second image P2, the third image, and the fourth image P4 captured by the third imaging unit 63. In addition, a first difference image and a second difference image are generated based on the first image P1, the second image P2, the third image, and the fourth image P4 captured by the fourth imaging unit 64. Then, the robot control device 30 determines the position and orientation of the assembly target object O in the imaging unit coordinate system CC and the operation target object N in the imaging unit coordinate system CC based on the first difference image and the second difference image. Is calculated. The position and orientation of the assembly target object O are a three-dimensional position and a three-dimensional orientation calculated based on the two first difference images (that is, stereo images). Further, the position and orientation of the operation target N are a three-dimensional position and a three-dimensional orientation calculated based on the two second difference images (that is, stereo images).

  Further, the assembly object O in this example is gripped in advance by the second end effector E2 of the second arm, as shown in FIG. Accordingly, the robot control device 30 operates the second arm in step S130 described above to move the assembly object O to the inside of the range where the third imaging unit 63 and the fourth imaging unit 64 can perform stereo imaging. Thereby, the robot control apparatus 30 can omit the process of waiting until the user attaches the assembly target object O to the jig F in the process of step S130 shown in FIG. As a result, the robot control device 30 can cause the robot 20 to perform a predetermined operation without causing the user to perform the operation.

  As described above, the robot control apparatus 30 according to the second modification of the embodiment replaces the robot 20 that is a single-arm robot with the first object (this object) In one example, a first image in which a first range that does not include the operation target N) is captured, and a second range that includes at least a part of the first range and that includes the first target is captured. The arm (in this example, the first arm) is moved based on the position of the first object detected using the two images. Thereby, the robot control apparatus 30 can make the robot 60 perform the work on the first object with high accuracy.

  Note that the robot control device 30 replaces the configuration in which the first arm is moved based on the movement amount converted in step S220 or the movement amount multiplied by the gain in step S260, or the movement amount converted in step S220 or step S260. The second arm may be moved based on the amount of movement multiplied by the gain. That is, the robot control device 30 may be configured to operate the second arm and cause the robot 60 to perform the work of assembling the assembly object O to the operation object N as a predetermined work.

  Moreover, in the robot system 2, the structure by which the assembly target object O is attached to the jig | tool F may be sufficient like embodiment. In this case, the robot 60 holds the operation target N in advance by either one or both of the first arm and the second arm.

  Further, the robot system 2 may be configured to further include an imaging unit 10 as shown in FIG. FIG. 12 is a diagram illustrating another example of the configuration of the robot system 2 according to the second modification of the embodiment. In this case, the robot control device 30 captures the first image P1, the second image P2, the third image, and the fourth image P4 by the imaging unit 10, and the first image P1 and the second image P2 from the imaging unit 10. Each of the image P2, the third image, and the fourth image P4 is acquired.

  As described above, the robot (in this example, the robot 20 or the robot 60) is configured such that the first image (in this example) in which the first range not including the first object (in this example, the operation target N) is captured. , The first image P1) and a second image (in this example, the second image P2) in which the second range including the first object is captured and overlaps at least a part of the first range. The arm (in this example, the arm A or the first arm) is moved based on the position of the first object detected in this way. Thereby, the robot can perform the work (the predetermined work in this example) related to the first object with high accuracy.

  In addition, the robot generates a difference image based on the difference between the first image and the second image (in this example, the first difference image P12 and the second difference image P34), and is detected using the generated difference image. The arm is moved based on the position of the first object. Thereby, the robot can perform the work regarding the first object with high accuracy based on the position of the first object detected using the difference image.

  Further, the robot moves the arm based on the position of the first object detected using the first image and the second image, and a gain related to the amount of movement for moving the arm and the gain of the visual servo. Thereby, the robot can perform the work regarding the first object with high accuracy based on the position and gain of the first object detected using the first image and the second image.

  The robot moves the arm based on a difference between the position of the first object detected using the first image and the second image and a target position that is a target for changing the position of the first object. . Thereby, the robot can perform the work regarding the first object with high accuracy based on the difference between the position of the first object detected using the first image and the second image and the target position.

  The robot also detects the second object (in this example, assembly in this example) detected using the third image (in this example, second image P2) and the fourth image (in this example, fourth image P4). The first object and the second object are assembled based on the position of the object O) and the target position information. Thereby, the robot performs an operation of assembling the first object and the second object based on the position of the second object detected using the third image and the fourth image and the target position information. It can be performed with high accuracy.

  In the robot, the third image is the second image. Thereby, the robot can shorten the time required for the work based on the position of the first object.

  In addition, the robot moves the arm based on the posture of the first object detected using the first image and the second image. Thereby, the robot can perform the work regarding the first object with high accuracy based on the position and orientation of the first object.

  The robot moves the arm based on a difference between the posture of the first object detected using the first image and the second image and a target posture that is a target for changing the posture of the first object. . As a result, the robot can accurately perform the work on the first object based on the difference between the posture of the first object detected using the first image and the second image and the target posture. .

  Further, the robot moves the arm based on the position of the first object detected using the first image and the second image, target position information, and the position of the second object. As a result, the robot can work on the first object based on the position of the first object detected using the first image and the second image, the target position information, and the position of the second object. Can be performed with high accuracy.

  The robot also detects the position of the first object detected using the first image and the second image, the target position information, the third image, and the second object detected using the fourth image. The first object and the second object are assembled on the basis of the positions. Thereby, the robot detects the position of the first object detected using the first image and the second image, the target position information, the third image, and the second object detected using the fourth image. Based on the position of the object, the work related to the first object can be accurately performed.

  Further, the robot moves the arm based on the posture of the first object detected using the first image and the second image and the target posture information. Thereby, the robot can perform the work regarding the first object with high accuracy based on the posture of the first object detected using the first image and the second image and the target posture information.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and changes, substitutions, deletions, and the like are possible without departing from the gist of the present invention. May be.

  Further, a program for realizing the function of an arbitrary component in the above-described apparatus (for example, robot control apparatus 30) is recorded on a computer-readable recording medium, and the program is read into a computer system and executed. You may make it do. Here, the “computer system” includes hardware such as an OS (Operating System) and peripheral devices. The “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD (Compact Disk) -ROM, or a storage device such as a hard disk built in the computer system. . Furthermore, “computer-readable recording medium” means a volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

In addition, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the functions described above. Further, the program may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

DESCRIPTION OF SYMBOLS 1, 2 ... Robot system, 10 ... Imaging part, 20, 60 ... Robot, 30 ... Robot control apparatus, 31 ... CPU, 32 ... Memory | storage part, 33 ... Input reception part, 34 ... Communication part, 35 ... Display part, 36 DESCRIPTION OF SYMBOLS ... Control part, 40 ... Imaging control part, 42 ... Image acquisition part, 46 ... Position and orientation calculation part, 50 ... Robot control part, 61 ... 1st imaging part, 62 ... 2nd imaging part, 63 ... 3rd imaging part, 64: Fourth imaging unit

Claims (13)

With an arm,
A first image in which a first range that does not include the first object is captured, and a second image in which a second range that includes at least a part of the first range and includes the first object is captured Moving the arm based on the position of the first object detected using
robot. A difference image based on the difference between the first image and the second image is generated, and the arm is moved based on the position of the first object detected using the generated difference image.
The robot according to claim 1. Moving the arm based on the position of the first object detected using the first image and the second image and a gain relating to a moving amount for moving the arm and the gain of the visual servo;
The robot according to claim 1 or 2. The first object is gripped by the arm;
The arm is moved based on a difference between a position of the first object detected using the first image and the second image and a target position which is a target for changing the position of the first object. ,
The robot according to any one of claims 1 to 3. The first object is an object to be assembled to the second object by the arm,
A third image in which a third range not including the second object is captured, and a fourth image in which a fourth range including the second object is captured in a range that overlaps at least a part of the third range. Based on the position of the second object detected using an image and target position information indicating a relative position between the position and the target position, the first object and the second object are Assemble,
The robot according to claim 4. The third image is the second image;
The robot according to claim 5. Moving the arm based on a posture of the first object detected using the first image and the second image;
The robot according to any one of claims 1 to 6. The arm is moved based on a difference between a posture of the first object detected using the first image and the second image and a target posture that is a target for changing the posture of the first object. ,
The robot according to claim 7. The first object is an object to which a second object gripped by the arm is assembled,
A relative position between the position of the first object detected using the first image and the second image, and a target position that is a target for changing the position of the second object. Moving the arm based on the indicated target position information and the position of the second object;
The robot according to any one of claims 1 to 3. A third image in which a position of the first object detected using the first image and the second image, the target position information, and a third range not including the second object are captured; And the position of the second object detected using a fourth image in which a fourth range including the second object is captured and is a range that overlaps at least a part of the third range. Assembling the first object and the second object,
The robot according to claim 9. A relative posture between the posture of the first object detected using the first image and the second image and a target posture which is a target for changing the posture and the posture of the second object. Moving the arm based on the target posture information shown,
The robot according to claim 9 or 10. Controlling the robot according to any one of claims 1 to 11,
Robot control device. A robot according to any one of claims 1 to 11,
A robot controller for controlling the robot;
A robot system comprising:

Images