WO2024202042A1 - 演算装置、ロボット制御方法、処理方法、表示方法、コンピュータプログラム、記録媒体、経路生成方法及び搬送方法 - Google Patents
演算装置、ロボット制御方法、処理方法、表示方法、コンピュータプログラム、記録媒体、経路生成方法及び搬送方法 Download PDFInfo
- Publication number
- WO2024202042A1 WO2024202042A1 PCT/JP2023/013635 JP2023013635W WO2024202042A1 WO 2024202042 A1 WO2024202042 A1 WO 2024202042A1 JP 2023013635 W JP2023013635 W JP 2023013635W WO 2024202042 A1 WO2024202042 A1 WO 2024202042A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- model
- end effector
- computing device
- robot
- target object
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Program-controlled manipulators
- B25J9/16—Program controls
- B25J9/1656—Program controls characterised by programming, planning systems for manipulators
- B25J9/1664—Program controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
- B25J9/1666—Avoiding collision or forbidden zones
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Program-controlled manipulators
- B25J9/16—Program controls
- B25J9/1656—Program controls characterised by programming, planning systems for manipulators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Program-controlled manipulators
- B25J9/16—Program controls
- B25J9/1602—Program controls characterised by the control system, structure, architecture
- B25J9/161—Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Program-controlled manipulators
- B25J9/16—Program controls
- B25J9/1628—Program controls characterised by the control loop
- B25J9/163—Program controls characterised by the control loop learning, adaptive, model based, rule based expert control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Program-controlled manipulators
- B25J9/16—Program controls
- B25J9/1656—Program controls characterised by programming, planning systems for manipulators
- B25J9/1671—Program controls characterised by programming, planning systems for manipulators characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Program-controlled manipulators
- B25J9/16—Program controls
- B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
Definitions
- the present invention relates to the technical fields of, for example, robot-related computing devices, display methods, path generation methods, computer programs, recording media, robot control methods, and transport methods.
- Patent Document 1 An example of a robot that performs a specified process on an object is described in Patent Document 1. In the technical field of such robots, it is required to appropriately pre-configure the robot so that it performs the specified process on the object.
- a computing device includes a signal generating unit that generates a display signal for displaying, on a display device, a movement trajectory of a moving point that moves in association with the driving of the robot when the robot is driven without moving the robot, and an output unit that outputs the display signal generated by the signal generating unit to the display device.
- a calculation device that generates a movement path of a moving point that moves in association with the driving of a robot to which an end effector that processes a target object is attached, and that generates a second movement path of the moving point that moves in association with the driving of the robot to which a first end effector is attached as the end effector, based on a first movement path of the moving point that moves in association with the driving of the robot to which a second end effector is attached as the end effector, and a three-dimensional model of a second end effector different from the first end effector.
- a computing device that is used to set a processing target position of an object by an end effector that processes the object, and that includes a signal generating unit that generates a display signal for displaying on a display device at least one of the object model and the end effector model moving on a display screen including an object model that is a three-dimensional model of the object and an end effector model that is a three-dimensional model of the end effector in accordance with instruction information related to a user's instruction on the display screen to set the processing target position, and an output unit that outputs the display signal generated by the signal generating unit to the display device, and the signal generating unit generates the display signal for displaying on the display device at least one of the object model and the end effector model moving along at least both a first axis and a second axis intersecting the first axis based on the instruction information.
- a computing device used for calibrating an imaging device attached to a robot including a signal generating unit that generates a display signal for displaying, on a display device, information regarding the accuracy of the calibration of the imaging device, which is generated based on image data generated by imaging a reference member with the imaging device attached to the robot, and an output unit that outputs the display signal generated by the signal generating unit to the display device.
- a calculation device that generates a movement path of a moving point that moves in association with the driving of a robot to which an end effector and an imaging device are attached for processing a target object, and that generates a first partial path, which is a part of the movement path, based on a first end effector model that is an extension of at least a part of the three-dimensional model of the end effector, and a first peripheral object model that is a three-dimensional model of a first peripheral object located in the periphery of the robot, and generates a second partial path, which is at least another part of the movement path, based on a second end effector model in which the extension amount of the three-dimensional model of the end effector is different from that of the first end effector model, a second peripheral object model that is a three-dimensional model of a second peripheral object located in the periphery of the robot, and the imaging result obtained by the imaging device.
- a computing device that generates a movement path of a moving point that moves in association with the driving of a robot equipped with an end effector capable of holding a target object, the computing device generating at least a portion of the movement path based on an end effector model based on a three-dimensional model of the end effector, a target object model based on a three-dimensional model of the target object held by the end effector, and a peripheral object model that is a three-dimensional model of peripheral objects located around the robot.
- a computing device that controls a robot equipped with an end effector and an imaging device that processes a target object, and that generates a partial model, which is a three-dimensional model of at least a part of the target object, from an original model, which is a three-dimensional model of the target object, and calculates at least one of the position and orientation of the target object by performing a matching process based on the partial model and target object image data acquired by imaging the target object with the imaging device, and generates the control signal based on the calculation result of at least one of the position and orientation of the target object.
- a display method includes generating a display signal for displaying on a display device a movement trajectory of a moving point that moves as the robot is driven when the robot is driven without moving the robot, and outputting the generated display signal to the display device.
- a computer program is provided that causes a computer to execute the display method provided by the eighth aspect.
- a recording medium is provided on which the computer program provided by the ninth aspect is recorded.
- a path generation method for generating a movement path of a moving point that moves in association with the driving of a robot to which an end effector that processes a target object is attached, the path generation method including generating a second movement path of the moving point that moves in association with the driving of the robot to which a first end effector is attached as the end effector, based on a first movement path of the moving point that moves in association with the driving of the robot to which a second end effector is attached as the end effector, and a three-dimensional model of a second end effector different from the first end effector.
- a computer program is provided that causes a computer to execute the display method provided by the eleventh aspect.
- a recording medium is provided on which the computer program provided by the twelfth aspect is recorded.
- a display method used to set a processing target position of an object by an end effector that processes the object including: generating a display signal for displaying on a display device at least one of the object model and the end effector model moving on a display screen including an object model that is a three-dimensional model of the object and an end effector model that is a three-dimensional model of the end effector, based on instruction information regarding a user's instruction, in order to set the processing target position; and outputting the generated display signal to the display device, the generating step generating the display signal for displaying on the display device at least one of the object model and the end effector model moving along at least both a first axis and a second axis intersecting the first axis, based on the instruction information.
- a computer program is provided that causes a computer to execute the display method provided by the fourteenth aspect.
- a recording medium is provided on which the computer program provided in the fifteenth aspect is recorded.
- a display method used for calibrating an imaging device attached to a robot including generating a display signal for displaying information on the accuracy of the calibration of the imaging device, the information being generated based on image data generated by imaging a reference member with the imaging device attached to the robot, on a display device, and outputting the generated display signal to the display device.
- a computer program is provided that causes a computer to execute the display method provided by the seventeenth aspect.
- a recording medium is provided on which the computer program provided by the eighteenth aspect is recorded.
- a path generation method for generating a movement path of a moving point that moves in association with the driving of a robot equipped with an end effector and an imaging device that processes a target object
- the path generation method including: generating a first partial path that is a part of the movement path based on a first end effector model that is an extension of at least a part of the three-dimensional model of the end effector and a first peripheral object model that is a three-dimensional model of a first peripheral object located in the periphery of the robot; and generating a second partial path that is at least another part of the movement path based on a second end effector model in which the extension amount of the three-dimensional model of the end effector is different from that of the first end effector model, a second peripheral object model that is a three-dimensional model of a second peripheral object located in the periphery of the robot, and an imaging result obtained by the imaging device.
- a computer program is provided that causes a computer to execute the display method provided by the twenty-first aspect.
- a recording medium is provided on which the computer program provided in the twenty-first aspect is recorded.
- a robot control method for controlling a robot based on at least a part of the movement path generated by the calculation device provided in the sixth aspect.
- a transport method for transporting the target object held by the end effector based on at least a part of the movement path generated by the calculation device provided by the sixth aspect.
- a robot control method for controlling a robot equipped with an end effector and an imaging device for processing a target object
- the robot control method including: generating a partial model, which is a three-dimensional model of at least a part of the target object, from an original model, which is a three-dimensional model of the target object; calculating at least one of the position and orientation of the target object by performing a matching process based on the partial model and target object image data acquired by imaging the target object with the imaging device; and generating the control signal based on the calculation result of at least one of the position and orientation of the target object.
- a computer program is provided that causes a computer to execute the display method provided by the twenty-fifth aspect.
- a recording medium is provided on which the computer program provided in the twenty-sixth aspect is recorded.
- FIG. 1 is a block diagram showing the configuration of a robot system according to the present embodiment.
- FIG. 2 is a side view showing the appearance of the robot of this embodiment.
- Each of Figures 3(a) to 3(d) is a cross-sectional view showing an example of the operation of a robot for holding a workpiece accommodated in a storage box, which is an example of a mounting device, using an end effector.
- Each of Figures 4(a) to 4(d) shows an example of the operation of a robot releasing a workpiece held by an end effector so as to place the workpiece on a pallet, which is an example of a mounting device.
- FIG. 5 is a block diagram showing the configuration of the control device of this embodiment.
- FIG. 5 is a block diagram showing the configuration of the control device of this embodiment.
- FIG. 6 is a flowchart showing the flow of the robot control process.
- FIG. 7 is a perspective view showing an example of the calibration member.
- FIG. 8 is a flowchart showing the flow of accuracy information display processing for displaying calibration accuracy information on a display device.
- FIG. 9 shows a first calibration image data group and a second calibration image data group.
- FIG. 10 shows an example of how the calibration index values are displayed.
- FIG. 11 shows the processing position and orientation setting screen.
- FIG. 12 shows the processing position and orientation setting screen.
- FIG. 13 shows a moving end effector model.
- FIG. 14 shows a moving end effector model.
- FIG. 15 shows a moving end effector model.
- FIG. 16(a) and 16(b) shows the relationship between the end effector model and the object model.
- FIG. 16(a) and 16(b) shows the relationship between the end effector model and the object model.
- FIG. 17 shows the relationship between the end effector model and the object model.
- FIG. 18 shows the relationship between the end effector model and the object model.
- FIG. 19 is a flowchart showing the flow of the route generation process.
- FIG. 20 shows a route generation screen.
- FIG. 21 shows a route generation screen.
- FIG. 22 shows a route generation screen.
- FIG. 23 shows a route generation screen.
- Each of Figures 24(a) to 24(c) shows the movement path of the tool centre point.
- Figures 25(a) to 25(b) shows the movement path of the tool centre point.
- Each of Figures 26(a) and 26(b) is a cross-sectional view showing an extended end effector model.
- FIG. 27(a) and 27(b) are cross-sectional views showing the positional relationship between the extended end effector model and the surrounding object model, together with the positional relationship between the end effector and the surrounding object.
- Figures 28(a) to 28(c) is a cross-sectional view showing the positional relationship between the extended end effector model and the surrounding object model.
- Figures 29(a) to 29(d) is a cross-sectional view showing an extended end effector model.
- FIG. 30 shows an example of a display of information regarding the results of the matching process.
- Figure 31(a) is a cross-sectional view showing the movement path of the tool center point when the end effector is performing a holding process
- Figure 31(b) is a cross-sectional view showing an end effector model and a held object model used to generate the movement path of the tool center point when the end effector is holding an object
- Figure 31(c) is a cross-sectional view showing the end effector model used to generate the movement path of the tool center point when the end effector is not holding an object.
- Figure 32(a) is a cross-sectional view showing the movement path of the tool center point when the end effector is performing a release process
- Figure 32(b) is a cross-sectional view showing the end effector model and held object model used to generate the movement path of the tool center point when the end effector is holding an object
- Figure 32(c) is a cross-sectional view showing the end effector model used to generate the movement path of the tool center point when the end effector is not holding an object.
- FIG. 33(a) and FIG. 33(b) shows an example of a model generation screen.
- FIG. 34 shows a multiple part model.
- FIG. 35 is a flowchart showing the flow of the matching process performed in the fifth modified example of the fine movement operation.
- Fig. 1 is a block diagram showing the overall configuration of the robot system SYS.
- the robot system SYS includes a robot 1, an imaging device 2, a control device 3, and an end effector 4.
- the robot 1 is a device capable of performing a predetermined process on an object OBJ.
- An example of the robot 1 is shown in FIG. 2.
- FIG. 2 is a side view showing the external appearance of the robot 1.
- the robot 1 includes, for example, a base 11, a robot arm 12, and a robot control device 13.
- the base 11 is a component that serves as the foundation of the robot 1.
- the base 11 is placed on a support surface S, such as a floor surface.
- the base 11 may be fixed to the support surface S.
- the base 11 may be movable relative to the support surface S.
- the base 11 may be capable of self-propelling on the support surface S.
- the base 11 may be installed on an automatic guided vehicle (AGV).
- AGV automatic guided vehicle
- FIG. 2 shows an example in which the base 11 is fixed to the support surface S.
- the robot arm 12 is attached to the base 11.
- the robot arm 12 is a device in which multiple links 121 are connected via joints 122.
- An actuator is built into the joint 122.
- the link 121 may be rotatable around an axis defined by the joint 122 by the actuator built into the joint 122.
- At least one link 121 may be expandable and contractible along the direction in which the link 121 extends.
- the device including the device in which multiple links 121 are connected via joints 122 and the base 11 may be referred to as the robot arm 12.
- the end effector 4 is attached to the robot arm 12. That is, the end effector 4 is attached to the robot 1. In the example shown in FIG. 2, the end effector 4 is attached to the tip of the robot arm 12.
- the end effector 4 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the end effector 4. That is, the robot 1 moves the end effector 4.
- the end effector 4 is a device that performs a predetermined process (in other words, a predetermined operation) on the object OBJ.
- the end effector 4 may be called a processing device because it performs a predetermined process on the object OBJ.
- the end effector 4 may perform a holding process to hold (in other words, hold) an object OBJ as an example of the predetermined process.
- the end effector 4 may be considered to be performing a holding process on the object OBJ that the end effector 4 is to hold.
- An end effector 4 capable of performing a holding process may be referred to as a holding device.
- holding the object OBJ may include grasping the object OBJ.
- holding the object OBJ may include grasping the object OBJ using a hand gripper, which is an example of the end effector 4 and will be described later.
- Holding the object OBJ may include adsorbing the object OBJ.
- holding the object OBJ may include adsorbing (vacuum adsorption) the object OBJ using a vacuum gripper, which is an example of the end effector 4 and will be described later.
- holding the object OBJ may include adsorbing the object OBJ using a magnetic adsorption gripper, which is an example of the end effector 4 and will be described later.
- the end effector 4 may perform a release process to release (i.e., let go of) the object OBJ that it is holding.
- the end effector 4 may be considered to be performing a release process on the object OBJ that it is holding.
- An end effector 4 capable of performing a release process may be referred to as a release device.
- the end effector 4 may perform a release process to release (i.e., release) the first object OBJ that it is holding onto the second object OBJ so that the first object OBJ that it is holding is placed onto the second object OBJ.
- the end effector 4 may be considered to be performing a release process on the second object OBJ on which the end effector 4 is to place the first object OBJ.
- the end effector 4 may be considered to be performing a release process on the first object OBJ that the end effector 4 is to release.
- an end effector 4 capable of holding and releasing is a hand gripper.
- a hand gripper is an end effector 4 capable of holding an object OBJ by physically pinching the object OBJ using multiple (e.g., two, three, or four) finger members or claw members.
- Another example of an end effector 4 capable of holding and releasing is at least one of a vacuum gripper and a magnetically attracted gripper.
- a vacuum gripper is an end effector 4 capable of holding an object OBJ by vacuum-attracting the object OBJ.
- a magnetically attracted gripper is an end effector 4 capable of holding an object OBJ by magnetic force.
- Figure 2 shows an example in which the end effector 4 is a hand gripper.
- the robot 1 may use an end effector 4 capable of performing holding and releasing processes to perform a placement process for placing an object OBJ at a desired position.
- the robot 1 may hold a first object OBJ using the end effector 4, and then perform a placement process for placing the first object OBJ held by the end effector 4 at a desired position of a second object OBJ that is different from the first object OBJ.
- the robot 1 may use the end effector 4 capable of performing holding and releasing processes to perform a fitting process for fitting a first object OBJ into a second object OBJ different from the first object OBJ.
- the robot 1 may use the end effector 4 to hold the first object OBJ, and then perform a fitting process for fitting the first object OBJ held by the end effector 4 into a second object OBJ different from the first object OBJ.
- the above fitting process can be said to be a predetermined process performed by the end effector 4 on the first object OBJ, and also a predetermined process performed by the end effector 4 on the second object OBJ.
- the object OBJ on which the end effector 4 performs a predetermined process may include a workpiece W, as shown in FIG. 2.
- the workpiece W may include, for example, a part or member used to manufacture a desired product.
- the workpiece W may include, for example, a part or member to be processed to manufacture a desired product.
- the workpiece W may include, for example, a part or member to be transported to manufacture a desired product.
- the object OBJ on which the end effector 4 performs a predetermined process may include a placement device T on which the workpiece W is placed, as shown in FIG. 2.
- An example of the placement device T is a pallet.
- the placement device T may be disposed on the support surface S.
- the placement device T may be fixed to the support surface S.
- the placement device T may be movable relative to the support surface S.
- the placement device T may be self-propelled on the support surface S.
- FIG. 2 shows an example in which the placement device T is self-propelled on the support surface S.
- the placement device T may be referred to as an automatic guided vehicle (AGV).
- AGV automatic guided vehicle
- the placement device T When the placement device T is movable relative to the support surface S, the workpiece W placed on the placement device T also moves relative to the support surface S as the placement device T moves. For this reason, the movable placement device T may function as a moving device that moves the workpiece W.
- a belt conveyor may be used as the placement device T.
- the placement device T may be referred to as a placement member.
- the placement device T is not limited to a pallet, and may be a storage box capable of storing the workpiece W.
- the storage box may be, for example, a returnable box.
- the object OBJ does not have to include the placement device T.
- the placement device T is capable of self-propelling on the support surface S
- the self-propelled device capable of self-propelling on the support surface S and the pallet may be separate entities.
- the pallet may be installed on the self-propelled device.
- An apparatus including both the self-propelled device and the pallet may be referred to as the placement device T.
- the object OBJ does not have to include at least one of the self-propelled device and the pallet.
- the self-propelled device may be referred to as an automatic transport vehicle.
- the member installed on the self-propelled device is not limited to a pallet, and may be a storage box capable of storing the workpiece W.
- the placement device T may be capable of flying above the support surface S.
- the placement device T may be referred to as a flying device.
- the flying device capable of flying above the support surface S and the pallet may be separate.
- the pallet may be installed on the flying device.
- An apparatus including the flying device and the pallet may be referred to as the placement device T.
- the flying device may be referred to as an unmanned aerial vehicle.
- the workpiece W does not have to be placed on the placement device T.
- the workpiece W may be placed on the support surface S.
- the member installed on the flying device is not limited to a pallet, and may be a storage box capable of storing the workpiece W.
- the above-mentioned holding process may include a process of holding the workpiece W placed on the placement device T, which is stationary or moving.
- the above-mentioned holding process may include a process of holding the workpiece W placed on the support surface S.
- the above-mentioned release process may include a process of releasing the workpiece W held by the end effector 4 in order to place the workpiece W held by the end effector 4 at a desired position on the placement device T, which is stationary or moving.
- the above-mentioned release process may include a process of releasing the workpiece W held by the end effector 4 in order to place the workpiece W held by the end effector 4 at a desired position on the support surface S.
- the above-mentioned release process may include a process of releasing the first workpiece W held by the end effector 4 in order to fit the first workpiece W held by the end effector 4 into a second workpiece W placed on the placement device T, which is stationary or moving.
- the above-mentioned release process may include a process of releasing the first workpiece W held by the end effector 4 in order to fit the first workpiece W held by the end effector 4 into a second workpiece W placed on the support surface S.
- the robot control device 13 controls the operation of the robot 1.
- the robot controller 13 may control the operation of the robot arm 12.
- the robot controller 13 may control the operation of the robot arm 12 so that a desired link 121 rotates around an axis defined by a desired joint 122.
- the robot controller 13 may control the operation of the robot arm 12 so that an end effector 4 attached to the robot arm 12 is positioned at a desired position.
- the robot controller 13 may control the operation of the robot arm 12 so that an end effector 4 attached to the robot arm 12 moves to a desired position.
- the robot control device 13 may also control the operation of the end effector 4 attached to the robot 1.
- the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 holds the object OBJ at a desired timing.
- the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a holding process at a desired timing.
- the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 releases the object OBJ held by the end effector 4 at a desired timing.
- the robot control device 13 may control the operation of the end effector 4 so that the end effector 4 performs a release process at a desired timing.
- the robot control device 13 may control the timing at which the hand gripper opens and closes. If the end effector 4 is a vacuum gripper, the robot control device 13 may control the timing at which the vacuum device of the vacuum gripper is turned on and off.
- FIG. 2 shows an example in which the robot 1 is a robot arm 12 (i.e., a vertical articulated robot).
- the robot 1 may be a robot different from a vertical articulated robot.
- the robot 1 may be a SCARA robot (i.e., a horizontal articulated robot).
- the robot 1 may be a parallel link robot.
- the robot 1 may be a dual-arm robot having two robot arms 12.
- the robot 1 may be a Cartesian coordinate robot.
- the robot 1 may be a cylindrical coordinate robot.
- the robot 1 may be referred to as a movable device.
- the movable device may include at least one of an automatic guided vehicle and an unmanned aerial vehicle in addition to the robot 1.
- the robot 1 may be installed in at least one of an automatic guided vehicle and an unmanned aerial vehicle.
- the imaging device 2 captures an image of an object OBJ.
- the imaging device 2 is equipped with a camera 21, a camera 22, and a projection device 23.
- the imaging device 2 does not necessarily have to include either the camera 21 or 22.
- the imaging device 2 does not necessarily have to include the projection device 23.
- the camera 21 is capable of capturing an image of the object OBJ.
- the camera 21 captures an image of the object OBJ to generate image data IMG_2D.
- the image data IMG_2D generated by the camera 21 is output from the camera 21 to the control device 3.
- the camera 21 is a monocular camera.
- the camera 21 is capable of capturing an image of the object OBJ using a monocular camera (in other words, an image sensor).
- the camera 21 is not limited to a monocular camera.
- the camera 21 may be a stereo camera capable of capturing an image of the object OBJ using two monocular cameras, or may include three or more monocular cameras.
- the camera 21 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.
- camera 22 is capable of capturing an image of object OBJ.
- camera 22 is a stereo camera.
- camera 22 is a stereo camera capable of capturing an image of object OBJ using two monocular cameras (in other words, two image sensors).
- Camera 22 captures an image of object OBJ to generate image data IMG_3D.
- camera 22 since camera 22 is a stereo camera, camera 22 generates image data IMG_3D including two image data generated by the two monocular cameras.
- the image data IMG_3D generated by camera 22 is output from camera 22 to control device 3.
- camera 22 is not limited to a stereo camera.
- Camera 22 may be a monocular camera, or may include three or more monocular cameras.
- camera 22 may be at least one of a light field camera, a plenoptic camera, and a multispectral camera.
- the projection device 23 is a device capable of irradiating the projection light onto the object OBJ.
- the projection device 23 is a device capable of projecting a desired projection pattern onto the object OBJ by irradiating the projection light onto the object OBJ.
- the projection pattern may be referred to as the intensity distribution of the light projected onto the object OBJ. If the intensity distribution of the projected light changes, the projection pattern also changes.
- the desired projection pattern may include, for example, a random pattern.
- the random pattern may be a projection pattern having a different pattern for each unit irradiation area.
- the random pattern may include a random dot pattern.
- the desired projection pattern may include, for example, a one-dimensional or two-dimensional grid pattern.
- the desired projection pattern may include other projection patterns.
- the camera 22 captures the object OBJ onto which the projection pattern from the projection device 23 is projected.
- the object OBJ onto which the projection pattern is projected is reflected in the image indicated by the image data IMG_3D.
- the camera 21 does not need to capture the object OBJ onto which the projection pattern is projected.
- the camera 21 may capture an object OBJ onto which a projection pattern is not projected.
- the image represented by the image data IMG_2D may not include the object OBJ onto which the projection pattern is projected.
- the image represented by the image data IMG_2D may include the object OBJ onto which the projection pattern is not projected.
- the projection light for projecting a desired projection pattern onto the object OBJ may be called pattern light or structured light.
- the projection light may include pattern light or structured light.
- the projection light may be light of a uniform projection pattern (i.e., a uniform intensity distribution).
- the projection device 23 may be considered to illuminate the object OBJ with the projection light by irradiating the object OBJ with the projection light.
- the projection device 23 may be capable of functioning as an illumination device that illuminates the object OBJ.
- the projection light may be referred to as illumination light.
- the projection light (illumination light) used as an illumination device may be light with a uniform intensity distribution.
- the projection light does not have to be light that can project a desired projection pattern onto the object OBJ.
- the projection light i.e., illumination light
- the projection light may be any light that can illuminate the object OBJ.
- the imaging device 2 is attached to the robot arm 12, similar to the end effector 4. That is, the cameras 21 and 22 and the projection device 23 are attached to the robot arm 12.
- the cameras 21 and 22 and the projection device 23 may be attached to the tip of the robot arm 12, similar to the end effector 4.
- the cameras 21 and 22 and the projection device 23 can be moved by the movement of the robot arm 12. That is, the robot arm 12 moves the cameras 21 and 22 and the projection device 23.
- the imaging device 2 does not have to be attached to the robot arm 12.
- the imaging device 2 may be attached to any position where it can irradiate the object OBJ with projection light and capture an image of the object OBJ.
- At least one of the cameras 21, 22, and the projection device 23 may be attached to the robot arm 12, and at least one of the cameras 21, 22, and the projection device 23 may be attached to a location different from the robot arm 12.
- at least one of the cameras 21 and 22 may be attached to a structure such as a pillar that is located at a position where at least one of the cameras 21 and 22 can capture an image of the object OBJ.
- the projection device 23 may be attached to a structure such as a pillar that is located at a position where the projection device 23 can irradiate the projection light to the object OBJ.
- the cameras 21 and 22 may capture the object OBJ in synchronization with each other.
- the cameras 21 and 22 may capture the object OBJ at the same time. That is, the cameras 21 and 22 may capture the object OBJ such that the 2D imaging time at which the camera 21 captures the object OBJ is the same as the 3D imaging time at which the camera 22 captures the object OBJ.
- the cameras 21 and 22 may capture the object OBJ such that the 2D imaging time at which the camera 21 captures the object OBJ to generate image data IMG_2D is the same as the 3D imaging time at which the camera 22 captures the object OBJ to generate image data IMG_3D.
- the cameras 21 and 22 may capture an image of the object OBJ under the control of the control device 3.
- the time in other words, the timing
- the control device 3 may control the cameras 21 and 22 so that the cameras 21 and 22 capture the image of the object OBJ in synchronization with each other.
- the control device 3 may control the cameras 21 and 22 so that the cameras 21 and 22 capture the image of the object OBJ at the same time.
- the control device 3 may control the cameras 21 and 22 so that the 2D imaging time and the 3D imaging time are the same time.
- an error may occur in at least one calculation result of the position and orientation of the object OBJ described later (i.e., the accuracy of at least one of the calculated positions and orientations of the object OBJ may decrease).
- an error occurring in at least one calculation result of the position and orientation of the object OBJ may cause a control error of the robot arm 12.
- a control error of the robot arm 12 becomes a movement error of the end effector 4, and there are cases where the end effector 4 cannot properly perform a predetermined process on the object OBJ.
- the first allowable upper limit may be considered to be equivalent to the first allowable upper limit of the movement error of the end effector 4 by the robot arm 12.
- the 2D imaging time and the 3D imaging time may be considered to be substantially the same time.
- the synchronization error of the imaging process of the cameras 21 and 22 may be a synchronization control error of the imaging process of the cameras 21 and 22 by the control device 3.
- cameras 21 and 22 do not have to capture the object OBJ at the same time.
- cameras 21 and 22 may capture the object OBJ such that the 2D imaging time at which camera 21 captures the object OBJ and the 3D imaging time at which camera 22 captures the object OBJ are different times.
- the state in which "the 2D imaging time and the 3D imaging time are different times" may include a state in which "the 2D imaging time and the 3D imaging time cannot be considered to be substantially the same time because the time difference between the 2D imaging time and the 3D imaging time is greater than a first allowable upper limit value.”
- the cameras 21 and 22 may capture the object OBJ such that the 2D imaging time and the 3D imaging time are the same.
- the control device 3 may control the cameras 21 and 22 such that the 2D imaging time and the 3D imaging time are the same.
- the state in which the cameras 21 and 22 are displaced relative to the object OBJ may mean a state in which the relative positional relationship between the cameras 21 and 22 and the object OBJ has changed.
- the state in which the cameras 21 and 22 are displaced relative to the object OBJ may mean a state in which the cameras 21 and 22 are moving relative to the object OBJ.
- the cameras 21 and 22 when the cameras 21 and 22 capture the object OBJ during a period in which there is no relative displacement between each of the cameras 21 and 22 and the object OBJ, the cameras 21 and 22 do not need to capture the object OBJ so that the 2D imaging time and the 3D imaging time are the same.
- the control device 3 does not need to control the cameras 21 and 22 so that the 2D imaging time and the 3D imaging time are the same.
- the cameras 21 and 22 may capture the object OBJ so that the 2D imaging time and the 3D imaging time are different times.
- the control device 3 may control the cameras 21 and 22 so that the 2D imaging time and the 3D imaging time are different times.
- the state in which there is no relative displacement between each of the cameras 21 and 22 and the object OBJ may include a state in which the relative positional relationship between each of the cameras 21 and 22 and the object OBJ has not changed.
- the state in which the cameras 21 and 22 and the object OBJ are not displaced relative to each other may include a state in which the cameras 21 and 22 and the object OBJ are not moving relative to each other.
- the state in which the cameras 21 and 22 and the object OBJ are not displaced relative to each other may include a state in which the cameras 21 and 22 and the object OBJ are stationary.
- the cameras 21 and 22 may capture images of the object OBJ such that the 2D imaging time and the 3D imaging time are the same.
- the control device 3 may control the cameras 21 and 22 such that the 2D imaging time and the 3D imaging time are the same.
- the control device 3 performs robot control processing.
- the robot control processing may include processing for generating a robot control signal for controlling the robot 1.
- the control device 3 may perform end effector processing.
- the end effector control processing may include processing for generating an end effector control signal for controlling the end effector 4.
- the end effector processing may or may not be included in the robot control processing. In other words, the end effector control signal generated by the control device 3 may or may not be included in the robot control signal.
- the robot control processing may mean a process for generating at least one of a robot control signal and an end effector control signal.
- the robot control signal may mean at least one of a signal for controlling the robot 1 and a signal for controlling the end effector 4. Note that the robot control signal may simply be referred to as a control signal.
- the robot control signal may include a signal for controlling the robot 1 to move the robot 1. That is, the robot control signal may include a signal for controlling the robot arm 12 to move the robot arm 12. Specifically, the robot control signal may include a signal for controlling the robot 1 to move the moving reference point MBP, which moves with the movement of the robot 1 (i.e., the driving of the robot 1), to a desired position. That is, the robot control signal may include a signal for controlling the robot arm 12 to move the moving reference point MBP, which moves with the movement of the robot arm 12, to a desired position.
- the robot control signal may include a signal for controlling the robot 1 to move the moving reference point MBP, which moves in accordance with the movement of the end effector 4 in accordance with the movement of the robot 1 (i.e., the driving of the robot 1), to a desired position. That is, the robot control signal may include a signal for controlling the robot arm 12 to move the moving reference point MBP, which moves in accordance with the movement of the end effector 4 in accordance with the movement of the robot 1 (i.e., the driving of the robot 1), to a desired position.
- the moving reference point MBP may also be referred to simply as the moving point.
- the tool center point TCP of the end effector 4 may mean a point at which the end effector 4 contacts the object OBJ (e.g., the workpiece W).
- the tool center point TCP may mean a point in the space between the tips of multiple jaw members provided in the hand gripper.
- the tool center point TCP may mean a point in the space between multiple suction ports formed in the vacuum gripper and each capable of sucking gas.
- Such a tool center point TCP may be set by a user of the robot system SYS. Alternatively, the tool center point TCP may be set in advance for each end effector 4. The tool center point TCP may be at least a part of the robot arm 12. The tool center point TCP may be at least a part of the end effector 4.
- At least one point other than the tool center point TCP may be used as the movement reference point MBP.
- the movement reference point MBP since at least a part of the robot 1 moves with the movement of the robot 1, at least a part of the robot 1 may be used as the movement reference point MBP.
- At least one point of the robot 1 that is included in at least a part of the robot 1 may be used as the movement reference point MBP.
- an arbitrary point on the surface of the robot 1 may be used as the movement reference point MBP.
- an arbitrary point inside the robot 1 may be used as the movement reference point MBP.
- at least a part of the robot arm 12 moves with the movement of the robot 1, at least a part of the robot arm 12 may be used as the movement reference point MBP.
- At least one point of the robot arm 12 that is included in at least a part of the robot arm 12 may be used as the movement reference point MBP.
- an arbitrary point on the surface of the robot arm 12 may be used as the movement reference point MBP.
- any point inside the robot arm 12 may be used as the movement reference point MBP.
- at least a part of the end effector 4 attached to the robot 1 moves as the robot 1 moves, at least a part of the end effector 4 may be used as the movement reference point MBP.
- At least one point of the end effector 4 that is included in at least a part of the end effector 4 may be used as the movement reference point MBP.
- any point on the surface of the end effector 4 may be used as the movement reference point MBP.
- any point inside the end effector 4 may be used as the movement reference point MBP.
- the tool center point TCP may be considered to be a point determined based on the end effector 4. In this case, any point determined based on the end effector 4 may be used as the movement reference point MBP. As a further example, the tool center point TCP may be considered to be a point determined based on the robot 1 (particularly, the robot arm 12) to which the end effector 4 is attached. In this case, any point determined based on the robot 1 (particularly, the robot arm 12) may be used as the movement reference point MBP.
- the robot control signal may include, in addition to a signal for controlling the robot 1 to move the robot 1, a signal for controlling the end effector 4 to perform a predetermined process.
- the robot control signal may include a signal for controlling the end effector 4 to perform a holding process to hold the workpiece W.
- the robot control signal may include a signal for controlling the end effector 4 to perform a release process to release the held workpiece W.
- the robot control signal generated by the control device 3 is output to the robot control device 13 of the robot 1.
- the robot control device 13 controls the operation of the robot 1 based on the robot control signal generated by the control device 3. Therefore, the robot control signal may include a signal for controlling the operation of the robot 1.
- the robot control device 13 may control the robot arm 12 based on the robot control signal. For example, the robot control device 13 may control the operation of the robot arm 12 by controlling the operation of an actuator built into the joint 122 based on the robot control signal.
- the robot control device 13 may control the end effector 4 based on the robot control signal.
- the robot control device 13 may control the operation of the end effector 4 by controlling the operation of an actuator that moves a hand gripper that constitutes the end effector 4 based on the robot control signal.
- the robot control device 13 may control the operation of the end effector 4 by controlling the operation of a vacuum device of a vacuum gripper that constitutes the end effector 4 based on the robot control signal.
- the robot control signal may include a signal that can be used as is by the robot control device 13 to control the operation of the robot 1.
- the robot control signal may include a signal that can be used as is by the robot control device 13 as a robot drive signal used by the robot control device 13 to control the operation of the robot 1.
- the robot control device 13 may use the robot control signal as is to control the operation of the robot 1.
- the control device 3 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 as a robot control signal
- the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the actuator built into the joint 122 of the robot arm 12.
- the robot control signal may include a signal that can be used as is by the robot control device 13 to control the operation of the end effector 4.
- the robot control signal may include a signal that can be used as is by the robot control device 13 as an end effector drive signal used by the robot control device 13 to control the operation of the end effector 4.
- the control device 3 may generate a drive signal (end effector drive signal) for an actuator that moves a hand gripper that constitutes the end effector 4 as a robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the actuator of the end effector 4.
- control device 3 may generate a drive signal (end effector drive signal) for driving a vacuum device of a vacuum gripper that constitutes the end effector 4 as a robot control signal, and the robot control device 13 may use the robot control signal generated by the control device 3 as is to control the vacuum device of the end effector 4.
- drive signal end effector drive signal
- robot control device 13 may use the robot control signal generated by the control device 3 as is to control the vacuum device of the end effector 4.
- the robot control signal includes a signal that can be used directly by the robot control device 13 to control the operation of at least one of the robot 1 and the end effector 4, the robot 1 does not need to be equipped with the robot control device 13.
- the robot control signal may include a signal that can be used by the robot controller 13 to generate a robot drive signal for controlling the operation of the robot 1.
- the robot controller 13 may generate a robot drive signal for controlling the operation of the robot 1 based on the robot control signal, and control the operation of the robot 1 based on the generated robot drive signal.
- the robot controller 13 may generate a drive signal for an actuator built into the joint 122 of the robot arm 12 based on the robot control signal, and control the actuator built into the joint 122 of the robot arm 12 based on the generated signal.
- the robot control signal may include a signal that the robot control device 13 can use to generate an end effector drive signal for controlling the operation of the end effector 4.
- the robot control device 13 may generate an end effector drive signal for controlling the operation of the end effector 4 based on the end effector control signal, and control the operation of the end effector 4 based on the generated end effector drive signal.
- the robot control device 13 may generate a drive signal for the actuator of the end effector 4 based on the robot control signal, and control the actuator of the end effector 4 based on the generated signal.
- FIG. 3(a) to 3(d) shows an example of the operation of the robot 1 to hold (i.e., pick up) a workpiece W stored in a storage box CB, which is an example of a mounting device T, using the end effector 4. That is, each of Figures 3(a) to 3(d) shows an example of the operation of the robot 1 when the end effector 4 performs a holding process.
- Figures 3(a) to 3(d) show an example in which a single workpiece W is stored in the storage box CB. However, multiple workpieces W may be stored in the storage box CB.
- the storage box CB may also be referred to as a container.
- the multiple workpieces W may be stored in a regular pattern in the storage box CB.
- the multiple workpieces W may be located at regular positions in the storage box CB.
- the multiple workpieces W may be located at predetermined regular positions in the storage box CB.
- the multiple workpieces W having predetermined regular postures may be located in the storage box CB.
- the multiple workpieces W may be stored irregularly in the storage box CB.
- the multiple workpieces W may be located at irregular positions in the storage box CB.
- the storage box CB may store multiple workpieces W having irregular postures.
- the state in which multiple workpieces W are irregularly stored in the storage box CB may be considered equivalent to the state in which multiple workpieces W are randomly or carelessly stored in the storage box CB.
- the orientation of a first workpiece W among the multiple workpieces W may be different from the orientation of a second workpiece W among the multiple workpieces W that is different from the first workpiece W.
- the posture of the first workpiece W may be different from the posture of the second workpiece W.
- a third workpiece W different from the first workpiece W may overlap the first workpiece W
- a fourth workpiece W different from the second workpiece W may not overlap the second workpiece W.
- a third workpiece W may overlap the first workpiece W
- a fourth workpiece W may overlap the second workpiece W
- the size of the area where the first workpiece W and the third workpiece W overlap may be different from the size of the area where the second workpiece W and the fourth workpiece W overlap.
- each of Fig. 4(a) to Fig. 4(d) shows an example of the operation of the robot 1 releasing the workpiece W held by the end effector 4 so as to place the workpiece W held by the end effector 4 on a pallet PLT, which is an example of a placement device T.
- each of Fig. 4(a) to Fig. 4(d) shows an example of the operation of the robot 1 when the end effector 4 performs a release process.
- the robot 1 may move the tool center point TCP (i.e., the movement reference point MBP, the same below) to the first fine approach position PF1.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP to the first fine approach position PF1.
- the first fine approach position PF1 is a position that satisfies the condition that each of the cameras 21 and 22 can capture an image of the workpiece W when the tool center point TCP is located at the first fine approach position PF1.
- the first fine approach position PF1 is a position that satisfies the condition that the workpiece W (e.g., the workpiece W to be processed by the end effector 4) is located within the imaging angle of view of each of the cameras 21 and 22 when the tool center point TCP is located at the first fine approach position PF1.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1.
- the first waiting position PW1 may mean a position where the tool center point TCP should be located under a situation where the robot 1 waits before the end effector 4 performs a predetermined process (here, holding process) on the workpiece W.
- the first waiting position PW1 may mean a position where the end effector 4 should wait before the end effector 4 performs a predetermined process (here, holding process) on the workpiece W.
- the first waiting position PW1 may mean a position farther away from the workpiece W (container box CB) than the first fine approach position PF1.
- each of the cameras 21 and 22 may not be able to capture an image of the workpiece W.
- the operation of moving the tool center point TCP from the first standby position PW1 to the first fine approach position PF1 may be considered equivalent to the operation of moving the cameras 21 and 22, which are located at positions that satisfy the condition that each of the cameras 21 and 22 cannot capture an image of the workpiece W, to positions that satisfy the condition that each of the cameras 21 and 22 can capture an image of the workpiece W.
- the operation of moving the tool center point TCP from the first standby position PW1 to the first fine approach position PF1 may be considered equivalent to the operation of moving the tool center point TCP closer to the workpiece W until each of the cameras 21 and 22 can capture an image of the workpiece W.
- each of the cameras 21 and 22 may be able to capture an image of the workpiece W.
- each of the cameras 21 and 22 may or may not be able to capture an image of the workpiece W.
- each of the cameras 21 and 22 may or may not be able to capture an image of the workpiece W.
- At least one of the cameras 21 and 22 may actually capture an image of the workpiece W before the tool center point TCP reaches the first fine approach position PF1.
- at least one of the cameras 21 and 22 may actually capture an image of the workpiece W before the tool center point TCP reaches the first fine approach position PF1.
- at least one of the cameras 21 and 22 does not have to actually capture an image of the workpiece W before the tool center point TCP reaches the first fine approach position PF1.
- the movement path MRr of the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1 may be predetermined by the recipe file 321.
- the recipe file 321 includes information that specifies the content of a series of operations that the robot system SYS should perform using at least one of the robot 1 and the end effector 4.
- the recipe file 321 includes information regarding the content of the settings of the robot system SYS for performing a series of operations using at least one of the robot 1 and the end effector 4.
- control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1 based on the recipe file 321.
- control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP along the movement path MRr defined by the recipe file 321.
- the robot 1 may move the tool center point TCP from the first fine approach position PF1 to the holding target position PG.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the first fine approach position PF1 to the holding target position PG.
- the holding target position PG is a position that satisfies the condition that the end effector 4 holds the workpiece W when the tool center point TCP is located at the holding target position PG.
- the holding target position PG is a position where the end effector 4 holds the workpiece W.
- the holding target position PG may typically be a relative position determined based on the workpiece W to be held by the end effector 4.
- the holding target position PG is typically determined in advance by the recipe file 321.
- the holding target position PG may also be referred to as a position where the end effector 4 can hold the workpiece W when the tool center point TCP is located at the holding target position PG.
- each of the cameras 21 and 22 can capture an image of the object OBJ (particularly, the workpiece W).
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the first fine approach position PF1 to the holding target position PG based on at least one of the image data IMG_2D and IMG_3D output from the imaging device 2.
- the control device 3 may calculate at least one of the position and the orientation of the workpiece W in the global coordinate system of the robot system SYS based on at least one of the image data IMG_2D and IMG_3D.
- control device 3 may calculate at least one of the position of the workpiece W along the X-axis of the global coordinate system, the position of the workpiece W along the Y-axis of the global coordinate system, and the position of the workpiece W along the Z-axis of the global coordinate system as the position of the workpiece W in the global coordinate system.
- control device 3 may calculate at least one of the following orientations of the workpiece W in the global coordinate system: the orientation of the workpiece W about the X-axis of the global coordinate system, the orientation of the workpiece W about the Y-axis of the global coordinate system, and the orientation of the workpiece W about the Z-axis of the global coordinate system.
- control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the first fine approach position PF1 to the holding target position PG based on at least one of the calculated position and orientation of the workpiece W.
- the "position of an object” may mean at least one of the position of an object along a first axis, the position of an object along a second axis perpendicular to the first axis, and the position of an object along a third axis perpendicular to both the first and second axes.
- the "posture of an object” may mean at least one of the posture of an object around a first axis, the posture of an object around a second axis perpendicular to the first axis, and the posture of an object around a third axis perpendicular to both the first and second axes.
- the "posture of an object” may mean at least one of the amount of rotation of an object around a first axis, the amount of rotation of an object around a second axis perpendicular to the first axis, and the amount of rotation of an object around a third axis perpendicular to both the first and second axes.
- object posture may mean at least one of the following: the position of the object in a rotational direction about a first axis, the position of the object in a rotational direction about a second axis perpendicular to the first axis, and the position of the object in a rotational direction about a third axis perpendicular to both the first and second axes.
- the global coordinate system is a coordinate system that serves as the reference for the robot system SYS.
- the global coordinate system may be a coordinate system that serves as the reference for the robot 1.
- the robot coordinate system described below may be used as the global coordinate system, or a coordinate system different from the robot coordinate system may be used.
- the global coordinate system may also be referred to as a world coordinate system.
- control device 3 may bring the tool center point TCP closer to the workpiece W while sequentially calculating at least one of the position and orientation of the workpiece W based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the first fine approach position PF1 to the holding target position PG (i.e., while the end effector 4 is moving).
- control device 3 may bring the end effector 4 closer to the workpiece W while sequentially calculating at least one of the position and orientation of the workpiece W based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the first fine approach position PF1 to the holding target position PG (i.e., while the end effector 4 is moving).
- control device 3 may calculate at least one of the position and the posture of the workpiece W based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the first fine approach position PF1, and may move the tool center point TCP closer to the workpiece W based on the calculation result of at least one of the position and the posture of the workpiece W.
- control device 3 may calculate at least one of the position and the posture of the workpiece W based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the first fine approach position PF1, calculate a movement path MRf of the tool center point TCP from the first fine approach position PF1 to the holding target position PG based on the calculation result of at least one of the position and the posture of the workpiece W, and move the tool center point TCP along the calculated movement path MRf.
- control device 3 may calculate at least one of the position and posture of the workpiece W based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22 at the time when the tool center point TCP is located at the first waiting position PW1, calculate a movement path MRf of the tool center point TCP from the first waiting position PW1 to the holding target position PG based on the calculation result of at least one of the position and posture of the workpiece W, and move the tool center point TCP along the calculated movement path MRf.
- information on the movement path MRf may be stored in the recipe file 321.
- the operation of calculating at least one of the position and posture of the workpiece W based on at least one of the image data IMG_2D and IMG_3D, and moving the end effector 4 (typically, moving it relative to the object OBJ) based on the calculation result of at least one of the position and posture of the workpiece W may be referred to as a fine movement operation.
- the fine movement operation may be considered to be an operation that moves the tool center point TCP with higher accuracy (i.e., moves the end effector 4 with relatively higher accuracy) compared to the rough movement operation described later.
- the movement accuracy of the tool center point TCP by the fine movement operation may be the same as the movement accuracy of the tool center point TCP by the rough movement operation described later.
- the control device 3 when moving the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1, the control device 3 does not need to calculate at least one of the position and orientation of the workpiece W based on at least one of the image data IMG_2D and IMG_3D.
- the operation of moving the end effector 4 (typically, moving it relative to the object OBJ) without calculating at least one of the position and orientation of the workpiece W based on at least one of the image data IMG_2D and IMG_3D may be called a rough movement operation.
- control device 3 may perform a rough movement operation by moving the end effector 4 based on at least one of the image data IMG_2D and IMG_3D.
- the tool center point TCP is moving from the first waiting position PW1 to the first fine approach position PF1 (i.e., while the end effector 4 is moving)
- at least one of the positions and orientations of the workpiece W is sequentially calculated based on at least one of the image data IMG_2D and IMG_3D
- a movement path MRr of the tool center point TCP from the first waiting position PW1 (or from the current position) to the first fine approach position PF1 is calculated based on at least one of the positions and orientations of the workpiece W sequentially calculated, and the tool center point TCP is moved along the calculated movement path MRr.
- the control device 3 may calculate at least one of the position and posture of the workpiece W based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the first waiting position PW1, calculate a movement path MRr of the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1 based on the calculation result of at least one of the position and posture of the workpiece W, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may sequentially calculate at least one of the position and orientation of the storage box CB in which the workpiece W is stored (or a surrounding object located around the workpiece W, the same applies hereinafter in this paragraph) based on at least one of the image data IMG_2D and IMG_3D, and calculate a movement path MRr of the tool center point TCP from the first waiting position PW1 (or from the current position) to the first fine approach position PF1 based on at least one of the position and orientation of the storage box CB that is sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may calculate at least one of the position and orientation of the storage box CB (or a surrounding object located around the workpiece W, the same applies hereinafter in this paragraph) in which the workpiece W is stored based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the first waiting position PW1, calculate a movement path MRr of the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1 based on the calculation result of at least one of the position and orientation of the storage box CB, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may perform a rough movement operation by calculating at least one of the position and posture of the storage box CB in which the workpiece W is stored based on at least one of the image data IMG_2D and IMG_3D, calculating a movement path of the tool center point TCP from the first fine approach position PF1 to the holding target position PG based on the calculation result of at least one of the position and posture of the storage box CB, and moving the tool center point TCP along the calculated movement path.
- control device 3 may calculate at least one of the position and orientation of the workpiece W in a coordinate system different from the global coordinate system based on at least one of the image data IMG_2D and IMG_3D, and generate a robot control signal based on at least one of the calculated position and orientation of the workpiece W.
- the end effector 4 may hold the workpiece W.
- the control device 3 may generate a robot control signal for controlling the end effector 4 to hold the workpiece W.
- the robot 1 may move the end effector 4 so that the end effector 4 holding the workpiece W is retracted. Specifically, in order to retract the end effector 4, the robot 1 may move the tool center point TCP from the holding target position PG to the first fine approach position PF1, as shown in FIG. 3(c). In this case, the control device 3 may generate a robot control signal for moving the robot 1 so as to move (i.e., return) the tool center point TCP from the holding target position PG to the first fine approach position PF1.
- control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the holding target position PG to the first fine approach position PF1, based on at least one of the image data IMG_2D and IMG_3D output from the imaging device 2.
- the robot 1 may move the tool center point TCP along the same path as the path taken to move the tool center point TCP from the first fine approach position PF1 to the holding target position PG.
- the robot 1 does not have to move the tool center point TCP from the holding target position PG to the first fine approach position PF1.
- the robot 1 may move the tool center point TCP to a position that is moved a predetermined distance along the Z axis from the holding target position PG.
- the robot 1 may move the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2.
- the second waiting position PW2 may mean a position where the tool center point TCP should be located under a situation where the robot 1 waits after the end effector 4 performs a predetermined process (here, holding process) on the workpiece W.
- the second waiting position PW2 may mean a position where the end effector 4 should wait after the end effector 4 performs a predetermined process (here, holding process) on the workpiece W.
- the second waiting position PW2 may be the same position as the first waiting position PW1, or may be a position different from the second waiting position PW2.
- the movement path MRr of the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2 may be determined in advance by the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2 based on the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP along the movement path MRr determined by the recipe file 321.
- control device 3 when moving the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2, the control device 3 does not need to calculate the position of the object OBJ (for example, the storage box CB in the case of FIG. 3) based on at least one of the image data IMG_2D and IMG_3D.
- the movement path MRr of the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2 and the movement path MRr of the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1 are represented by the same reference symbol "MRr.” This is because the movement path MRr of the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2 and the movement path MRr of the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1 are both movement paths along which the tool center point TCP moves by a rough movement operation and are both determined in advance by the recipe file 321.
- the movement path MRr of the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2 and the movement path MRr of the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1 may be different from each other.
- the first waiting position PW1 and the second waiting position PW2 may be different from each other.
- the movement path MRr of the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2 and the movement path MRr of the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1 may be the same.
- the first waiting position PW1 and the second waiting position PW2 may be the same.
- control device 3 may sequentially calculate at least one of the positions and orientations of the workpiece W based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the first fine approach position PF1 to the second waiting position PW2 (i.e., while the end effector 4 is moving), and may calculate a movement path MRr of the tool center point TCP from the first fine approach position PF1 (or from the current position) to the second waiting position PW2 based on at least one of the positions and orientations of the workpiece W that are sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may sequentially calculate at least one of the position and posture of the storage box CB in which the workpiece W is stored (or a peripheral object located around the workpiece W; the same applies hereinafter in this paragraph) based on at least one of the image data IMG_2D and IMG_3D, calculate a movement path MRr of the tool center point TCP from the first fine approach position PF1 (or from the current position) to the second waiting position PW2 based on at least one of the position and posture of the storage box CB that is sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may calculate at least one of the position and posture of the workpiece W based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the first fine approach position PF1, calculate a movement path MRr of the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2 based on the calculation results of the at least one of the position and posture of the workpiece W, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may calculate at least one of the position and orientation of the storage box CB (or a surrounding object located around the workpiece W, the same applies hereinafter in this paragraph) in which the workpiece W is stored based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the first fine approach position PF1, calculate a movement path MRr of the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2 based on the calculation result of at least one of the position and orientation of the storage box CB, and move the tool center point TCP along the calculated movement path MRr.
- the end effector 4 may release the workpiece W that it is holding to another mounting device T (e.g., a pallet PLT or a storage box CB).
- the second waiting position PW2 may be referred to as a release target position where the end effector 4 releases the workpiece W.
- the end effector 4 may assemble the workpiece W that it is holding to another object (e.g., another workpiece W).
- the second waiting position PW2 may be referred to as an assembly target position where the end effector 4 assembles the workpiece W to another object.
- the robot 1 may move so that the end effector 4 holds the multiple workpieces W in sequence.
- the robot 1 may move so that the end effector 4 holds the multiple workpieces W that are stored in an orderly manner in the storage box CB in sequence.
- the robot 1 may move so that the end effector 4 holds the multiple workpieces W that are stored in an irregular manner in the storage box CB in sequence.
- the robot 1 may move the tool center point TCP (i.e., the movement reference point MBP, the same below) to the second fine approach position PF2.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP to the second fine approach position PF2.
- the second fine approach position PF2 is a position that satisfies the condition that each of the cameras 21 and 22 can capture an image of the placement device T (i.e., the above-mentioned pallet PLT) when the tool center point TCP is located at the second fine approach position PF2.
- the second fine approach position PF2 is a position that satisfies the condition that the pallet PLT is located within the imaging angle of view of each of the cameras 21 and 22 when the tool center point TCP is located at the second fine approach position PF2.
- the control device 3 may generate a robot control signal for moving the robot 1 to move the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2.
- the third waiting position PW3 may mean a position where the tool center point TCP should be located under a situation where the robot 1 is waiting before the end effector 4 performs a predetermined process (here, release process) on the workpiece W and the pallet PLT.
- the third waiting position PW3 may mean a position where the end effector 4 should be waiting before the end effector 4 performs a predetermined process (here, release process) on the workpiece W and the pallet PLT.
- each of the cameras 21 and 22 may not be able to capture an image of the pallet PLT.
- the operation of moving the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 may be considered equivalent to the operation of moving the cameras 21 and 22, which are located at positions that satisfy the condition that each of the cameras 21 and 22 cannot capture an image of the pallet PLT, to positions that satisfy the condition that each of the cameras 21 and 22 can capture an image of the pallet PLT.
- the operation of moving the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 may be considered equivalent to the operation of moving the tool center point TCP closer to the pallet PLT until each of the cameras 21 and 22 can capture an image of the pallet PLT.
- each of the cameras 21 and 22 may not be able to capture an image of the pallet PLT, or may be able to capture an image of the pallet PLT.
- each of the cameras 21 and 22 may not be able to capture an image of the pallet PLT, or may be able to capture an image of the pallet PLT.
- each of the cameras 21 and 22 may not be able to capture an image of the pallet PLT, or may be able to capture an image of the pallet PLT.
- At least one of the cameras 21 and 22 may actually capture an image of the pallet PLT before the tool center point TCP reaches the second fine approach position PF2.
- at least one of the cameras 21 and 22 may actually capture an image of the pallet PLT before the tool center point TCP reaches the second fine approach position PF2.
- at least one of the cameras 21 and 22 does not have to actually capture an image of the pallet PLT before the tool center point TCP reaches the second fine approach position PF2.
- the movement path MRr of the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 may be determined in advance by the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 based on the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP along the movement path MRr determined by the recipe file 321.
- the robot 1 may subsequently perform the release process shown in Figures 4(a) to 4(d).
- the third standby position PW3 shown in Figure 4(a) may be the same position as the first fine approach position PF1 or the second standby position PW2 shown in Figure 3(d).
- the third standby position PW3 shown in Figure 4(a) may be a position near the first fine approach position PF1 or the second standby position PW2 shown in Figure 3(d).
- the robot 1 may move the tool center point TCP from the second fine approach position PF2 to the release target position PR.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the second fine approach position PF2 to the release target position PR.
- the release target position PR is a position that satisfies the condition that the end effector 4 releases the workpiece W when the tool center point TCP is located at the release target position PR.
- the release target position PR is a position where the end effector 4 releases the workpiece W.
- the release target position PR may typically be a relative position determined based on the pallet PLT from which the end effector 4 should release the workpiece W.
- the release target position PR is typically determined in advance by the recipe file 321.
- the release target position PR may be referred to as a position where the end effector 4 can appropriately release (place) the workpiece W onto the pallet PLT (e.g., the workpiece W can be placed without damaging it by contact with the pallet PLT, etc.) when the tool center point TCP is located at the release target position PR.
- each of the cameras 21 and 22 can capture an image of the object OBJ (particularly, the pallet PLT).
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the second fine approach position PF2 to the release target position PR based on at least one of the image data IMG_2D and IMG_3D output from the imaging device 2.
- the control device 3 may calculate at least one of the position and orientation of the pallet PLT in the global coordinate system of the robot system SYS based on the image data IMG_2D and IMG_3D.
- control device 3 may calculate at least one of the position of the pallet PLT along the X-axis of the global coordinate system, the position of the pallet PLT along the Y-axis of the global coordinate system, and the position of the pallet PLT along the Z-axis of the global coordinate system as the position of the pallet PLT in the global coordinate system.
- control device 3 may calculate, as the orientation of the pallet PLT in the global coordinate system, at least one of the orientation of the pallet PLT about the X-axis of the global coordinate system, the orientation of the pallet PLT about the Y-axis of the global coordinate system, and the orientation of the pallet PLT about the Z-axis of the global coordinate system.
- control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the second fine approach position PF2 to the release target position PR based on at least one of the calculated position and orientation of the pallet PLT.
- control device 3 may bring the tool center point TCP closer to the pallet PLT while sequentially calculating at least one of the position and orientation of the pallet PLT based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the second fine approach position PF2 to the release target position PR (i.e., while the end effector 4 is moving).
- control device 3 may bring the end effector 4 closer to the pallet PLT while sequentially calculating at least one of the position and orientation of the pallet PLT based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the second fine approach position PF2 to the release target position PR (i.e., while the end effector 4 is moving).
- control device 3 may calculate at least one of the position and the orientation of the pallet PLT based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the second fine approach position PF2, and may move the tool center point TCP closer to the pallet PLT based on the calculation result of at least one of the position and the orientation of the pallet PLT.
- control device 3 may calculate at least one of the position and the orientation of the pallet PLT based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the second fine approach position PF2, and may calculate a movement path MRf of the tool center point TCP from the second fine approach position PF2 to the release target position PR based on the calculation result of at least one of the position and the orientation of the pallet PLT, and may move the tool center point TCP along the calculated movement path MRf.
- control device 3 may calculate at least one of the position and the attitude of the pallet PLT based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22 at the time when the tool center point TCP is located at the third waiting position PW3, calculate a movement path MRf of the tool center point TCP from the third waiting position PW3 to the release target position PR based on the calculation result of at least one of the position and the attitude of the pallet PLT, and move the tool center point TCP along the calculated movement path MRf.
- the operation of moving the tool center point TCP from the second fine approach position PF2 or the third waiting position PW3 to the release target position PR based on at least one of the image data IMG_2D and IMG_3D may be regarded as an example of a fine movement operation.
- information on the movement path MRf may be stored in the recipe file 321.
- the control device 3 when moving the tool center point TCP from the third standby position PW3 to the second fine approach position PF2, the control device 3 does not need to calculate at least one of the position and orientation of the pallet PLT based on at least one of the image data IMG_2D and IMG_3D. Therefore, the operation of moving the tool center point TCP from the third standby position PW3 to the second fine approach position PF2 without using at least one of the image data IMG_2D and IMG_3D may be considered to be an example of a rough movement operation.
- control device 3 may perform a rough movement operation by moving the end effector 4 based on at least one of the image data IMG_2D and IMG_3D.
- the control device 3 may sequentially calculate at least one of the positions and orientations of the pallet PLT based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the third waiting position PW3 to the second fine approach position PF2 (i.e., while the end effector 4 is moving), calculate a movement path MRr of the tool center point TCP from the third waiting position PW3 (or from the current position) to the second fine approach position PF2 based on at least one of the positions and orientations of the pallet PLT sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- control device 3 may perform a rough movement operation by calculating at least one of the position and orientation of the pallet PLT based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the third waiting position PW3, calculating a movement path MRr of the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 based on the calculation result of at least one of the position and orientation of the pallet PLT, and moving the tool center point TCP along the calculated movement path MRr.
- the control device 3 may sequentially calculate at least one of the positions and orientations of surrounding objects located around the pallet PLT based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the third waiting position PW3 to the second fine approach position PF2 (i.e., while the end effector 4 is moving), calculate a movement path MRr of the tool center point TCP from the third waiting position PW3 (or from the current position) to the second fine approach position PF2 based on at least one of the positions and orientations of the surrounding objects sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may calculate at least one of the positions and orientations of surrounding objects located around the pallet PLT based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the third waiting position PW3, calculate a movement path MRr of the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 based on the calculation result of at least one of the positions and orientations of the surrounding objects, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may perform a rough movement operation by calculating at least one of the positions and orientations of surrounding objects located around the pallet PLT based on at least one of the image data IMG_2D and IMG_3D, calculating a movement path of the tool center point TCP from the second fine approach position PF2 to the release target position PR based on the calculation result of at least one of the positions and orientations of the surrounding objects, and moving the tool center point TCP along the calculated movement path.
- control device 3 may calculate at least one of the position and orientation of the pallet PLT in a coordinate system different from the global coordinate system based on at least one of the image data IMG_2D and IMG_3D, and generate a robot control signal based on at least one of the calculated position and orientation of the pallet PLT.
- the end effector 4 may release the workpiece W that it is holding.
- the control device 3 may generate a robot control signal for controlling the end effector 4 to release the workpiece W that it is holding.
- the end effector 4 places the workpiece W that it is holding on the pallet PLT.
- the robot 1 may move the end effector 4 so that the end effector 4 that has released the workpiece W retreats. Specifically, in order to retreat the end effector 4, as shown in FIG. 4(c), the robot 1 may move the tool center point TCP from the release target position PR to the second fine approach position PF2. In this case, the control device 3 may generate a robot control signal for moving the robot 1 so as to move (i.e., return) the tool center point TCP from the release target position PR to the second fine approach position PF2.
- control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the release target position PR to the second fine approach position PF2, based on at least one of the image data IMG_2D and IMG_3D output from the imaging device 2.
- the robot 1 may move the tool center point TCP along the same path as the path taken to move the tool center point TCP from the second fine approach position PF2 to the release target position PR.
- the robot 1 does not have to move the tool center point TCP from the release target position PR to the second fine approach position PF2.
- the robot 1 may move the tool center point TCP to a position that is a predetermined distance along the Z axis from the release target position PR.
- the robot 1 may move the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4.
- the fourth waiting position PW4 may mean a position where the tool center point TCP should be located under a situation where the robot 1 waits after the end effector 4 performs a predetermined process (here, release process) on the workpiece W.
- the fourth waiting position PW4 may mean a position where the end effector 4 should wait after the end effector 4 performs a predetermined process (here, release process) on the workpiece W.
- the fourth waiting position PW4 may be the same position as the third waiting position PW3, or may be a position different from the third waiting position PW3.
- the movement path MRr of the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4 may be predetermined by the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4 based on the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP along the movement path MRr determined by the recipe file 321.
- control device 3 when moving the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4, the control device 3 does not need to calculate the position of the object OBJ based on at least one of the image data IMG_2D and IMG_3D.
- the movement path MRr of the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4 and the movement path MRr of the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 are indicated by the same reference symbol "MRr.” This is because the movement path MRr of the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4 and the movement path MRr of the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 are common in that they are movement paths along which the tool center point TCP moves by a rough movement operation and are determined in advance by the recipe file 321.
- the movement path MRr of the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4 and the movement path MRr of the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 may be different from each other.
- the third waiting position PW3 and the fourth waiting position PW4 may be different from each other.
- the movement path MRr of the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4 and the movement path MRr of the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 may be the same.
- the third waiting position PW3 and the fourth waiting position PW4 may be the same.
- the control device 3 may sequentially calculate at least one of the positions and orientations of the pallet PLT based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the second fine approach position PF2 to the fourth waiting position PW4 (i.e., while the end effector 4 is moving), and may calculate a movement path MRr of the tool center point TCP from the second fine approach position PF2 (or from the current position) to the fourth waiting position PW4 based on at least one of the positions and orientations of the pallet PLT sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may sequentially calculate at least one of the positions and orientations of surrounding objects located around the pallet PLT based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the second fine approach position PF2 to the fourth waiting position PW4 (i.e., while the end effector 4 is moving), calculate a movement path MRr of the tool center point TCP from the second fine approach position PF2 (or from the current position) to the fourth waiting position PW4 based on at least one of the positions and orientations of the surrounding objects sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may calculate at least one of the position and attitude of the pallet PLT based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the second fine approach position PF2, calculate a movement path MRr of the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4 based on the calculation results of at least one of the position and attitude of the pallet PLT, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may calculate at least one of the positions and orientations of surrounding objects located around the pallet PLT based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the second fine approach position PF2, calculate a movement path MRr of the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4 based on the calculation result of at least one of the positions and orientations of the surrounding objects, and move the tool center point TCP along the calculated movement path MRr.
- the robot 1 may move so that the end effector 4 releases multiple workpieces W onto the pallet PLT in sequence. That is, the robot 1 may move so that the end effector 4 places multiple workpieces W on the pallet PLT in sequence. For example, the robot 1 may move so that the end effector 4 places multiple workpieces W on the pallet PLT in a regular order (i.e., places multiple workpieces W in an aligned order). For example, the robot 1 may move so that the end effector 4 places multiple workpieces W on the pallet PLT in an irregular order (i.e., places multiple workpieces W in random positions and/or with random postures).
- the robot 1 may move so that the end effector 4 holds and then releases multiple workpieces W in sequence. That is, the robot 1 may alternate between the holding process shown in FIG. 3(a) to FIG. 3(d) and the release process shown in FIG. 4(a) to FIG. 4(d). For example, the robot 1 may perform a first holding process to hold a first workpiece W, then a first release process to release the first workpiece W to the pallet PLT, then a holding process to hold a second workpiece W different from the first workpiece W, and then a release process to release the second workpiece W to the pallet PLT.
- the second waiting position PW2 in the first holding process may be the same as the third waiting position PW3 or the second fine approach position PF2 in the first release process.
- the fourth waiting position PW4 in the first release process may be the same as the first waiting position PW1 or the first fine approach position PF1 in the second holding process.
- the second waiting position PW2 in the second holding process may be the same as the third waiting position PW3 or the second fine approach position PF2 in the second release process.
- the robot 1 may operate in the same manner as the operation modes shown in Figures 3(a) to 3(d) and 4(a) to 4(d). For example, the robot 1 may move the tool center point TCP from the pre-wait position PW_pre to the fine approach position PF.
- the fine approach position PF is a position that satisfies the condition that each of the cameras 21 and 22 can capture an image of the object OBJ when the tool center point TCP is located at the fine approach position PF.
- the fine approach position PF is a position that satisfies the condition that the object OBJ (e.g., the workpiece W to be processed by the end effector 4) is located within the imaging angle of view of each of the cameras 21 and 22 when the tool center point TCP is located at the fine approach position PF.
- the pre-wait position PW_pre may mean the position where the tool center point TCP should be located in a situation where the robot 1 is waiting before the end effector 4 performs a predetermined process on the object OBJ.
- the pre-wait position PW_pre may mean the position where the end effector 4 should wait before the end effector 4 performs a predetermined process on the object OBJ.
- Each of the first fine approach position PF1 and the second fine approach position PF2 described above is a specific example of the fine approach position PF.
- Each of the first wait position PW1 and the third wait position PW3 described above is a specific example of the pre-wait position PW_pre.
- each of the cameras 21 and 22 may not be able to capture an image of the object OBJ.
- the operation of moving the tool center point TCP from the pre-wait position PW_pre to the fine approach position PF may be considered equivalent to an operation of moving the cameras 21 and 22, which are located at positions that satisfy the condition that each of the cameras 21 and 22 cannot capture an image of the object OBJ, to a position that satisfies the condition that each of the cameras 21 and 22 can capture an image of the object OBJ.
- the operation of moving the tool center point TCP from the pre-wait position PW_pre to the fine approach position PF may be considered equivalent to an operation of moving the tool center point TCP closer to the object OBJ until each of the cameras 21 and 22 is able to capture an image of the object OBJ.
- each of the cameras 21 and 22 may not be able to capture an image of the object OBJ, or may be able to capture an image of the object OBJ.
- each of the cameras 21 and 22 may not be able to capture an image of the object OBJ, or may be able to capture an image of the object OBJ.
- each of the cameras 21 and 22 may not be able to capture an image of the object OBJ, or may be able to capture an image of the object OBJ.
- At least one of the cameras 21 and 22 may actually capture an image of the object OBJ before the tool center point TCP reaches the fine approach position PF.
- at least one of the cameras 21 and 22 may actually capture an image of the object OBJ before the tool center point TCP reaches the fine approach position PF.
- at least one of the cameras 21 and 22 does not have to actually capture an image of the object OBJ before the tool center point TCP reaches the fine approach position PF.
- the movement path MRr of the tool center point TCP from the pre-wait position PW_pre to the fine approach position PF may be predetermined by the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the pre-wait position PW_pre to the fine approach position PF based on the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP along the movement path MRr determined by the recipe file 321.
- the robot 1 may move the tool center point TCP from the fine approach position PF2 to a processing target position PTP, which will be described later.
- the processing target position PTP may mean a position where the end effector 4 performs a predetermined processing on the object OBJ. More specifically, the processing target position PTP may mean the position of the tool center point TCP (i.e., the position of the movement reference point MBP) when the end effector 4 performs a predetermined processing on the object OBJ.
- the processing target position PTP may typically be a relative position determined based on the object OBJ on which the end effector 4 performs a predetermined processing.
- the processing target position PTP is typically determined in advance by the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the fine approach position PF to the processing target position PTP.
- Each of the above-mentioned holding target position PG and release target position PR is a specific example of the processing target position PTP.
- the processing target position PTP may be referred to as the position at which the end effector 4 can process an object OBJ (e.g., a workpiece W to be processed by the end effector 4) when the tool center point TCP is located at the processing target position PTP.
- object OBJ e.g., a workpiece W to be processed by the end effector 4
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the fine approach position PF to the processing target position PTP based on at least one of the image data IMG_2D and IMG_3D output from the imaging device 2.
- the control device 3 may calculate at least one of the position and orientation of the object OBJ in the global coordinate system of the robot system SYS based on at least one of the image data IMG_2D and IMG_3D.
- control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the fine approach position PF to the processing target position PTP based on at least one of the calculated position and orientation of the object OBJ.
- control device 3 may bring the tool center point TCP closer to the object OBJ while sequentially calculating at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the fine approach position PF to the processing target position PTP (i.e., while the end effector 4 is moving).
- control device 3 may bring the end effector 4 closer to the object OBJ while sequentially calculating at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the fine approach position PF to the processing target position PTP (i.e., while the end effector 4 is moving).
- control device 3 may calculate at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the fine approach position PF, and may move the tool center point TCP closer to the workpiece W based on the calculation result of at least one of the position and orientation of the object OBJ.
- control device 3 may calculate at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the fine approach position PF, calculate a movement path MRf of the tool center point TCP from the first fine approach position PF1 to the holding target position PG based on the calculation result of at least one of the position and orientation of the object OBJ, and move the tool center point TCP along the calculated movement path MRf.
- control device 3 may calculate at least one of the position and the orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22 at the time when the tool center point TCP is located at the pre-wait position PW_pre, calculate a movement path MRf of the tool center point TCP from the pre-wait position PW_pre to the processing target position PTP based on the calculation result of at least one of the position and the orientation of the object OBJ, and move the tool center point TCP along the calculated movement path MRf.
- the operation of moving the tool center point TCP from the fine approach position PF or the pre-wait position PW_pre to the processing target position PTP based on at least one of the image data IMG_2D and IMG_3D may be regarded as an example of a fine movement operation.
- information on the movement path MRf may be stored in the recipe file 321.
- the control device 3 when moving the tool center point TCP from the pre-wait position PW_pre to the fine approach position PF, the control device 3 does not need to calculate at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D. Therefore, the operation of moving the tool center point TCP from the pre-wait position PW_pre to the fine approach position PF without using at least one of the image data IMG_2D and IMG_3D may be considered to be an example of a rough movement operation.
- control device 3 may perform a rough movement operation by moving the end effector 4 based on at least one of the image data IMG_2D and IMG_3D.
- the control device 3 may sequentially calculate at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the pre-waiting position PW_pre to the fine approach position PF (i.e., while the end effector 4 is moving), calculate a movement path MRr of the tool center point TCP from the pre-waiting position PW_pre (or from the current position) to the fine approach position PF based on at least one of the position and orientation of the object OBJ sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may calculate at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the pre-wait position PW_pre, calculate a movement path MRr of the tool center point TCP from the pre-wait position PW_pre to the fine approach position PF based on the calculation result of at least one of the position and orientation of the object OBJ, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may sequentially calculate at least one of the positions and orientations of surrounding objects located around the object OBJ based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the pre-waiting position PW_pre to the fine approach position PF (i.e., while the end effector 4 is moving), calculate a movement path MRr of the tool center point TCP from the pre-waiting position PW_pre (or from the current position) to the fine approach position PF based on at least one of the positions and orientations of the surrounding objects sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may calculate at least one of the positions and orientations of surrounding objects located around the object OBJ based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the advance waiting position PW_pre, calculate a movement path MRr of the tool center point TCP from the advance waiting position PW_pre to the fine approach position PF based on the calculation result of at least one of the positions and orientations of the surrounding objects, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may perform a rough movement operation by calculating at least one of the positions and orientations of surrounding objects located around the object OBJ based on at least one of the image data IMG_2D and IMG_3D, calculating a movement path of the tool center point TCP from the fine approach position PF to the processing target position PTP based on the calculation result of at least one of the positions and orientations of the surrounding objects, and moving the tool center point TCP along the calculated movement path.
- control device 3 may calculate at least one of the position and orientation of the object OBJ in a coordinate system different from the global coordinate system of the robot system SYS based on at least one of the image data IMG_2D and IMG_3D, and generate a robot control signal based on at least one of the calculated position and orientation of the object OBJ.
- the end effector 4 may perform a predetermined process on the object OBJ.
- the control device 3 may generate a robot control signal for controlling the end effector 4 to perform the predetermined process on the object OBJ.
- the robot 1 may move the end effector 4 so that the end effector 4 is retracted. Specifically, to retract the end effector 4, the robot 1 may move the tool center point TCP from the processing target position PTP to the fine approach position PF.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move (i.e., return) the tool center point TCP from the processing target position PTP to the fine approach position PF.
- control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the processing target position PTP to the fine approach position PF, based on at least one of the image data IMG_2D and IMG_3D output from the imaging device 2.
- the robot 1 may move the tool center point TCP along the same path as the path taken to move the tool center point TCP from the fine approach position PF to the processing target position PTP.
- the robot 1 may not move the tool center point TCP from the processing target position PTP to the fine approach position PF.
- the robot 1 may move the tool center point TCP to a position that is moved a predetermined distance along the Z axis from the processing target position PTP.
- the robot 1 may move the tool center point TCP from the fine approach position PF to the post-waiting position PW_post.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the fine approach position PF to the post-waiting position PW_post.
- the post-waiting position PW_post may mean the position where the tool center point TCP should be located under the circumstances where the robot 1 waits after the end effector 4 has performed a predetermined process on the object OBJ.
- the post-waiting position PW_post may mean the position where the end effector 4 should wait after the end effector 4 has performed a predetermined process on the object OBJ.
- the post-waiting position PW_post may be the same position as the pre-waiting position PW_pre, or may be a position different from the pre-waiting position PW_pre.
- Each of the second standby position PW2 and the fourth standby position PW4 described above is a specific example of a post-standby position PW_post.
- the movement path MRr of the tool center point TCP from the fine approach position PF to the post-waiting position PW_post may be determined in advance by the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP from the fine approach position PF to the post-waiting position PW_post based on the recipe file 321.
- the control device 3 may generate a robot control signal for moving the robot 1 so as to move the tool center point TCP along the movement path MRr determined by the recipe file 321.
- control device 3 when moving the tool center point TCP from the fine approach position PF to the post-waiting position PW_post, the control device 3 does not need to calculate at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D.
- the movement path MRr of the tool center point TCP from the fine approach position PF to the post waiting position PW_post and the movement path MRr of the tool center point TCP from the pre waiting position PW_pre to the fine approach position PF are indicated by the same reference symbol "MRr". This is because the movement path MRr of the tool center point TCP from the fine approach position PF to the post waiting position PW_post and the movement path MRr of the tool center point TCP from the pre waiting position PW_pre to the fine approach position PF are common in that they are movement paths along which the tool center point TCP moves by a rough movement operation and are determined in advance by the recipe file 321.
- the movement path MRr of the tool center point TCP from the fine approach position PF to the post waiting position PW_post and the movement path MRr of the tool center point TCP from the pre waiting position PW_pre to the fine approach position PF may be different from each other. That is, the pre-wait position PW_pre and the post-wait position PW_post may be different from each other.
- the movement path MRr of the tool center point TCP from the fine approach position PF to the post-wait position PW_post may be the same as the movement path MRr of the tool center point TCP from the pre-wait position PW_pre to the fine approach position PF. That is, the pre-wait position PW_pre and the post-wait position PW_post may be the same.
- control device 3 may sequentially calculate at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the fine approach position PF to the post waiting position PW_post (i.e., while the end effector 4 is moving), calculate a movement path MRr of the tool center point TCP from the fine approach position PF (or from the current position) to the post waiting position PW_post based on at least one of the position and orientation of the object OBJ sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may sequentially calculate at least one of the positions and orientations of surrounding objects located around the object OBJ based on at least one of the image data IMG_2D and IMG_3D while the tool center point TCP is moving from the fine approach position PF to the post waiting position PW_post (i.e., while the end effector 4 is moving), calculate a movement path MRr of the tool center point TCP from the fine approach position PF (or from the current position) to the post waiting position PW_post based on at least one of the positions and orientations of the surrounding objects sequentially calculated, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may calculate at least one of the position and posture of the object OBJ based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the fine approach position PF, calculate a movement path MRr of the tool center point TCP from the fine approach position PF to the post waiting position PW_post based on the calculation results of the at least one of the position and posture of the object OBJ, and move the tool center point TCP along the calculated movement path MRr.
- the control device 3 may calculate at least one of the positions and orientations of surrounding objects located around the object OBJ based on at least one of the image data IMG_2D and IMG_3D generated by the cameras 21 and 22, respectively, at the time when the tool center point TCP is located at the fine approach position PF, calculate a movement path MRr of the tool center point TCP from the fine approach position PF to the post-waiting position PW_post based on the calculation result of at least one of the positions and orientations of the surrounding objects, and move the tool center point TCP along the calculated movement path MRr.
- the fine approach position PF may be considered to be closer to the object OBJ than each of the pre-wait position PW_pre, which is the movement start position of the tool center point TCP, and the post-wait position PW_post, which is the movement end position of the tool center point TCP.
- the fine approach position PF may be considered to be closer to the object OBJ than each of the movement start position of the end effector 4 and the movement end position of the end effector 4.
- the fine approach position PF may be referred to as the first approach position.
- the processing target position PTP may be considered to be closer to the object OBJ than each of the fine approach positions PF.
- the processing target position PTP may be referred to as the second approach position.
- each of the control device 3, the robot control device 13, and the imaging device 2 is used to control the robot 1.
- a system including at least one of the control device 3, the robot control device 13, and the imaging device 2 may be referred to as a robot control system, a control system, or an imaging system.
- a system including at least two of the control device 3, the robot control device 13, and the imaging device 2 may be referred to as a robot control system, a control system, or an imaging system.
- a system including the control device 3, the robot control device 13, and the imaging device 2 may be referred to as a robot control system, a control system, or an imaging system.
- Fig. 5 is a block diagram showing the configuration of the control device 3.
- the control device 3 includes a calculation device 31, a storage device 32, and a communication device 33.
- the control device 3 may further include an input device 34 and a display device 35.
- the control device 3 does not have to include at least one of the input device 34 and the display device 35.
- the calculation device 31, the storage device 32, the communication device 33, the input device 34, and the display device 35 may be connected via a data bus 36.
- the arithmetic device 31 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array).
- the arithmetic device 31 reads a computer program.
- the arithmetic device 31 may read a computer program stored in the storage device 32.
- the arithmetic device 31 may read a computer program stored in a computer-readable and non-transient recording medium using a recording medium reading device (not shown) provided in the control device 3.
- the arithmetic device 31 may obtain (i.e., download or read) a computer program from a device (not shown) located outside the control device 3 via the communication device 33 (or other communication device).
- the arithmetic device 31 executes the read computer program.
- a logical function block for executing the processing to be performed by the control device 3 (for example, the robot control processing described above) is realized within the calculation device 31.
- the calculation device 31 can function as a controller for realizing a logical function block for executing the processing to be performed by the control device 3.
- a computation model that can be constructed by machine learning may be implemented in the computation device 31 by the computation device 31 executing a computer program.
- An example of a computation model that can be constructed by machine learning is, for example, a computation model including a neural network (so-called artificial intelligence (AI)).
- learning of the computation model may include learning of parameters of the neural network (for example, at least one of weights and biases).
- the computation device 31 may execute a robot control process using the computation model.
- the operation of executing the robot control process may include the operation of executing the robot control process using the computation model.
- a computation model that has already been constructed by offline machine learning using teacher data may be implemented in the computation device 31.
- the computation model implemented in the computation device 31 may be updated by online machine learning on the computation device 31.
- the computing device 31 may execute the robot control process using a computation model implemented in a device external to the computing device 31 (i.e., a device provided outside the control device 3) in addition to or instead of the computation model implemented in the computing device 31.
- the recording medium for recording the computer program executed by the arithmetic unit 31 may be at least one of the following: CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, optical disks such as Blu-ray (registered trademark), magnetic media such as magnetic tape, magneto-optical disk, semiconductor memory such as USB memory, and any other medium capable of storing a program.
- the recording medium may include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which a computer program is implemented in a state in which it can be executed in at least one form such as software and firmware).
- each process or function included in the computer program may be realized by a logical processing block realized within the computing device 31 (i.e., a computer) as the computing device 31 executes the computer program, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the computing device 31, or may be realized in a form that combines logical processing blocks and partial hardware modules that realize some elements of the hardware.
- FPGA Field Programmable Gate Array
- ASIC Application Specific Integrated Circuit
- the calculation device 31 realizes a camera setting unit 311, a calibration unit 312 which may be called a “calculation unit”, a processing position and orientation setting unit 313 which may be called a “setting unit”, a recognition setting unit 314, a path generating unit 315, a robot control unit 316, a display signal generating unit 317 which may be called a “signal generating unit”, and a display signal output unit 318 which may be called an "output unit”.
- the storage device 32 can store desired data.
- the storage device 32 may temporarily store a computer program executed by the arithmetic device 31.
- the storage device 32 may temporarily store data that is temporarily used by the arithmetic device 31 when the arithmetic device 31 is executing a computer program.
- the storage device 32 may store data that the control device 3 stores for a long period of time.
- the storage device 32 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, an optical magnetic disk device, an SSD (Solid State Drive), and a disk array device.
- the storage device 32 may include a non-temporary recording medium.
- the storage device 32 stores a recipe file 321.
- the recipe file 321 includes information that specifies the content of a series of operations that the robot system SYS should perform using at least one of the robot 1 and the end effector 4.
- the control device 3 generates the recipe file 321 before actually moving the robot 1.
- the recipe file 321 generated by the control device 3 is stored in the storage device 32.
- the communication device 33 is capable of communicating with each of the robot 1 and the imaging device 2 via a communication network (not shown).
- the communication device 33 may be capable of communicating with another device different from the robot 1 and the imaging device 2 in addition to or instead of at least one of the robot 1 and the imaging device 2 via a communication network (not shown).
- the communication device 33 may receive (i.e., acquire) image data IMG_2D and IMG_3D from the imaging device 2.
- the communication device 33 may transmit (i.e., output) a robot control signal to the robot 1.
- the input device 34 is a device that accepts information input to the control device 3 from outside the control device 3.
- the input device 34 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by a user of the control device 3.
- the input device 34 may include a recording medium reading device that can read information recorded as data on a recording medium that can be attached externally to the control device 3.
- information can be input as data to the control device 3 from a device external to the control device 3 via the communication device 33.
- the communication device 33 may function as an input device that accepts information input to the control device 3 from outside the control device 3.
- the display device 35 is a device capable of outputting information as an image to the outside of the control device 3.
- the display device 35 is a display device (a so-called display) capable of displaying an image.
- information can be output as data from the control device 3 to a device external to the control device 3 via the communication device 33.
- the communication device 33 may function as an output device that outputs information to a device external to the control device 3.
- Fig. 5 is a flowchart showing the flow of the robot control processing.
- control device 3 is started and then connected to each of the robot 1 and the imaging device 2 (step S1). Specifically, initial settings are performed regarding the connections between the control device 3 and each of the robot 1 and the imaging device 2 so that the control device 3 can communicate with each of the robot 1 and the imaging device 2.
- control device 3 generates the recipe file 321 (step S2). That is, the control device 3 generates the recipe file 321 before starting a predetermined process on the object OBJ using the end effector 4. The control device 3 generates the recipe file 321 before moving the robot 1.
- the control device 3 may perform initial settings of the imaging device 2 (step S21).
- the camera setting unit 311 may set imaging parameters that specify the operation content of the imaging device 2.
- the camera setting unit 311 may set imaging parameters that specify the operation content of the imaging device 2 so that the camera 21 can appropriately generate image data IMG_2D.
- the camera setting unit 311 may set imaging parameters that specify the operation content of the imaging device 2 so that the camera 21 can appropriately generate image data IMG_3D.
- An example of the imaging parameters is a parameter that specifies at least one of the operation content of the cameras 21 and 22.
- An example of the parameters that specify at least one of the operation content of the cameras 21 and 22 is at least one of the exposure time, sensitivity, aperture value, exposure value, gamma value, shutter speed, and white balance.
- An example of the imaging parameters is a parameter that specifies the operation content of the projection device 23.
- parameters that define the operation of the projection device 23 include at least one of the intensity of the projection light, the projection pattern of the projection light, the projection timing of the projection light, and the projection time of the projection light.
- the control device 3 may perform initial settings of the imaging device 2 based on the characteristics of the object OBJ captured by the imaging device 2.
- the characteristics of the object OBJ include at least one of the size of the object OBJ, the shape of the object OBJ, and the optical characteristics of the object OBJ (e.g., optical characteristics for visible light).
- the optical characteristics of the object OBJ include at least one of the reflectance, the absorbance, the scattering degree, and the diffusion degree.
- the control device 3 may perform initial settings of the imaging device 2 based on the conditions of the environment in which the object OBJ is located, in addition to or instead of the characteristics of the object OBJ captured by the imaging device 2.
- the conditions of the environment in which the object OBJ is located include the brightness of the environment (e.g., the intensity of ambient light such as sunlight or lighting).
- Information regarding the results of the initial settings of the imaging device 2 may be stored in the storage device 32 as at least part of the recipe file 321.
- the control device 3 (particularly, the camera setting unit 311) does not need to perform initial settings of the imaging device 2. In this case, the control device 3 (particularly, the calculation device 31) does not need to be equipped with the camera setting unit 311.
- the control device 3 may perform a calibration process for the imaging device 2 (step S21).
- the calibration process is a process for relating the robot coordinate system, which is the coordinate system of the robot 1, to the imaging coordinate system, which is the coordinate system of the imaging device 2.
- the robot coordinate system may be a three-dimensional coordinate system determined based on the robot 1.
- the robot coordinate system is typically a three-dimensional coordinate system set on the base 11 provided on the robot 1.
- the robot coordinate system is typically a three-dimensional coordinate system set on the base 11 that does not move even when the robot 1 operates (i.e., the robot arm 12 moves).
- the robot coordinate system may typically be a three-dimensional coordinate system different from the three-dimensional coordinate system set on the base 11 provided on the robot 1.
- the robot coordinate system may be a three-dimensional coordinate system set on the tip (i.e., the hand) of the robot arm 12.
- the robot coordinate system may be a three-dimensional coordinate system set on the end effector 4 attached to the robot arm 12.
- the robot coordinate system may be a three-dimensional coordinate system whose origin is the tool center point TCP of the end effector 4 attached to the robot arm 12.
- the imaging coordinate system may be a three-dimensional coordinate system defined based on the imaging device 2.
- the imaging coordinate system may include at least one of a three-dimensional coordinate system defined with respect to camera 21 and a three-dimensional coordinate system defined with respect to camera 22.
- the three-dimensional coordinate system defined with respect to camera 21 is referred to as the 2D imaging coordinate system
- the three-dimensional coordinate system defined with respect to camera 22 is referred to as the 3D imaging coordinate system.
- the control device 3 may perform at least one of a process of associating the robot coordinate system with the 2D imaging coordinate system and a process of associating the robot coordinate system with the 3D imaging coordinate system as a calibration process.
- the 2D imaging coordinate system is a coordinate system defined by the mutually orthogonal X-axis (2D), Y-axis (2D), and Z-axis (2D). At least one of the X-axis (2D), Y-axis (2D), and Z-axis (2D) may be an axis along the optical axis AX21 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) of the camera 21.
- the 3D imaging coordinate system is a coordinate system defined by the X-axis (3D), Y-axis (3D), and Z-axis (3D) that are mutually orthogonal.
- At least one of the X-axis (3D), Y-axis (3D), and Z-axis (3D) may be an axis along the optical axis AX22 (see FIG. 2) of the optical system (particularly, the final optical element such as an objective lens) of the camera 22.
- the 3D imaging coordinate system may be a coordinate system based on any one of the three or more monocular cameras provided in the camera 22.
- the 3D imaging coordinate system may be a coordinate system based on one of the two monocular cameras provided in the camera 22.
- At least one of the X axis (3D), Y axis (3D), and Z axis (3D) may be an axis aligned with the optical axis of the optical system (particularly the final optical element such as the objective lens) of the monocular camera.
- the calibration process for relating the robot coordinate system and the imaging coordinate system may include a process for generating a calibration matrix for converting three-dimensional coordinates in either the robot coordinate system or the imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system and the imaging coordinate system.
- the calibration matrix may include a translation transformation matrix for translating three-dimensional coordinates in either the robot coordinate system or the imaging coordinate system.
- the calibration matrix may include a translation transformation matrix for converting the position of an object in either the robot coordinate system or the imaging coordinate system into the position of the object in the other of the robot coordinate system and the imaging coordinate system.
- the calibration matrix may include a rotation transformation matrix for rotating three-dimensional coordinates in either the robot coordinate system or the imaging coordinate system.
- the calibration matrix may include a rotation transformation matrix for converting the posture of an object in either the robot coordinate system or the imaging coordinate system into the posture of the object in the other of the robot coordinate system and the imaging coordinate system.
- the calibration unit 312 may generate a calibration matrix including at least one of a translation transformation matrix and a rotation transformation matrix by performing the calibration process.
- the calibration unit 312 may control the imaging device 2 to capture an image of a predetermined calibration member CP used for the calibration process, and generate the above-mentioned calibration matrix based on the calibration image data IMG_calib generated by the imaging device 2.
- the calibration member CP may be located at a known position on the robot coordinate system.
- the calibration unit 312 may control the camera 21 to capture an image of a predetermined calibration member CP used for the calibration process, and use the image data IMG_2D generated by the camera 21 as the calibration image data IMG_calib to generate a calibration matrix for converting three-dimensional coordinates in either the robot coordinate system or the 2D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system and the 2D imaging coordinate system.
- the calibration unit 312 may control the camera 22 to capture an image of a specific calibration member CP used for the calibration process, and use the image data IMG_3D generated by the camera 22 as the calibration image data IMG_calib to generate a calibration matrix for converting three-dimensional coordinates in either the robot coordinate system or the 3D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system and the 3D imaging coordinate system.
- the calibration process for associating the robot coordinate system with the imaging coordinate system may be considered to be equivalent to a process for determining the relative positional relationship between the robot 1 (particularly, the robot arm 12) that is the reference of the robot coordinate system and the imaging device 2 that is the reference of the imaging coordinate system.
- the calibration process for associating the robot coordinate system with the imaging coordinate system may be considered to be equivalent to a process for determining the relative position of the imaging device 2 with respect to the robot 1 (particularly, the robot arm 12).
- the above-mentioned calibration matrix is a matrix specific to the relative positional relationship between the robot 1 (particularly, the robot arm 12) and the imaging device 2.
- the calibration process for associating the robot coordinate system with the 2D imaging coordinate system may be considered to be equivalent to a process for determining the relative positional relationship between the robot 1 (particularly, the robot arm 12) and the camera 21.
- the calibration process for associating the robot coordinate system with the 3D imaging coordinate system may be considered to be equivalent to a process for determining the relative positional relationship between the robot 1 (particularly, the robot arm 12) and the camera 22.
- Information regarding the results of the calibration process may be stored in the storage device 32 as at least part of the recipe file 321.
- the control device 3 (particularly, the calibration unit 312) does not need to perform calibration processing of the imaging device 2. In this case, the control device 3 (particularly, the calculation device 31) does not need to be equipped with the calibration unit 312.
- the control device 3 may set processing position and orientation information PAI (step S23).
- the processing position and orientation information PAI includes information that can be used to calculate at least one of the position where the end effector 4 should be located when the end effector 4 performs a predetermined process on the object OBJ (i.e., the processing target position PTP described above) and the orientation that the end effector 4 should take when the end effector 4 performs a predetermined process on the object OBJ (hereinafter referred to as the processing target orientation PTA).
- the processing target position PTP where the end effector 4 performs a predetermined process on the object OBJ may be considered to mean the relative position of the end effector 4 with the object OBJ as the reference.
- the processing target position PTP where the end effector 4 performs a predetermined process on the object OBJ may be considered to mean the relative position of the tool center point TCP with the object OBJ as the reference.
- the processing target orientation PTA in which the end effector 4 performs a predetermined process on the object OBJ may be considered to mean the relative orientation of the end effector 4 with the object OBJ as the reference.
- the processing target orientation PTA in which the end effector 4 performs a predetermined process on the object OBJ may be considered to mean the relative orientation of the tool center point TCP with the object OBJ as the reference.
- the processing position and orientation information PAI may include information indirectly indicating the processing target position PTP using the position of the end effector 4 with respect to the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ.
- the processing position and orientation information PAI may include information indicating the position of the end effector 4 with respect to the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ, as information for calculating the processing target position PTP.
- the processing position and orientation information PAI may include information indirectly indicating the processing target orientation PTA using the orientation of the end effector 4 with respect to the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ.
- the processing position and orientation information PAI may include information indicating the orientation of the end effector 4 with respect to the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ, as information for calculating the processing target orientation PTA.
- the processing target position PTP where the end effector 4 is actually located is a position in the global coordinate system
- the processing target orientation PTA actually taken by the end effector 4 is an orientation in the global coordinate system
- the processing position and orientation information PAI may indicate at least one of the processing target position PTP and the processing target orientation PTA using at least one of the position and orientation of the end effector 4 based on the object OBJ in a coordinate system different from the global coordinate system.
- the processing position and orientation setting unit 313 generates the processing position and orientation information PAI using an object model OM, which is a three-dimensional model of the object OBJ, and an end effector model EM, which is a three-dimensional model of the end effector 4.
- the processing position and orientation information PAI may indicate at least one of the processing target position PTP and the processing target orientation PTA by indicating at least one of the position and orientation of the end effector 4 based on the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ in a coordinate system based on the object model OM or the end effector model EM.
- the processing position and orientation information PAI may indicate at least one of the processing target position PTP and the processing target orientation PTA by indicating at least one of the position and orientation of the end effector 4 based on the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ in a coordinate system defined separately from the global coordinate system with the object OBJ located in the global coordinate system as the reference.
- the processing position and orientation information PAI may indicate at least one of the processing target position PTP and the processing target orientation PTA by indicating at least one of the position and orientation of the end effector 4 with respect to the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ in a coordinate system defined separately from the global coordinate system with the end effector 4 located in the global coordinate system as a reference.
- the processing position and orientation information PAI may indicate at least one of the processing target position PTP and the processing target orientation PTA by indicating at least one of the position and orientation of the end effector 4 with respect to the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ in a coordinate system defined separately from the global coordinate system with the tool center point TCP located in the global coordinate system as a reference.
- the position of the end effector 4 with respect to the object OBJ may mean the position of the tool center point TCP with respect to the object OBJ.
- the orientation of the end effector 4 relative to the object OBJ may refer to the orientation of the tool center point TCP relative to the object OBJ.
- processing position and orientation information PAI indicating the position of the end effector 4 based on the object OBJ may be considered to indicate the positional relationship between the object OBJ and the end effector 4.
- processing position and orientation information PAI indicating the position of the end effector 4 based on the object OBJ may be considered to indicate the position of the object OBJ based on the end effector 4.
- processing position and orientation information PAI indicating the orientation of the end effector 4 based on the object OBJ may be considered to indicate the orientation relationship between the object OBJ and the end effector 4.
- processing position and orientation information PAI indicating the orientation of the end effector 4 based on the object OBJ may be considered to indicate the orientation of the object OBJ based on the end effector 4.
- the control device 3 may calculate at least one of the processing target position PTP and the processing target orientation PTA in the global coordinate system based on the calculation result of at least one of the position and orientation of the object OBJ in the global coordinate system and the processing position and orientation information PAI generated by the processing position and orientation setting unit 313 when performing a fine movement operation.
- the robot control unit 316 may calculate the processing target position PTP in the global coordinate system by changing the calculation result of the position of the object OBJ in the global coordinate system by an amount of change corresponding to the relative position of the end effector 4 with respect to the object OBJ indicated by the processing position and orientation information PAI.
- the robot control unit 316 may calculate the processing target orientation PTA in the global coordinate system by changing the calculation result of the orientation of the object OBJ in the global coordinate system by an amount of change corresponding to the relative orientation of the end effector 4 with respect to the object OBJ indicated by the processing position and orientation information PAI.
- the robot control unit 316 may control the robot 1 so that the end effector 4 moves toward the calculated processing target position PTP and/or so that the end effector 4 takes the calculated processing target orientation PTA.
- the robot control unit 316 may generate a movement path MRf along which the end effector 4 moves toward the calculated processing target position PTP, and control the robot 1 so that the end effector 4 moves along the generated movement path MRf and/or so that the orientation of the end effector 4 after moving along the movement path MRf becomes the processing target orientation PTA.
- the processing position and orientation setting unit 313 may set, as the processing target position PTP, processing position and orientation information PAI that can be used to calculate the above-mentioned holding target position PG (see Figures 3(b) to 3(c)) at which the end effector 4 holds the workpiece W. Furthermore, the processing position and orientation setting unit 313 may set, as the processing target orientation PTA, processing position and orientation information PAI that can be used to calculate the orientation of the end effector 4 holding the workpiece W.
- the processing position and orientation setting unit 313 may set, as the processing target position PTP, processing position and orientation information PAI that can be used to calculate the above-mentioned release target position PR (see Figures 4(b) to 4(c)) at which the end effector 4 releases the workpiece W. Furthermore, the processing position and orientation setting unit 313 may set processing position and orientation information PAI that can be used to calculate the orientation of the end effector 4 that releases the workpiece W as the processing target orientation PTA.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI that can be used to calculate at least one of the position of the end effector 4 along the X axis of the global coordinate system, the position of the end effector 4 along the Y axis of the global coordinate system, and the position of the end effector 4 along the Z axis of the global coordinate system as the processing target position PTP.
- processing position and orientation information PAI that can be used to calculate at least one of the position of the end effector 4 along the X axis of the global coordinate system, the position of the end effector 4 along the Y axis of the global coordinate system, and the position of the end effector 4 along the Z axis of the global coordinate system as the processing target position PTP.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI that can be used to calculate at least one of the position of the end effector 4 around the X axis of the global coordinate system, the position of the end effector 4 around the Y axis of the global coordinate system, and the position of the end effector 4 around the Z axis of the global coordinate system as the processing target orientation PTA.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI that can be used to calculate at least one of the position of the end effector 4 in the rotation direction around the X axis of the global coordinate system, the position of the end effector 4 in the rotation direction around the Y axis of the global coordinate system, and the position of the end effector 4 in the rotation direction around the Z axis of the global coordinate system as the processing target orientation PTA.
- processing position and orientation information PAI that can be used to calculate at least one of the position of the end effector 4 in the rotation direction around the X axis of the global coordinate system, the position of the end effector 4 in the rotation direction around the Y axis of the global coordinate system, and the position of the end effector 4 in the rotation direction around the Z axis of the global coordinate system as the processing target orientation PTA.
- the processing position and orientation setting unit 313 may set the processing target position PTP by setting the relative positional relationship between the end effector 4 and the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ.
- the processing position and orientation setting unit 313 may set the processing target position PTP by setting the relative position of the end effector 4 with respect to the object OBJ on which the end effector 4 performs a predetermined processing.
- the processing target position PTP where the end effector 4 performs a predetermined processing on the object OBJ may be considered to mean the relative position of the end effector 4 with the object OBJ as the reference.
- the processing target position PTP where the end effector 4 performs a predetermined processing on the object OBJ may be considered to mean the relative position of the tool center point TCP with the object OBJ as the reference.
- Information regarding the set processing target position PTP may be stored in the storage device 32 as at least part of the recipe file 321.
- the control device 3 (particularly, the processing position and orientation setting unit 313) does not need to set the processing position and orientation information PAI. In this case, the control device 3 (particularly, the calculation device 31) does not need to be equipped with the processing position and orientation setting unit 313.
- the control device 3 may set object recognition parameters used to calculate at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D (step S24).
- An example of the object recognition parameter is a matching determination threshold, which will be described later.
- the matching determination threshold will be described later together with the process of step S33 in FIG. 6 for calculating at least one of the position and orientation of the object OBJ.
- Information regarding the set object recognition parameters may be stored in the storage device 32 as at least part of the recipe file 321.
- the control device 3 (particularly, the recognition setting unit 314) does not need to set object recognition parameters. In this case, the control device 3 (particularly, the calculation device 31) does not need to be equipped with the recognition setting unit 314.
- the control device 3 may generate a movement path MRr of the end effector 4 between the fine approach position PF and the pre-wait position PW_pre or the post-wait position PW_post (step S25).
- the pre-wait position PW_pre and the post-wait position PW_post are collectively referred to as the wait positions PW.
- the path generating unit 315 may generate the movement path MRr of the tool center point TCP (or the movement path MRr of an arbitrary movement reference point MBP) as the movement path MRr of the end effector 4.
- the movement path MRr generated by the path generating unit 315 indicates the target of the movement path to which the end effector 4 (tool center point TCP) should move at the fine approach position PF and the wait position PW, so the movement path MRr may be referred to as the target movement path.
- the path generating unit 315 may generate a series of paths including the movement path of the end effector 4 between the fine approach position PF and the processing target position PTP as the movement path MRr, in addition to the movement path of the end effector 4 between the fine approach position PF and the pre-wait position PW_pre or the post-wait position PW_post.
- the path generating unit 315 may generate the movement path MRf of the end effector 4 between the fine approach position PF and the processing target position PTP generated by the robot control unit 316 based on at least one of the image data IMG_2D and IMG_3D, without using the image data IMG_2D and IMG_3D.
- the robot control unit 316 may not need to generate the movement path MRf of the end effector 4 between the fine approach position PF and the processing target position PTP when performing the above-mentioned fine movement operation.
- the robot control unit 316 may control the robot 1 so that the end effector 4 moves between the fine approach position PF and the processing target position PTP along a movement path generated in advance by the path generation unit 315.
- the path generating unit 315 may generate at least one of a movement path MRr of the tool center point TCP from the first waiting position PW1 to the first fine approach position PF1 and a movement path MRr of the tool center point TCP from the first fine approach position PF1 to the second waiting position PW2.
- the end effector 4 performs a release process to release the workpiece W as shown in FIG. 4(a) to FIG.
- the path generating unit 315 may generate at least one of a movement path MRr of the tool center point TCP from the third waiting position PW3 to the second fine approach position PF2 and a movement path MRr of the tool center point TCP from the second fine approach position PF2 to the fourth waiting position PW4.
- Information about the generated movement path MRr may be stored in the storage device 32 as at least part of the recipe file 321.
- the control device 3 (particularly, the path generation unit 315) does not need to generate the movement path MRr. In this case, the control device 3 (particularly, the calculation device 31) does not need to be equipped with the path generation unit 315.
- control device 3 controls the operation of the robot 1 and the end effector 4 so that the end effector 4 performs a predetermined process (steps S31 to S37).
- control device 3 starts moving the robot 1 based on the recipe file 321.
- the robot control unit 316 generates a robot control signal for controlling the robot 1 so that the tool center point TCP moves from the pre-wait position PW_pre to the fine approach position PF based on the recipe file 321 previously generated in step S2 (step S31). For example, when the end effector 4 performs a holding process to hold the workpiece W as shown in FIG. 3(a) to FIG. 3(d), the robot control unit 316 may generate a robot control signal for controlling the robot 1 so that the tool center point TCP moves along the movement path MRr from the first wait position PW1 to the first fine approach position PF1 indicated by the recipe file 321. For example, when the end effector 4 performs a release process to release the workpiece W as shown in FIG.
- the robot control unit 316 may generate a robot control signal for controlling the robot 1 so that the tool center point TCP moves along the movement path MRr from the second wait position PW2 to the second fine approach position PF2 indicated by the recipe file 321.
- the robot 1 moves the tool center point TCP from the pre-wait position PW_pre to the fine approach position PF along the movement path MRr indicated by the recipe file 321.
- the robot control unit 316 After the tool center point TCP starts moving along the movement path MRr, the robot control unit 316 generates a robot control signal for controlling the robot 1 to move the tool center point TCP from the fine approach position PF to the processing target position PTP (steps S32 to S34).
- the robot control unit 316 acquires image data IMG_2D from the camera 21 using the communication device 33 (step S32).
- the camera 21 captures an image of the object OBJ at a predetermined 2D imaging rate.
- the camera 21 may capture an image of the object OBJ at a 2D imaging rate that captures an image of the object OBJ tens to hundreds of times (for example, 500 times) per second.
- the camera 21 generates image data IMG_2D at a period corresponding to the predetermined 2D imaging rate.
- the camera 21 may generate tens to hundreds of pieces of image data IMG_2D (for example, 500 pieces) per second.
- the control device 3 acquires the image data IMG_2D each time the camera 21 generates the image data IMG_2D.
- the robot control unit 316 may acquire tens to hundreds of pieces of image data IMG_2D (for example, 500 pieces) per second.
- the camera 21 does not have to capture the object OBJ according to a predetermined 2D imaging rate. For example, after the tool center point TCP starts moving along the movement path MRr, the camera 21 may capture the object OBJ one or more times and generate image data IMG_2D. Then, the control device 3 may acquire the generated image data IMG_2D.
- the image data IMG_2D may be referred to as two-dimensional position data indicating the two-dimensional position of the object OBJ reflected in the image data IMG_2D.
- the image data IMG_2D (two-dimensional position data) may be data indicating the two-dimensional position of at least a portion of the surface of the object OBJ.
- the image data IMG_2D (two-dimensional position data) is data indicating the two-dimensional position of each of multiple points of the object OBJ.
- the image data IMG_2D (two-dimensional position data) may be data indicating the two-dimensional position of each of multiple points on the surface of the object OBJ.
- the image data IMG_2D (two-dimensional position data) may be data indicating the two-dimensional position of each of multiple points corresponding to multiple parts of the surface of the object OBJ.
- the robot control unit 316 further acquires image data IMG_3D from the camera 22 using the communication device 33 (step S32).
- the camera 22 captures the object OBJ at a predetermined 3D imaging rate.
- the 3D imaging rate is the same as the 2D imaging rate.
- the 3D imaging rate may be different from the 2D imaging rate.
- the camera 22 may capture the object OBJ at a 3D imaging rate that captures the object OBJ tens to hundreds of times (for example, 500 times) per second.
- the camera 22 generates image data IMG_3D at a period corresponding to the predetermined 3D imaging rate.
- the camera 22 may generate tens to hundreds of pieces of image data IMG_3D (for example, 500 pieces) per second.
- the robot control unit 316 acquires the image data IMG_3D each time the camera 22 generates the image data IMG_3D. That is, the control device 3 may acquire tens to hundreds (for example, 500) of image data IMG_3D per second.
- the camera 22 does not have to capture the object OBJ according to a predetermined 3D imaging rate. For example, after the tool center point TCP starts to move along the movement path MRr, the camera 22 may capture the object OBJ one or more times to generate image data IMG_3D. Then, the control device 3 may acquire the generated image data IMG_3D.
- the robot control unit 316 Each time the robot control unit 316 acquires image data IMG_3D, the robot control unit 316 generates three-dimensional position data WSD based on the acquired image data IMG_3D.
- the three-dimensional position data WSD is data indicating the three-dimensional position of the object OBJ reflected in the image data IMG_3D.
- the three-dimensional position data WSD may be data indicating the three-dimensional position of at least a part of the surface of the object OBJ.
- the three-dimensional position data WSD is data indicating the three-dimensional position of each of multiple points of the object OBJ.
- the three-dimensional position data WSD may be data indicating the three-dimensional position of each of multiple points on the surface of the object OBJ.
- the three-dimensional position data WSD may be data indicating the three-dimensional position of each of multiple points corresponding to multiple parts of the surface of the object OBJ.
- the image represented by the image data IMG_3D reflects the object OBJ onto which the projection pattern is projected.
- the projection pattern reflected in the image represented by the image data IMG_3D reflects the three-dimensional shape of at least a part of the surface of the object OBJ onto which the projection pattern is projected.
- the shape of the projection pattern reflected in the image represented by the image data IMG_3D reflects the three-dimensional shape of at least a part of the surface of the object OBJ onto which the projection pattern is projected.
- the robot control unit 316 can calculate the three-dimensional shape of at least a part of the surface of the object OBJ based on the projection pattern reflected in the image represented by the image data IMG_3D.
- the three-dimensional shape of at least a part of the surface of the object OBJ essentially indicates the three-dimensional position of each of the multiple points of the object OBJ. This is because each of the multiple points of the object OBJ is included in the surface of the object OBJ. Therefore, the process of calculating the three-dimensional shape of at least a portion of the surface of the object OBJ may be considered to be substantially equivalent to the process of calculating the three-dimensional position of each of multiple points of the object OBJ. Therefore, the robot control unit 316 can generate the three-dimensional position data WSD based on the image data IMG_3D.
- the depth image data is image data in which depth information is associated with each pixel of the depth image indicated by the depth image data in addition to brightness information.
- the depth information is information indicating the distance (i.e., depth) between each part of the object OBJ reflected in each pixel and the camera 22.
- the depth image data may be image data in which the brightness information of each pixel indicates the depth of each part of the object OBJ (i.e., the distance between each part of the object OBJ and the camera 22).
- the robot control unit 316 may calculate the distance between each part of the object OBJ reflected in the image indicated by the image data IMG_3D and the camera 22 based on the projection pattern reflected in the image indicated by the image data IMG_3D, and generate a depth image by associating the calculated distance with each pixel of the image indicated by the image data IMG_3D as depth information.
- Point cloud data is data indicating a collection of points in a three-dimensional space corresponding to each part of the object OBJ reflected in the image indicated by the image data IMG_3D.
- Point cloud data may be referred to as data indicating the three-dimensional position of each point corresponding to each part of the object OBJ.
- the robot control unit 316 may generate the point cloud data by a well-known method.
- the robot control unit 316 may generate the point cloud data based on the depth image data and the camera parameters of the camera 22.
- the imaging device 2 (robot system SYS) may not include the projection device 23, and the projection device 23 may not project a projection pattern.
- the robot control unit 316 may generate the three-dimensional position data WSD based on image data IMG_3D generated by imaging an object OBJ, onto which no projection pattern is projected, with the camera 22, by a known method.
- the robot control unit 316 may acquire image data IMG_2D from the camera 21 before the tool center point TCP reaches the fine approach position PF (step S32). That is, the robot control unit 316 may acquire image data IMG_2D from the camera 21 while the tool center point TCP is moving along the movement path MRr. In other words, the robot control unit 316 may acquire image data IMG_2D from the camera 21 while the camera 21 is moving relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRr.
- the robot control unit 316 may acquire image data IMG_2D from the camera 21 after the tool center point TCP reaches the fine approach position PF.
- the robot control unit 316 may acquire image data IMG_2D from the camera 21 when the tool center point TCP has reached the fine approach position PF and the robot 1 is stopped (i.e., the camera 21 is stopped).
- the robot control unit 316 may acquire image data IMG_3D from the camera 22 before the tool center point TCP reaches the fine approach position PF (step S32). That is, the robot control unit 316 may acquire image data IMG_3D from the camera 22 while the tool center point TCP is moving along the movement path MRr. In other words, the robot control unit 316 may acquire image data IMG_3D from the camera 22 while the camera 22 is moving relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRr.
- the robot control unit 316 may acquire image data IMG_3D from the camera 22 after the tool center point TCP reaches the fine approach position PF.
- the robot control unit 316 may acquire image data IMG_3D from the camera 22 when the tool center point TCP has reached the fine approach position PF and the robot 1 is stopped (i.e., the camera 22 is stopped).
- the robot control unit 316 may also generate the three-dimensional position data WSD before the tool center point TCP reaches the fine approach position PF. That is, the robot control unit 316 may generate the three-dimensional position data WSD while the tool center point TCP is moving along the movement path MRr. In other words, the robot control unit 316 may generate the three-dimensional position data WSD while the camera 22 is moving relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRr. However, the robot control unit 316 may also generate the three-dimensional position data WSD after the tool center point TCP reaches the fine approach position PF. The robot control unit 316 may generate the three-dimensional position data WSD in a state in which the tool center point TCP reaches the fine approach position PF and the robot 1 is stopped (that is, the camera 22 is stopped).
- the robot control unit 316 calculates at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and the three-dimensional position data WSD (step S33).
- the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ by performing a matching process using at least one of the image data IMG_2D and the three-dimensional position data WSD.
- the robot control unit 316 may perform a 2D matching process using the image data IMG_2D (two-dimensional position data) and two-dimensional model data indicating a two-dimensional model serving as a reference for the object OBJ.
- the robot control unit 316 may perform an object detection process as the 2D matching process, in which the two-dimensional model indicated by the two-dimensional model data is used as a template image in the image indicated by the image data IMG_2D to detect the object OBJ indicated by the template image.
- the robot control unit 316 may perform an object detection process as the matching process, in which the robot control unit 316 detects a similar image portion similar to the template image in the image indicated by the image data IMG_2D to detect the object OBJ in the image indicated by the image data IMG_2D.
- the 2D matching process itself may be the same as an existing matching process.
- the robot control unit 316 may perform 2D matching processing using well-known methods such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed-Upped Robust Feature).
- the two-dimensional model data may be data based on the three-dimensional model data of the object OBJ.
- the two-dimensional model data may be CAD (Computer Aided Design) model data of the object OBJ, or may be model data (data showing a polygon model or mesh model) of the object OBJ generated based on a prior measurement result using a known three-dimensional shape measurement device or the camera 22.
- the two-dimensional model data may be two-dimensional image data showing at least a portion of the two-dimensional models of the multiple objects OBJ generated by virtually projecting at least a portion of the three-dimensional model shown by the three-dimensional model data of the object OBJ from multiple different directions onto a virtual plane perpendicular to each of the multiple different directions.
- the two-dimensional model data does not have to be data based on the three-dimensional model data of the object OBJ.
- the two-dimensional model data may be a characteristic location on an image shown by image data IMG_2D generated in advance by capturing an image of the object OBJ with the camera 21.
- the characteristic location on the image represented by the image data IMG_2D may be at least one of a characteristic point and an edge of an object OBJ that appears in the image represented by the image data IMG_2D. If the characteristic location on the image represented by the image data IMG_2D includes an edge of the object OBJ, the two-dimensional model may be considered to be a model of at least a portion of the edge of the object OBJ (edge model).
- the characteristic location on the image represented by the image data IMG_2D may be at least one of a plurality of characteristic points of the object OBJ that appear in the image represented by the image data IMG_2D, and at least a portion of the edges of the object OBJ.
- the characteristic location on the image represented by the image data IMG_2D may be detected by well-known image processing.
- the robot control unit 316 may perform at least one of translation, enlargement, reduction, and/or rotation of the two-dimensional model of the object OBJ reflected in the template image so that a characteristic location (e.g., at least one of a feature point and an edge) in the entire two-dimensional model of the object OBJ reflected in the template image approaches (typically coincides with) a characteristic location in the entire object OBJ reflected in the image indicated by the image data IMG_2D.
- the robot control unit 316 can identify the positional relationship between the coordinate system of the two-dimensional model data and the 2D imaging coordinate system.
- the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ in the 2D imaging coordinate system from at least one of the position and orientation of the object OBJ in the coordinate system of the two-dimensional model data based on the positional relationship between the coordinate system of the two-dimensional model data and the 2D imaging coordinate system.
- the robot control unit 316 converts at least one of the position and orientation of the object OBJ in the 2D imaging coordinate system into at least one of the position and orientation of the object OBJ in the robot coordinate system based on the calibration matrix indicated by the recipe file 321.
- the robot coordinate system is used as the global coordinate system
- at least one of the position and orientation of the object OBJ in the robot coordinate system corresponds to at least one of the position and orientation of the object OBJ in the global coordinate system.
- the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ in the global coordinate system from at least one of the position and orientation of the object OBJ in the robot coordinate system, using a conversion matrix for converting three-dimensional coordinates in either the robot coordinate system or the global coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the global coordinate system.
- the robot control unit 316 may calculate a matching similarity, which is the similarity between the template image (i.e., the two-dimensional model of the object OBJ) and the image indicated by the image data IMG_2D (particularly, the image portion to which the object OBJ of the template image is fitted).
- the robot control unit 316 may perform at least one of translation, enlargement, reduction, and/or rotation of the two-dimensional model of the object OBJ reflected in the template image so as to maximize the matching similarity.
- the image portion of the image indicated by the image data IMG_2D to which the two-dimensional model of the object OBJ of the template image is fitted is detected as a similar image portion similar to the template image in the image indicated by the image data IMG_2D.
- the object OBJ indicated by the similar image portion is detected.
- the matching similarity may be considered to be equivalent to the correlation degree indicating the correlation between the template image and the image indicated by the image data IMG_2D.
- the correlation degree may also be considered an index indicating the correlation between the two-dimensional model of the object OBJ reflected in the template image and the object OBJ reflected in the image indicated by the image data IMG_2D.
- the matching similarity may also be referred to as a matching score.
- the robot control unit 316 may select the object OBJ as an object OBJ for which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may not select the object OBJ as an object OBJ for which the end effector 4 should perform a predetermined process.
- the camera 21 may capture an image of the multiple workpieces W and generate image data IMG_2D in which the multiple workpieces W are captured as multiple objects OBJ.
- the camera 21 may capture an image of the multiple workpieces W and generate image data IMG_2D in which the multiple workpieces W are captured as multiple objects OBJ.
- the robot control unit 316 may select one object OBJ corresponding to a matching similarity exceeding a matching determination threshold indicated by the recipe file 321 among the plurality of objects OBJ as one object OBJ on which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may select, from among the plurality of objects OBJ, an object OBJ that corresponds to a matching similarity that exceeds the matching judgment threshold and is closest to the end effector 4 as an object OBJ on which the end effector 4 should perform a predetermined process.
- the storage box CB mounting device T
- the robot control unit 316 may select, from among the plurality of objects OBJ, an object OBJ that corresponds to a matching similarity that exceeds the matching judgment threshold and has the largest Z coordinate along the Z axis (located at the highest position), as an object OBJ on which the end effector 4 should perform a predetermined process.
- the Z coordinate of each of the plurality of objects OBJ may be a Z position along the Z axis of the global coordinate system of the robot 1, calculated by the above-mentioned 2D matching process.
- the robot control unit 316 may select, from among the multiple objects OBJ, an object OBJ that corresponds to a matching similarity that exceeds the matching judgment threshold and for which the end effector 4 can perform a predetermined process, as an object OBJ for which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may select, from among the multiple objects OBJ reflected in the image indicated by the image data IMG_2D, an object OBJ that has the largest area reflected in the image indicated by the image data IMG_2D, as an object OBJ for which the end effector 4 should perform a predetermined process.
- the robot control unit 316 does not need to perform the 2D matching process sequentially for the multiple objects OBJ reflected in the image indicated by the image data IMG_2D.
- the robot control unit 316 may perform the 2D matching process in parallel for each of the multiple objects OBJ reflected in the image indicated by the image data IMG_2D.
- the robot control unit 316 may select at least two of the multiple objects OBJ as one object OBJ for which the end effector 4 should perform a predetermined process. In other words, the robot control unit 316 may select at least two objects OBJ for which the end effector 4 should perform a predetermined process simultaneously.
- the following description will be given of an example in which the robot control unit 316 selects one object OBJ of the multiple objects OBJ as one object OBJ for which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may calculate the matching similarity of the multiple objects OBJ, and then select an object OBJ for which the end effector 4 should perform a predetermined process based on the matching similarity of the multiple objects OBJ, and then calculate at least one of the position and orientation of the selected object OBJ. In this case, the robot control unit 316 may not need to calculate the position and orientation of the other object OBJ that was not selected as an object OBJ for which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may consider that it has calculated the above-mentioned matching similarity together with at least one of the position and orientation of the object OBJ. In this case, the robot control unit 316 may calculate the matching similarity of the multiple objects OBJ together with at least one of the position and orientation of each of the multiple objects OBJ, and then select an object OBJ for which the end effector 4 should perform a predetermined process based on the matching similarity of the multiple objects OBJ.
- the robot control unit 316 may perform a 3D matching process using the three-dimensional position data WSD and three-dimensional model data indicating a three-dimensional model serving as a reference for the object OBJ.
- the 3D matching process is, for example, a matching process using a three-dimensional model based on the three-dimensional position data WSD and a three-dimensional model indicated by the three-dimensional model data.
- the three-dimensional model based on the three-dimensional position data WSD may include a point cloud model, which is a three-dimensional model indicating the three-dimensional shape of the object OBJ using the point cloud indicated by the three-dimensional position data WSD.
- the three-dimensional model based on the three-dimensional position data WSD may include a polygon model or a mesh model generated by connecting a plurality of points constituting the point cloud indicated by the three-dimensional position data WSD with a surface (particularly, a polygonal surface having each point as a vertex).
- the three-dimensional model based on the three-dimensional position data WSD may include a solid model generated by connecting a plurality of points constituting the point cloud indicated by the three-dimensional position data WSD with a smooth surface.
- the 3D matching process itself may be the same as an existing matching process.
- the robot control unit 316 may perform the 3D matching process using a known method including at least one of ICP (Iterative Closest Point) and DSO (Direct Sparse Odometry).
- the three-dimensional model data may be CAD model data of the object OBJ, or may be model data (data representing a polygon model or mesh model) of the object OBJ generated based on a prior measurement result using a known three-dimensional shape measurement device or the camera 22.
- the three-dimensional model data may be at least a part of the CAD model represented by the CAD model data of the object OBJ, or may be at least a part of the model represented by the model data (e.g., data representing a polygon model, mesh model, or solid model) of the object OBJ generated in advance.
- the robot control unit 316 may, for example, translate, enlarge, reduce, and/or rotate the three-dimensional model represented by the three-dimensional model data in the 3D imaging coordinate system so that a characteristic location of the three-dimensional model represented by the three-dimensional model data approaches (typically coincides with) a characteristic location of the object OBJ whose three-dimensional position data WSD indicates.
- the robot control unit 316 may change the positional relationship between the coordinate system of the three-dimensional model data (for example, the coordinate system of a CAD model) and the 3D imaging coordinate system so that a characteristic location of the object OBJ whose three-dimensional position data WSD indicates approaches (typically coincides with) a characteristic location of the three-dimensional model represented by the three-dimensional model data.
- the robot control unit 316 can identify the positional relationship between the coordinate system of the three-dimensional model data and the 3D imaging coordinate system. Then, the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ in the 3D imaging coordinate system from at least one of the position and orientation of the three-dimensional model in the coordinate system of the three-dimensional model data based on the positional relationship between the coordinate system of the three-dimensional model data and the 3D imaging coordinate system.
- the robot control unit 316 converts at least one of the position and orientation of the object OBJ in the 3D imaging coordinate system into at least one of the position and orientation of the object OBJ in the robot coordinate system based on the calibration matrix indicated by the recipe file 321.
- the robot coordinate system is used as the global coordinate system
- at least one of the position and orientation of the object OBJ in the robot coordinate system corresponds to at least one of the position and orientation of the object OBJ in the global coordinate system.
- the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ in the global coordinate system from at least one of the position and orientation of the object OBJ in the robot coordinate system, using a conversion matrix for converting three-dimensional coordinates in either the robot coordinate system or the global coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the global coordinate system.
- the robot control unit 316 may calculate a matching similarity, which is the similarity between the three-dimensional model of the object OBJ indicated by the three-dimensional model data and the object OBJ whose three-dimensional position data WSD indicates.
- the robot control unit 316 may translate, enlarge, reduce, and/or rotate the three-dimensional model indicated by the three-dimensional model data so as to maximize the matching similarity.
- the matching similarity may be considered to be equivalent to the correlation degree indicating the correlation between the three-dimensional model indicated by the three-dimensional model data and the object OBJ whose three-dimensional position data WSD indicates.
- the matching similarity may be referred to as a matching score.
- the robot control unit 316 may select the object OBJ as an object OBJ for which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may not select the object OBJ as an object OBJ for which the end effector 4 should perform a predetermined process.
- the three-dimensional position data WSD may indicate the respective positions of the multiple objects OBJ.
- the camera 22 may capture the multiple workpieces W and generate image data IMG_3D in which the multiple workpieces W are captured as multiple objects OBJ.
- the camera 22 may capture the multiple workpieces W and generate image data IMG_3D in which the multiple workpieces W are captured as multiple objects OBJ.
- the robot control unit 316 may generate three-dimensional position data WSD indicating the respective positions of the multiple objects OBJ. In this case, the robot control unit 316 may perform the above-mentioned 3D matching process on each of the multiple objects OBJ. Thereafter, the robot control unit 316 may select one object OBJ having the maximum matching similarity among the plurality of objects OBJ as one object OBJ on which the end effector 4 should perform a predetermined process. Alternatively, the robot control unit 316 may select one object OBJ having a matching similarity that exceeds the matching determination threshold indicated by the recipe file 321 as one object OBJ on which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may select one object OBJ having a matching similarity that exceeds the matching determination threshold and that is closest to the end effector 4 as one object OBJ on which the end effector 4 should perform a predetermined process.
- the storage box CB mounting device T
- the robot control unit 316 may select, from among the plurality of objects OBJ, an object OBJ that corresponds to a matching similarity that exceeds the matching judgment threshold and has the largest Z coordinate along the Z axis (located at the highest position), as an object OBJ on which the end effector 4 should perform a predetermined process.
- the Z coordinate of each of the plurality of objects OBJ may be a Z position along the Z axis of the global coordinate system of the robot 1, calculated by the above-mentioned 3D matching process.
- the robot control unit 316 may select, from among the plurality of objects OBJ, an object OBJ that corresponds to a matching similarity that exceeds the matching judgment threshold and on which the end effector 4 can perform a predetermined process, as an object OBJ on which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may select at least two of the multiple objects OBJ as one object OBJ for which the end effector 4 should perform a predetermined process. In other words, the robot control unit 316 may select at least two objects OBJ for which the end effector 4 should perform a predetermined process simultaneously.
- the robot control unit 316 selects one object OBJ of the multiple objects OBJ as one object OBJ for which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may calculate the matching similarity of the multiple objects OBJ, and then select an object OBJ for which the end effector 4 should perform a predetermined process based on the matching similarity of the multiple objects OBJ, and then calculate at least one of the position and orientation of the selected object OBJ. In this case, the robot control unit 316 may not need to calculate the position and orientation of the other object OBJ that was not selected as an object OBJ for which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may consider that it has calculated the above-mentioned matching similarity together with at least one of the position and orientation of the object OBJ. In this case, the robot control unit 316 may calculate the matching similarity of the multiple objects OBJ together with at least one of the position and orientation of each of the multiple objects OBJ, and then select an object OBJ for which the end effector 4 should perform a predetermined process based on the matching similarity of the multiple objects OBJ.
- the robot control unit 316 may use the image data IMG_2D while calculating at least one of the position and orientation of the object OBJ without using the three-dimensional position data WSD. In this case, the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ in the global coordinate system from at least one of the position and orientation of the object OBJ in the 2D imaging coordinate system calculated based on the image data IMG_2D. In this case, the camera 22 does not need to image the object OBJ. The robot control unit 316 does not need to acquire the image data IMG_3D or generate the three-dimensional position data WSD in step S32.
- the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ by using the three-dimensional position data WSD without using the image data IMG_2D. In this case, the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ in the global coordinate system from at least one of the position and orientation of the object OBJ in the 3D imaging coordinate system calculated based on the image data IMG_3D. In this case, the camera 21 does not need to capture an image of the object OBJ. The robot control unit 316 does not need to acquire the image data IMG_2D in step S32.
- the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ by using both the image data IMG_2D and the three-dimensional position data WSD. In this case, the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ in the global coordinate system from a portion of the position and orientation of the object OBJ in the 2D imaging coordinate system calculated based on the image data IMG_2D and a portion of the position and orientation of the object OBJ in the 3D imaging coordinate system calculated based on the image data IMG_3D.
- the robot control unit 316 may set at least one of the initial position and initial orientation of a three-dimensional model used when performing the 3D matching process based on at least one of the position and orientation of the object OBJ in the 2D imaging coordinate system calculated based on the image data IMG_2D, and perform the 3D matching process using the three-dimensional model arranged at the set initial position and/or in the set initial orientation to calculate at least one of the position and orientation of the object OBJ in the 3D imaging coordinate system, and calculate at least one of the position and orientation of the object OBJ in the global coordinate system from at least one of the position and orientation of the object OBJ in the 3D imaging coordinate system.
- the robot control unit 316 may select an object OBJ that satisfies the condition that the sum or product of the matching similarity calculated by the 2D matching process (hereinafter, 2D matching similarity) and the matching similarity calculated by the 3D matching process (hereinafter, 3D matching similarity) is maximized, as an object OBJ on which the end effector 4 should perform a predetermined processing.
- the robot control unit 316 may select an object OBJ that satisfies the condition that the 2D matching similarity is maximized, as an object OBJ on which the end effector 4 should perform a predetermined processing.
- the robot control unit 316 may select an object OBJ that satisfies the condition that the 3D matching similarity is maximized, as an object OBJ on which the end effector 4 should perform a predetermined processing.
- the robot control unit 316 may select an object OBJ that satisfies the condition that both the 2D matching similarity and the 3D matching similarity exceed the matching judgment threshold as an object OBJ on which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may select an object OBJ that satisfies the condition that both the 2D matching similarity and the 3D matching similarity exceed the matching judgment threshold and is closest to the end effector 4 as an object OBJ on which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may select an object OBJ that satisfies the condition that both the 2D matching similarity and the 3D matching similarity exceed the matching judgment threshold and has the largest Z coordinate along the Z axis (is located at the highest position) as an object OBJ on which the end effector 4 should perform a predetermined process.
- the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ before the tool center point TCP reaches the fine approach position PF (step S33). In other words, the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ while the tool center point TCP is moving along the movement path MRr.
- the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ while the cameras 21 and 22 are moving relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRr. However, the robot control unit 316 may calculate at least one of the position and orientation of the object OBJ after the tool center point TCP reaches the fine approach position PF. The robot control unit 316 may calculate at least one of the position and orientation of the object OBJ when the tool center point TCP reaches the fine approach position PF and the robot 1 is stopped (i.e., the camera 22 is stopped).
- the robot control unit 316 generates a robot control signal for controlling the robot 1 to move the tool center point TCP from the fine approach position PF to the processing target position PTP based on at least one of the position and orientation of the object OBJ calculated in step S33 (step S34).
- the robot control unit 316 generates a robot control signal for controlling the robot 1 to move the tool center point TCP closer to the object OBJ (i.e., the end effector 4 closer to the object OBJ) based on at least one of the position and orientation of the object OBJ, and thereby to position the robot 1 at the processing target position PTP indicated by the recipe file 321 (step S34).
- the robot control unit 316 may generate a movement path MRf of the tool center point TCP between the fine approach position PF and the processing target position PTP based on at least one of the position and orientation of the object OBJ.
- the processing target position PTP means a position based on the object OBJ. Therefore, the robot control unit 316 calculates the processing target position PTP based on the object OBJ based on at least one of the position and orientation of the object OBJ. After that, the robot control unit 316 generates a movement path MRf that moves the tool center point TCP so that the tool center point TCP approaches the processing target position PTP.
- the robot control unit 316 may generate a movement path MRf based on at least one of the position and orientation of the object OBJ so that interference between at least one of the end effector 4, the robot 1, and the surrounding objects does not occur. After that, the robot control unit 316 may generate a robot control signal for controlling the robot 1 so that the tool center point TCP moves along the generated movement path MRf.
- the peripheral object is an object located in the vicinity of the robot 1.
- An example of a peripheral object is a mounting device T (e.g., a pallet) on which the workpiece W is placed.
- An example of a peripheral object is a stand on which the mounting device T is placed.
- An example of a peripheral object is an object (e.g., at least one of piping, shelves, walls, and ceilings) that exists in the factory in which the robot system SYS is installed.
- An example of a peripheral object is an object (e.g., a conveying machine such as a belt conveyor) that constitutes the production line in which the robot system SYS is installed.
- An example of a peripheral object is the imaging device 2 attached to the robot 1.
- An example of a peripheral object is at least one of the cameras 21 and 22 attached to the robot 1.
- the other object OBJ that is not selected as the object OBJ for which the end effector 4 should perform a predetermined process may be regarded as a peripheral object.
- the other object OBJ that is not selected as the object OBJ for which the end effector 4 should perform a predetermined process in a first period may be selected as the object OBJ for which the end effector 4 should perform a predetermined process in a second period different from the first period.
- the object OBJ for which the end effector 4 should perform a predetermined process in the first period may be regarded as a peripheral object in the second period different from the first period.
- the interference of at least one of the end effector 4, the robot 1, and the surrounding object may include interference between at least a part of the end effector 4 and at least a part of the robot 1.
- the interference of at least one of the end effector 4, the robot 1, and the surrounding object may include interference between at least a part of the end effector 4 and at least a part of the surrounding object.
- the interference of at least one of the end effector 4, the robot 1, and the surrounding object may include interference between at least a part of the robot 1 and at least a part of the surrounding object.
- the interference of at least one of the end effector 4, the robot 1, and the surrounding object may include interference between a first part of the robot 1 and a second part of the robot 1.
- the interference of two objects may include a state in which the two objects are in contact.
- the interference of two objects may include a state in which the distance between the two objects is equal to or less than the allowable distance AD1.
- the robot control unit 316 may use the end effector model EM, the robot model RM, the peripheral object model SM, and the object model OM. However, the robot control unit 316 may generate the movement path MRf without using at least one of the end effector model EM, the robot model RM, the peripheral object model SM, and the object model OM.
- the end effector model EM is a three-dimensional model of the end effector 4 attached to the robot arm 12.
- the end effector model EM may be a CAD model of the end effector 4.
- the end effector model EM may be a three-dimensional model (e.g., a point cloud model) obtained by actually measuring the three-dimensional shape of the end effector 4.
- the end effector model EM may be a polygon model or a mesh model generated by connecting multiple points constituting the point cloud model of the end effector 4 (particularly, polygonal faces with each point as a vertex).
- the three-dimensional model obtained by actually measuring the three-dimensional shape of the end effector 4 may be, for example, a model generated by measuring the end effector 4 with a well-known three-dimensional shape measuring device, or may be a model generated based on image data obtained by imaging the end effector 4 with the camera 22.
- the robot model RM is a three-dimensional model of the robot 1.
- the robot model RM may include a three-dimensional model of the base 11 of the robot 1.
- the robot model RM may include a three-dimensional model of the robot arm 12 of the robot 1.
- the robot model RM may be a CAD model of the robot 1.
- the robot model RM may be a three-dimensional model (e.g., a point cloud model) obtained by actually measuring the three-dimensional shape of the robot 1.
- the robot model RM may be a polygon model or a mesh model generated by connecting multiple points constituting the point cloud model of the robot 1 (particularly, polygonal faces with each point as a vertex).
- the three-dimensional model obtained by actually measuring the three-dimensional shape of the robot 1 may be, for example, a model generated by measuring the robot 1 with a well-known three-dimensional shape measuring device, or may be a model generated based on image data obtained by capturing an image of the robot 1 with the camera 22.
- the surrounding object model SM is a three-dimensional model of the surrounding objects located around the robot 1.
- the surrounding object model SM may be a CAD model of the surrounding object.
- the surrounding object model SM may be a three-dimensional model (e.g., a point cloud model) obtained by actually measuring the three-dimensional shape of the surrounding object.
- the surrounding object model SM may be a polygon model or a mesh model generated by connecting multiple points constituting the point cloud model of the surrounding object (particularly, polygonal faces with each point as a vertex).
- the three-dimensional model obtained by actually measuring the three-dimensional shape of the surrounding object may be, for example, a model generated by measuring the surrounding object with a well-known three-dimensional shape measurement device, or may be a model generated based on image data obtained by capturing an image of the surrounding object with the camera 22.
- the object model OM is a three-dimensional model of the object OBJ, which is the target object on which the end effector 4 performs a predetermined process.
- the object model OM may be a CAD model of the object OBJ.
- the object model OM may be a three-dimensional model (e.g., a point cloud model) obtained by actually measuring the three-dimensional shape of the object OBJ.
- the object model OM may be a polygon model generated by connecting multiple points constituting the point cloud model of the object OBJ (particularly, polygonal faces with each point as a vertex).
- a three-dimensional model that serves as a reference for the object OBJ used in the above-mentioned 3D matching process may be used as the object model OM.
- a three-dimensional model different from the three-dimensional model that serves as a reference for the object OBJ used in the above-mentioned 3D matching process may be used as the object model OM.
- the three-dimensional model obtained by actually measuring the three-dimensional shape of the object OBJ may be, for example, a model generated by measuring the object OBJ with a well-known three-dimensional shape measurement device, or may be a model generated based on image data obtained by capturing an image of the object OBJ with the camera 22.
- the object OBJ on which the end effector 4 performs a predetermined process may be considered to be an example of a peripheral object since it is an object located in the vicinity of the robot 1. Therefore, the object model OM may be considered to be an example of a peripheral object model SM.
- the robot control unit 316 may generate a movement path MRf of the tool center point TCP that does not cause interference with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM (note that the peripheral object model SM includes the object model OM, the same below), as a movement path MRf that does not cause interference with at least one of the end effector 4, the robot 1, and the peripheral object (including the object OBJ on which the end effector 4 performs a predetermined process, the same below).
- the robot control unit 316 may generate a movement path MRf that does not cause interference with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM by placing the end effector model EM, the robot model RM, and the peripheral object model SM in a predetermined virtual space and moving the end effector model EM and the robot model RM in the virtual space in accordance with the movements of the end effector 4 and the robot 1 when it is assumed that the robot 1 is driven.
- a movement path MRf that does not cause interference with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM by placing the end effector model EM, the robot model RM, and the peripheral object model SM in a predetermined virtual space and moving the end effector model EM and the robot model RM in the virtual space in accordance with the movements of the end effector 4 and the robot 1 when it is assumed that the robot 1 is driven.
- the robot control unit 316 may generate a path candidate from the fine approach position PF to the processing target position PTP (or a path candidate from the processing target position PTP to the fine approach position PF, the same below), and determine whether or not interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM when it is assumed that the tool center point TCP moves along the path candidate. If it is determined that interference does not occur with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM, the robot control unit 316 may set the generated path candidate as the movement path MRf.
- the robot control unit 316 may update (i.e., generate a new path candidate) the path candidate, and determine whether or not interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM when it is assumed that the tool center point TCP moves along the updated path candidate. In other words, the robot control unit 316 may generate the movement route MRf collectively.
- the robot control unit 316 may repeat the following operations: generating a portion of the candidate path; determining whether interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM when it is assumed that the tool center point TCP moves along the portion of the generated candidate path; and updating (newly generating) a portion of the candidate path if it is determined that interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM.
- the robot control unit 316 may generate a movement path MRf that gradually extends from the fine approach position PF toward the processing target position PTP. In other words, the robot control unit 316 may generate the movement path MRf little by little.
- the interference of at least one of the end effector model EM, the robot model RM, and the peripheral object model SM may include interference between at least a part of the end effector model EM and at least a part of the robot model RM.
- the interference of at least one of the end effector model EM, the robot model RM, and the peripheral object model SM may include interference between at least a part of the end effector model EM and at least a part of the peripheral object model SM.
- the interference of at least one of the end effector model EM, the robot model RM, and the peripheral object model SM may include interference between at least a part of the robot model RM and at least a part of the peripheral object model SM.
- the interference of at least one of the end effector model EM, the robot model RM, and the peripheral object model SM may include interference between a first part of the robot model RM and a second part of the robot model RM.
- the interference of the two models may include a state in which the two models are in contact.
- the interference of the two models may include a state in which the distance between the two models is equal to or less than the allowable distance AD2.
- the allowable distance AD2 may be a distance determined based on the allowable distance AD1 described above.
- the allowable distance AD2 may be set to a distance that satisfies the condition that the distance between the two objects is equal to or less than the allowable distance AD1 when two models corresponding to two objects respectively are arranged in a virtual space such that the positional relationship between the two models is the same as the positional relationship between the two objects and the posture relationship between the two models is the same as the posture relationship between the two objects.
- the allowable distance AD2 may be the same as or different from the allowable distance AD1 described above.
- step S34 three-dimensional position data WSD (particularly point cloud data) generated based on image data IMG_3D generated by the camera 22 capturing an image of the object OBJ may be used as the object model OM.
- the robot control unit 316 may convert the coordinates of each point indicated by the point cloud data of the object OBJ from the coordinates of the 3D imaging coordinate system to coordinates in virtual space, and place each point with the converted coordinates in the virtual space.
- the object model OM may be a polygon model or mesh model generated by connecting multiple points constituting the point cloud indicated by the three-dimensional position data WSD with a surface (particularly a polygonal surface with each point as a vertex), or a solid model generated by connecting multiple points constituting the point cloud indicated by the three-dimensional position data WSD with a smooth surface.
- the robot control unit 316 may place the polygon model, mesh model, or solid model in the virtual space.
- the robot control unit 316 may generate the movement path MRf before the tool center point TCP reaches the fine approach position PF (step S34). That is, the robot control unit 316 may generate the movement path MRf while the tool center point TCP is moving along the movement path MRr. In other words, the robot control unit 316 may generate the movement path MRf while the cameras 21 and 22 are moving relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRr.
- the robot control unit 316 may generate the movement path MRf after the tool center point TCP reaches the fine approach position PF.
- the robot control unit 316 may generate the movement path MRf when the tool center point TCP reaches the fine approach position PF and the robot 1 is stopped (i.e., the camera 22 is stopped).
- the robot control unit 316 may generate a robot control signal before the tool center point TCP reaches the fine approach position PF (step S34). That is, the robot control unit 316 may generate a robot control signal while the tool center point TCP is moving along the movement path MRr. In other words, the robot control unit 316 may generate a robot control signal while the cameras 21 and 22 are moving relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRr.
- the robot control unit 316 may generate a robot control signal after the tool center point TCP reaches the fine approach position PF.
- the robot control unit 316 may generate a robot control signal in a state in which the tool center point TCP reaches the fine approach position PF and the robot 1 is stopped (that is, the camera 22 is stopped).
- step S35 After that, the processes from step S32 to step S34 are repeated until the tool center point TCP is located at the processing target position PTP (step S35). Therefore, after the driving of the robot 1 is started so that the tool center point TCP moves from the fine approach position PF along the movement path MRf, the camera 21 may capture an image of the object OBJ while the camera 21 moves relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRf.
- the robot control unit 316 may acquire image data IMG_2D from the camera 21 while the camera 21 moves relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRf.
- the camera 22 may capture an image of the object OBJ while the camera 22 moves relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRf.
- the robot control unit 316 may acquire image data IMG_3D from the camera 22 while the camera 22 moves relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRf.
- the robot control unit 316 may calculate at least one of the position and the orientation of the object OBJ while at least one of the cameras 21 and 22 moves relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRf.
- the robot control unit 316 may generate (in this case, essentially update sequentially) the movement path MRf while at least one of the cameras 21 and 22 moves relative to the object OBJ by driving the robot 1 so that the tool center point TCP moves along the movement path MRf.
- the camera 21 may capture an image of the object OBJ while the camera 21 is stopped relative to the object OBJ.
- the robot control unit 316 may acquire image data IMG_2D from the camera 21 while the camera 21 is stopped relative to the object OBJ.
- the camera 22 may capture an image of the object OBJ while the camera 22 is stopped relative to the object OBJ.
- the robot control unit 316 may acquire image data IMG_3D from the camera 22 while the camera 22 is stopped relative to the object OBJ.
- the robot control unit 316 may calculate at least one of the position and the orientation of the object OBJ while at least one of the cameras 21 and 22 is stopped relative to the object OBJ.
- the robot control unit 316 may generate (in this case, essentially update sequentially) the movement path MRf while at least one of the cameras 21 and 22 is stopped relative to the object OBJ.
- An example of a situation where the processes from step S32 to step S34 are repeated is a situation where the object OBJ moves while the robot 1 is moving so that the tool center point TCP moves along the movement path MRf.
- the robot control unit 316 may sequentially calculate (i.e., repeatedly calculate) at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D.
- the robot control unit 316 may sequentially generate (in other words, sequentially update) the movement path MRf based on the end effector model EM, the robot model RM, and the peripheral object model SM, and at least one of the position and orientation of the object OBJ that is sequentially calculated (i.e., may be repeatedly calculated).
- step S32 to step S34 do not necessarily have to be repeated. For example, if the robot 1 moves the tool center point TCP based on the robot control signal generated in step S34, and the tool center point TCP reaches the processing target position PTP, the robot control unit 316 does not have to repeat the processes from step S32 to step S34.
- the robot control unit 316 After the tool center point TCP is positioned at the processing target position PTP, the robot control unit 316 generates a robot control signal for controlling the end effector 4 to perform a predetermined process on the object OBJ (step S36). As a result, the end effector 4 performs a predetermined process on the object OBJ (step S36). For example, the end effector 4 may perform a holding process to hold the workpiece W, which is an example of the object OBJ. For example, the end effector 4 may perform a release process to release the workpiece W, which is an example of the object OBJ, on the pallet PLT, which is an example of the object OBJ.
- the robot control unit 316 After the end effector 4 has performed a predetermined process on the object OBJ, the robot control unit 316 generates a robot control signal for controlling the robot 1 to retract the end effector 4 (step S37).
- the robot control unit 316 may generate a robot control signal for controlling the robot 1 to move (i.e., retract) the end effector 4 from the processing target position PTP to the fine approach position PF.
- the robot control unit 316 may generate the robot control signal by performing operations similar to those of steps S32 to S35.
- the robot control unit 316 may repeat the process of acquiring at least one of the image data IMG_2D and IMG_3D, calculating at least one of the position and orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D, and generating a robot control signal for controlling the robot 1 to move (i.e., retract) the end effector 4 from the processing target position PTP to the fine approach position PF based on at least one of the position and orientation of the object OBJ, until the end effector 4 moves from the processing target position PTP to the fine approach position PF.
- the robot control unit 316 may generate a robot control signal for controlling the robot 1 to move (i.e., retreat) the end effector 4 along the generated movement path MRf from the processing target position PTP to the fine approach position PF using the generated movement path MRf.
- the robot control unit 316 may generate a robot control signal in step S34 using the generated movement path MRf without performing the operations from step S32 to step S33.
- the robot control unit 316 may generate a robot control signal for controlling the robot 1 to move (i.e., retract) the end effector 4 from the fine approach position PF to the post-waiting position PW_post.
- the robot control unit 316 may generate the robot control signal by performing an operation similar to that of step S31.
- the robot control unit 316 may generate a robot control signal for controlling the robot 1 to move the end effector 4 from the fine approach position PF to the post-waiting position PW_post based on the recipe file 321.
- the calibration unit 312 may control the imaging device 2 (that is, at least one of the cameras 21 and 22) to capture a predetermined calibration member CP in order to perform the calibration processing.
- the calibration member CP may be a member on which a predetermined calibration pattern is formed.
- the calibration pattern of the calibration member CP may be a known pattern.
- the calibration pattern may be a pattern for defining a reference point of the calibration member CP. In this case, the calibration member CP may be considered to be a reference member on which a reference point is formed.
- FIG. 7 shows an example in which a checker pattern in which white rectangular patterns and black rectangular patterns are arranged in a matrix is formed on the calibration member CP as the predetermined calibration pattern.
- a checker pattern in which white rectangular patterns and black rectangular patterns are arranged in a matrix is formed on the calibration member CP as the predetermined calibration pattern.
- at least one vertex of the white rectangular pattern may be used as the reference point
- at least one vertex of the black rectangular pattern may be used as the reference point
- a boundary line between the white rectangular pattern and the black rectangular pattern may be used as the reference point.
- the calibration unit 312 acquires the calibration image data IMG_calib generated by the imaging device 2 capturing an image of the calibration member CP. Then, based on the calibration image data IMG_calib, the calibration unit 312 generates a calibration matrix for converting three-dimensional coordinates in either the robot coordinate system or the imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system and the imaging coordinate system.
- the calibration unit 312 may generate a calibration matrix for converting three-dimensional coordinates in either the robot coordinate system or the 2D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the 2D imaging coordinate system, based on the calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP.
- the calibration matrix for converting three-dimensional coordinates in either the robot coordinate system or the 2D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the 2D imaging coordinate system will be referred to as a "2D calibration matrix" as necessary.
- the calibration unit 312 may generate a calibration matrix for converting three-dimensional coordinates in either the robot coordinate system or the 3D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the 3D imaging coordinate system, based on the calibration image data IMG_calib generated by the camera 22 capturing an image of the calibration member CP.
- the calibration matrix for converting three-dimensional coordinates in either the robot coordinate system or the 3D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the 3D imaging coordinate system will be referred to as a "3D calibration matrix" as necessary.
- the calibration unit 312 generating a 2D calibration matrix for converting three-dimensional coordinates in either the robot coordinate system or the 2D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the 2D imaging coordinate system, based on the calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP.
- the calibration unit 312 generating a 3D calibration matrix for converting three-dimensional coordinates in either the robot coordinate system or the 3D imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the 3D imaging coordinate system, based on the calibration image data IMG_calib generated by the camera 22 capturing an image of the calibration member CP.
- the calibration unit 312 may generate a 2D calibration matrix by solving a PnP (Perspective n Point) problem based on the calibration image data IMG_calib.
- the PnP problem in this embodiment is equivalent to a problem of estimating at least one of the relative position and the relative orientation of the camera 21 with respect to the robot 1 from the relationship between the positions (three-dimensional positions) of n reference points of the calibration member CP located in the robot coordinate system and the positions (two-dimensional positions) of n reference points of the calibration member CP on the two-dimensional image shown by the calibration image data IMG_calib.
- n is a variable indicating an integer of 1 or more.
- the relative position of the camera 21 with respect to the robot 1 may be considered to include the relative orientation of the camera 21 with respect to the robot 1.
- the calibration unit 312 may calculate the relative position of the camera 21 with respect to the robot 1 in a direction along a predetermined translation axis as the relative position of the camera 21 with respect to the robot 1.
- the calibration unit 312 may calculate the relative position of the camera 21 with respect to the robot 1 in a direction along each of three mutually orthogonal translation axes as the relative position of the camera 21 with respect to the robot 1.
- the calibration unit 312 may calculate a translation transformation matrix that translates the three-dimensional coordinates of either the robot coordinate system defined based on the robot 1 or the 2D imaging coordinate system defined based on the camera 21 as the relative position of the camera 21 with respect to the robot 1, and converts them into three-dimensional coordinates in the other of the robot coordinate system and the imaging coordinate system.
- the calibration unit 312 may calculate the relative orientation of the camera 21 with respect to the robot 1 in the direction of rotation around a predetermined rotation axis as the relative orientation of the camera 21 with respect to the robot 1. In other words, the calibration unit 312 may calculate the relative amount of rotation of the camera 21 with respect to the robot 1 around a predetermined rotation axis (in other words, the relative orientation) as the relative orientation of the camera 21 with respect to the robot 1. In particular, the calibration unit 312 may calculate the relative orientation of the camera 21 with respect to the robot 1 in the direction of rotation around each of three mutually orthogonal rotation axes as the relative orientation of the camera 21 with respect to the robot 1.
- the calibration unit 312 may calculate the relative amount of rotation of the camera 21 with respect to the robot 1 around each of three mutually orthogonal rotation axes (in other words, the relative orientation) as the relative orientation of the camera 21 with respect to the robot 1.
- the calibration unit 312 may calculate a rotation transformation matrix that rotates and moves the three-dimensional coordinates of either the robot coordinate system defined based on the robot 1 or the 2D imaging coordinate system defined based on the camera 21 as the relative orientation of the camera 21 with respect to the robot 1, and converts them into three-dimensional coordinates in the other of the robot coordinate system and the imaging coordinate system.
- the calibration unit 312 can generate a 2D calibration matrix that includes at least one of the calculated translation transformation matrix and the calculated rotation transformation matrix.
- the calibration unit 312 may generate a 2D calibration matrix based on a single calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP once.
- the calibration unit 312 may calculate the positions of the n reference points in the two-dimensional image indicated by the calibration image data IMG_calib, and calculate at least one of the relative position and relative orientation of the camera 21 with respect to the robot 1 (i.e., the 2D calibration matrix) based on the relationship between the positions of the n reference points in the two-dimensional image and the positions of the n reference points in the robot coordinate system, which is known information.
- the calibration unit 312 may generate a 2D calibration matrix based on a plurality of calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP multiple times.
- the camera 21 may capture an image of the same calibration member CP from a plurality of different positions.
- the camera 21 may capture an image of the same calibration member CP in a plurality of different orientations.
- the calibration unit 312 may alternately repeat an operation of controlling the camera 21 to capture an image of the calibration member CP and an operation of changing at least one of the position and orientation of the camera 21 that captured the calibration member CP. That is, the calibration unit 312 may alternately repeat an operation of controlling the camera 21 to capture an image of the calibration member CP and an operation of generating a robot control signal for controlling the robot 1 to change at least one of the position and orientation of the camera 21 that captured the calibration member CP. In other words, the calibration unit 312 may alternately repeat an operation of controlling the camera 21 to capture an image of the calibration member CP and an operation of changing at least one of the relative positional relationship and the relative posture relationship between the camera 21 that captured the image of the calibration member CP and the calibration member CP.
- the calibration unit 312 may alternately repeat an operation of controlling the camera 21 to capture an image of the calibration member CP and an operation of generating a robot control signal for controlling the robot 1 to change at least one of the relative positional relationship and the relative posture relationship between the camera 21 that captured the image of the calibration member CP and the calibration member CP.
- the relative posture relationship between the camera 21 and the calibration member CP may be included in the relative positional relationship between the camera 21 and the calibration member CP.
- the calibration unit 312 may calculate the positions of n reference points in each of the multiple two-dimensional images represented by the multiple calibration image data IMG_calib, and calculate the relative position and relative orientation of the camera 21 with respect to the robot 1 (i.e., the 2D calibration matrix) based on the positions of the n reference points in each of the multiple two-dimensional images and the movement amount of the camera 21 (i.e., the movement amount of the robot 1).
- the calibration unit 312 may calculate the relative position of the camera 21 with respect to the robot 1 (i.e., the 2D calibration matrix) based on the positions of the n reference points in each of the multiple two-dimensional images and the movement amount of the camera 21 (i.e., the movement amount of the robot 1) by utilizing the fact that one reference point in the first two-dimensional image represented by the first two-dimensional image data and the same one reference point in the second two-dimensional image represented by the second two-dimensional image data correspond to the position of one reference point in the robot coordinate system.
- the robot 1 i.e., the 2D calibration matrix
- the camera 21 may capture images of the same calibration member CP from multiple different positions in order to improve the accuracy of the calibration process.
- the camera 21 may capture images of the same calibration member CP in multiple different postures in order to improve the accuracy of the calibration process.
- the multiple different positions at which the camera 21 captures the calibration member CP may be multiple different positions set randomly within a predetermined position range, or multiple different positions that have been determined in advance.
- the predetermined position range may be set by the user, or may be fixed.
- the multiple different positions that have been determined in advance may be determined so that the multiple imaging positions at which the camera 21 captures the calibration member CP are not biased.
- the multiple different postures at which the camera 21 captures the calibration member CP may be multiple different postures set randomly within a predetermined posture range, or multiple different postures that have been determined in advance.
- the predetermined posture range may be set by the user, or may be fixed.
- the multiple different postures that have been determined in advance may be determined so that the multiple imaging postures at which the camera 21 captures the calibration member CP are not biased.
- the calibration unit 312 In the calibration process, the calibration unit 312 generates the robot control signal, but the robot control unit 316 may also generate the robot control signal in the calibration process. In this case, the robot system SYS may alternately repeat an operation of using the calibration unit 312 to control the camera 21 to capture an image of the calibration member CP and an operation of using the robot control unit 316 to generate a robot control signal for controlling the robot 1 to change at least one of the position and posture of the camera 21 that captured the image of the calibration member CP.
- the robot system SYS may alternately repeat an operation of using the calibration unit 312 to control the camera 21 to capture an image of the calibration member CP and an operation of using the robot control unit 316 to generate a robot control signal for controlling the robot 1 to change the relative positional relationship between the camera 21 that captured the image of the calibration member CP and the calibration member CP.
- the calibration unit 312 may generate a 2D calibration matrix based on multiple calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP multiple times.
- the camera 21 may generate multiple calibration image data IMG_calib by capturing an image of the same calibration member CP from multiple different positions.
- the camera 21 may generate multiple calibration image data IMG_calib by capturing an image of the same calibration member CP in multiple different postures in order to improve the accuracy of the calibration process.
- the camera 21 may capture an image of the same calibration member CP from multiple different positions in order to improve the accuracy of the calibration process.
- the camera 21 may capture an image of the same calibration member CP in multiple different postures in order to improve the accuracy of the calibration process.
- the calibration process described above may be considered to be equivalent to so-called hand-eye calibration process.
- the calibration unit 312 may generate a calibration matrix by performing hand-eye calibration process.
- the hand-eye calibration process is, as an example, an existing process for calculating the positional relationship between the tool center point of a multi-joint robot and a camera used with the multi-joint robot, so a detailed description will be omitted, but the outline has essentially already been described.
- the end effector 4 is attached to the robot 1 (robot arm 12), but the end effector 4 does not have to be attached to the robot 1 during the calibration process.
- the calibration unit 312 may perform an operation of generating a 2D calibration matrix based on the calibration image data IMG_calib generated by the camera 21 and an operation of generating a 3D calibration matrix based on the calibration image data IMG_calib generated by the camera 22 in parallel.
- the calibration unit 312 may perform an operation of generating a 3D calibration matrix based on the calibration image data IMG_calib generated by the camera 22 after or before performing an operation of generating a 2D calibration matrix based on the calibration image data IMG_calib generated by the camera 21.
- both the cameras 21 and 22 are attached to the same robot 1, the cameras 21 and 22 can simultaneously capture the same calibration member CP.
- the calibration unit 312 may display information on the accuracy (in other words, accuracy) of the calibration processing on the display device 35.
- the information on the accuracy of the calibration processing is referred to as calibration accuracy information.
- the calibration accuracy information may include information on the accuracy (in other words, accuracy) of coordinate transformation by a calibration matrix generated by the calibration processing.
- the calibration accuracy information may include information on the accuracy of coordinate transformation by a 2D calibration matrix.
- the calibration accuracy information may include information on the accuracy of coordinate transformation by a 3D calibration matrix.
- the information on the accuracy of coordinate transformation by the calibration matrix may include information on the accuracy of coordinate transformation that uses the calibration matrix to transform three-dimensional coordinates in either the robot coordinate system or the imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the imaging coordinate system.
- the information on the accuracy of coordinate transformation by the calibration matrix may include information on the accuracy of coordinate transformation by a translation transformation matrix included in the calibration matrix.
- the information on the accuracy of coordinate transformation by a translation transformation matrix may include information on the accuracy of coordinate transformation that uses the translation transformation matrix to transform three-dimensional coordinates in either the robot coordinate system or the imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the imaging coordinate system.
- the information on the accuracy of coordinate transformation by the calibration matrix may include information on the accuracy of coordinate transformation by a rotation transformation matrix included in the calibration matrix.
- the information on the accuracy of coordinate transformation by a rotation transformation matrix may include information on the accuracy of coordinate transformation that uses the rotation transformation matrix to transform three-dimensional coordinates in either the robot coordinate system or the imaging coordinate system into three-dimensional coordinates in the other of the robot coordinate system or the imaging coordinate system.
- the calibration unit 312 may perform a calibration process using the calibration image data IMG_calib, and display calibration accuracy information on the display device 35 based on the results of the calibration process. For example, the calibration unit 312 may perform a calibration process using a single calibration image data IMG_calib, and display calibration accuracy information on the display device 35 based on the results of the calibration process. For example, the calibration unit 312 may perform a calibration process using multiple calibration image data IMG_calib, and display calibration accuracy information on the display device 35 based on the results of the calibration process.
- FIG. 8 is a flowchart showing the flow of the accuracy information display process for displaying the calibration accuracy information on the display device 35. Note that FIG. 8 shows an example of an operation in which a calibration process is performed using a plurality of calibration image data IMG_calib, and calibration accuracy information is displayed based on the results of the calibration process.
- the accuracy information display process for displaying calibration accuracy information related to the accuracy of coordinate transformation using a 2D calibration matrix will be described.
- the following explanation can be used as an explanation of the accuracy information display process for displaying calibration accuracy information related to the accuracy of coordinate transformation using a 3D calibration matrix by replacing the terms "camera 21", “2D imaging coordinate system”, and “2D calibration matrix” with the terms "camera 22", “3D imaging coordinate system”, and “3D calibration matrix”, respectively.
- the calibration unit 312 acquires N pieces of calibration image data IMG_calib from the camera 21 (step S220).
- N is a variable indicating an integer equal to or greater than 2.
- the calibration unit 312 may control the camera 21 to generate N pieces of calibration image data IMG_calib by capturing an image of the calibration member CP N times.
- the calibration unit 312 may change at least one of the position and the orientation of the camera 21 every time the camera 21 captures an image of the calibration member CP.
- the calibration unit 312 may change the relative positional relationship between the camera 21 and the calibration member CP every time the camera 21 captures an image of the calibration member CP.
- the N pieces of calibration image data IMG_calib may include calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP from a first position, and calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP from a second position different from the first position.
- the N pieces of calibration image data IMG_calib may include calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP in a first orientation, and calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP in a second orientation different from the first orientation.
- the N pieces of calibration image data IMG_calib may include calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP under a condition in which the relative positional relationship between the camera 21 and the calibration member CP is a first positional relationship, and calibration image data IMG_calib generated by the camera 21 capturing an image of the calibration member CP under a condition in which the relative positional relationship between the camera 21 and the calibration member CP is a second positional relationship different from the first positional relationship.
- the calibration unit 312 divides the N pieces of calibration image data IMG_calib acquired in step S220 into a first calibration image data group including X pieces of calibration image data IMG_calib and a second calibration image data group including N-X pieces of calibration image data IMG_calib excluding the X pieces of calibration image data IMG_calib (step S221).
- X is a variable indicating an integer greater than or equal to 1 and less than N.
- the number of calibration image data IMG_calib included in the first calibration image data group is smaller than the number of calibration image data IMG_calib included in the second calibration image data group.
- the calibration unit 312 divides the N calibration image data IMG_calib into a first calibration image data group and a second calibration image data group so as to satisfy the relationship "X ⁇ N-X".
- the calibration unit 312 may classify 10% to 20% of the N calibration image data IMG_calib acquired in step S220 into the first calibration image data group.
- the number of calibration image data IMG_calib included in the first calibration image data group may be the same as the number of calibration image data IMG_calib included in the second calibration image data group, or it may be greater than the number.
- the calibration unit 312 performs a calibration process based on the X pieces of calibration image data IMG_calib included in the first calibration image data group (step S222). That is, the calibration unit 312 calculates a 2D calibration matrix by performing the above-mentioned calibration process based on the X pieces of calibration image data IMG_calib included in the first calibration image data group (step S222).
- the calibration process of step S222 performed based on the X pieces of calibration image data IMG_calib included in the first calibration image data group is referred to as the first calibration process.
- the 2D calibration matrix generated by the first calibration process of step S222 is referred to as the first calibration matrix.
- the calibration unit 312 Before or after the processing of step S222, or in parallel, the calibration unit 312 performs a calibration process based on the N-X pieces of calibration image data IMG_calib included in the second calibration image data group (step S223). That is, the calibration unit 312 calculates a 2D calibration matrix by performing the above-mentioned calibration process based on the N-X pieces of calibration image data IMG_calib included in the second calibration image data group (step S223).
- the calibration process of step S223 performed based on the N-X pieces of calibration image data IMG_calib included in the second calibration image data group is referred to as the second calibration process.
- the 2D calibration matrix generated by the second calibration process of step S223 is referred to as the second calibration matrix.
- the calibration processes performed in steps S222 and S223 are typically provisional calibration processes performed to display calibration accuracy information. Therefore, as will be described in detail later, after performing the accuracy information display process shown in FIG. 8, the calibration unit 312 may perform a final calibration process to generate a 2D calibration matrix that is actually used to control the robot 1.
- the calibration unit 312 calculates a calibration error indicating the error between the first calibration matrix and the second calibration matrix (step S224). In other words, the calibration unit 312 calculates a calibration error indicating the error between the result of the first calibration process and the result of the second calibration process (step S224).
- the calibration unit 312 may calculate, as the calibration error, the error between the result of coordinate transformation using the first calibration matrix and the result of coordinate transformation using the second calibration matrix. Specifically, the calibration unit 312 may calculate, as the calibration error, the error between the result of coordinate transformation of one three-dimensional coordinate using the first calibration matrix and the result of coordinate transformation of the same one three-dimensional coordinate using the second calibration matrix.
- the 2D calibration matrix may include a translation transformation matrix.
- the calibration unit 312 may calculate, as the calibration error, a translation transformation error equivalent to the error between the result of coordinate transformation by the translation transformation matrix included in the first calibration matrix and the result of coordinate transformation by the translation transformation matrix included in the second calibration matrix.
- the calibration unit 312 may calculate, as the calibration error, a translation transformation error equivalent to the error between the result of coordinate transformation of one three-dimensional coordinate by the translation transformation matrix included in the first calibration matrix and the result of coordinate transformation of the same one three-dimensional coordinate by the translation transformation matrix included in the second calibration matrix.
- the translation transformation matrix included in the first calibration matrix is referred to as the first translation transformation matrix
- the translation transformation matrix included in the second calibration matrix is referred to as the second translation transformation matrix.
- the translation transformation error may include a transformation error in the translation direction along the X-axis in at least one of the robot coordinate system and the 2D imaging coordinate system.
- the translation transformation error may include a transformation error in the translation direction along the Y-axis (i.e., an axis perpendicular to the X-axis) in at least one of the robot coordinate system and the 2D imaging coordinate system.
- the translation transformation error may include a transformation error in the translation direction along the Z-axis (i.e., an axis perpendicular to the X-axis and Y-axis) in at least one of the robot coordinate system and the 2D imaging coordinate system.
- the 2D calibration matrix may include a rotation transformation matrix.
- the calibration unit 312 may calculate, as the calibration error, a rotation transformation error equivalent to the error between the result of coordinate transformation by the rotation transformation matrix included in the first calibration matrix and the result of coordinate transformation by the rotation transformation matrix included in the second calibration matrix.
- the calibration unit 312 may calculate, as the calibration error, a translation transformation error equivalent to the error between the result of coordinate transformation of one three-dimensional coordinate by the rotation transformation matrix included in the first calibration matrix and the result of coordinate transformation of the same one three-dimensional coordinate by the rotation transformation matrix included in the second calibration matrix.
- the rotation transformation matrix included in the first calibration matrix is referred to as the first rotation transformation matrix
- the rotation transformation matrix included in the second calibration matrix is referred to as the second rotation transformation matrix.
- the rotation transformation error may include a transformation error in the rotation direction around the X-axis in at least one of the robot coordinate system and the 2D imaging coordinate system.
- the rotation transformation error may include a transformation error in the rotation direction around the Y-axis in at least one of the robot coordinate system and the 2D imaging coordinate system.
- the rotation transformation error may include a transformation error in the rotation direction around the Z-axis in at least one of the robot coordinate system and the 2D imaging coordinate system.
- the calibration unit 312 repeats the process from step S221 to step S224 until the calibration error including at least one of the translation transformation error and the rotation transformation error is calculated M times in step S224 (step S225).
- M is a variable indicating an integer equal to or greater than 1. Since the calibration error is calculated once in step S224 each time the first calibration process in step S222 and the second calibration process in step S223 are performed once, the calibration unit 312 repeats the process from step S221 to step S224 until the first calibration process in step S222 and the second calibration process in step S223 are performed M times (step S225).
- the calibration unit 312 repeats the process from step S221 to step S224 until the unit calibration process including the first calibration process in step S222 and the second calibration process in step S223 is performed M times (step S225).
- the calibration unit 312 calculates M calibration errors.
- the M calibration errors are values that quantitatively indicate the accuracy of the calibration process. For this reason, the M calibration errors may be considered to be an example of calibration accuracy information.
- the calibration unit 312 re-divides the N pieces of calibration image data IMG_calib acquired in step S220 into a first calibration image data group and a second calibration image data group (step S221).
- the calibration unit 312 may re-divide the N pieces of calibration image data IMG_calib into the first calibration image data group and the second calibration image data group so as to change at least one of the X pieces of calibration image data IMG_calib included in the first calibration image data group and to change at least one of the N-X pieces of calibration image data IMG_calib included in the second calibration image data group.
- the calibration unit 312 may re-divide the N calibration image data IMG_calib into the first and second calibration image data groups so that at least one calibration image data IMG_calib different from the X calibration image data IMG_calib already selected as the calibration image data IMG_calib included in the first calibration image data group is newly selected as the calibration image data IMG_calib included in the first calibration image data group.
- the calibration unit 312 may re-divide the N calibration image data IMG_calib into the first and second calibration image data groups so that at least one calibration image data IMG_calib different from the N-X calibration image data IMG_calib already selected as the calibration image data IMG_calib included in the second calibration image data group is newly selected as the calibration image data IMG_calib included in the second calibration image data group.
- the two-dimensional image indicated by at least one of the N calibration image data IMG_calib may be regarded as an example of calibration accuracy information.
- the calibration unit 312 calculates a calibration index value based on the M calibration errors calculated in step S224 (step S226).
- the calibration index value is an index value that quantitatively indicates the accuracy of the calibration process, and is calculated from the M calibration errors calculated in step S224. For this reason, the calibration index value may be considered to be an example of calibration accuracy information.
- the calibration unit 312 may calculate the average value of the M calibration errors calculated in step S224 as the calibration index value.
- the calibration unit 312 may calculate the average value of the M translation transformation errors included in each of the M calibration errors calculated in step S224 as the calibration index value.
- the calibration unit 312 may calculate the average value of the M rotation transformation errors included in each of the M calibration errors calculated in step S224 as the calibration index value.
- the calibration unit 312 may use the single calibration error calculated in step S224 as the calibration index value itself.
- the calibration unit 312 may display the calibration index value calculated in step S226 on the display device 35 as calibration accuracy information (step S227).
- the calibration unit 312 may calculate the average value of the M calibration errors as the calibration index value.
- the calibration unit 312 may calculate the median value of the M calibration errors as the calibration index value, or may calculate the deviation of the M calibration errors as the calibration index value. In other words, the calibration unit 312 may calculate the statistical value of the M calibration errors as the calibration index value.
- the calibration accuracy information is displayed under the control of the display signal generation unit 317 and the display signal output unit 318.
- the desired information including the calibration information (specifically, the desired image showing the desired information) is also displayed under the control of the display signal generation unit 317 and the display signal output unit 318.
- the display signal generation unit 317 generates a display signal for displaying the desired information on the display device 35.
- the display signal output unit 318 outputs the display signal generated by the display signal generation unit 317 to the display device 35.
- the display device 35 displays the desired information based on the display signal acquired from the display signal output unit 318.
- outputting the display signal to the display device 35 may be considered to be substantially equivalent to displaying the desired information based on the display signal on the display device 35.
- step S227 the display signal generation unit 317, under the control of the calibration unit 312, generates a display signal for displaying the calibration index value calculated in step S226 on the display device 35. Thereafter, the display signal output unit 318 outputs the display signal generated by the display signal generation unit 317 to the display device 35. As a result, the display device 35 displays the calibration index value based on the display signal acquired from the display signal output unit 318.
- the calibration unit 312 may display a display screen 3520 including a display field 3521 for displaying the calibration index value on the display device 35.
- the user of the robot system SYS can properly understand the calibration index value.
- the calibration unit 312 may display the M calibration errors calculated in step S224 of FIG. 8 on the display device 35 in addition to (or instead of) the calibration index value.
- the calibration unit 312 may display a display screen 3520 including a display field 3522 for displaying the M calibration errors on the display device 35.
- the user of the robot system SYS can properly grasp the M calibration errors in addition to (or instead of) the calibration index value.
- the calibration unit 312 may be considered to be displaying the calibration accuracy information on the display device 35. Note that when the calibration unit 312 displays the M calibration errors on the display device 35 instead of the calibration index value, the calibration unit 312 does not need to calculate the calibration index value.
- the calibration unit 312 may display on the display device 35 a two-dimensional image indicated by at least one of the N pieces of calibration image data IMG_calib acquired in step S221 in addition to (or instead of) the calibration index value.
- the calibration unit 312 may display on the display device 35 a display screen 3520 including a display field 3523 for displaying the calibration image, which is a two-dimensional image indicated by the calibration image data IMG_calib.
- the calibration unit 312 may display on the display device 35, as calibration accuracy information, the two-dimensional image indicated by at least one of the N pieces of calibration image data IMG_calib acquired in step S221 and the M calibration errors calculated in step S224 of FIG. 8.
- the user or the calibration unit 312 may evaluate the quality of the N pieces of calibration image data IMG_calib based on the calibration accuracy information (e.g., at least one of the calibration index value and the calibration error) generated by the accuracy information display process.
- the user or the calibration unit 312 may evaluate the quality of the N pieces of calibration image data IMG_calib based on the calibration accuracy information (e.g., at least one of the calibration index value and the calibration error) in the process of generating the recipe file 321 (step S2 in FIG. 6) before actually starting to move the robot 1 by performing the processes of steps S31 to S37 in FIG. 6.
- the calibration unit 312 may perform necessary processing based on the evaluation result of the quality of the N pieces of calibration image data IMG_calib.
- the calibration unit 312 may perform necessary processing based on the evaluation results of the quality of the N pieces of calibration image data IMG_calib in the process of generating the recipe file 321 (step S2 in FIG. 6).
- the user or the calibration unit 312 may evaluate that there is a high possibility that the quality of all of the N pieces of calibration image data IMG_calib acquired in step S220 reaches the acceptable quality. In other words, the user or the calibration unit 312 may evaluate that there is a low possibility that the N pieces of calibration image data IMG_calib include at least one piece of calibration image data IMG_calib whose quality does not reach the acceptable quality.
- the state in which the quality of the calibration image data IMG_calib does not reach the acceptable quality may mean a state in which a calibration matrix with sufficiently high conversion accuracy cannot be calculated by a calibration process using the calibration image data IMG_calib.
- the calibration unit 312 may perform a final calibration process using all of the N pieces of calibration image data IMG_calib acquired in step S220. In other words, the calibration unit 312 may perform the calibration process without dividing the N pieces of calibration image data IMG_calib into the first and second calibration image data groups.
- the calibration matrix generated by the final calibration process may actually be used by the control device 3 to control the robot 1.
- the robot control unit 316 may use the calibration matrix generated by the final calibration process to calculate at least one of the position and orientation of the object OBJ in the global coordinate system.
- the robot control unit 316 may use the calibration matrix generated by the final calibration process to generate a robot control signal.
- the calibration unit 312 may perform a final calibration based on an instruction from the user received using the input device 34. For example, the user who checks the display screen 3520 shown in FIG. 10 may determine whether or not the calibration index value displayed on the display screen 3520 exceeds a predetermined first tolerance threshold. In addition, the calibration unit 312 may display information on the first tolerance threshold as calibration accuracy information on the display device 35 so that the user can appropriately determine whether or not the calibration index value exceeds the predetermined first tolerance threshold. If the user determines that the calibration index value does not exceed the predetermined first tolerance threshold, the user may input an instruction to the control device 3 using the input device 34 to request that a final calibration process be performed using all of the N pieces of calibration image data IMG_calib acquired in step S220.
- the calibration unit 312 may determine whether or not the calibration index value exceeds the predetermined first tolerance threshold regardless of the presence or absence of a user instruction. If the calibration unit 312 determines that the calibration index value does not exceed the predetermined first tolerance threshold, the calibration unit 312 may perform a final calibration process using all of the N calibration image data IMG_calib acquired in step S220.
- the first tolerance threshold may be set to an appropriate value that allows a distinction to be made, from the calibration index value, between a state in which a calibration matrix with sufficiently high conversion accuracy can be calculated by the calibration process and a state in which a calibration matrix with sufficiently high conversion accuracy cannot be calculated by the calibration process.
- the first tolerance threshold may be a fixed value that cannot be changed, or a variable value that can be changed.
- the first tolerance threshold may be set by the user, may be set in advance, or may be set appropriately by the calibration unit 312.
- a calibration matrix with sufficiently high conversion accuracy may mean a calibration matrix that satisfies the condition that the difference between the conversion result obtained by converting the three-dimensional coordinates of either the robot coordinate system or the imaging coordinate system using the calibration matrix and the correct value (in other words, the ideal value) of the conversion result of the three-dimensional coordinates of either the robot coordinate system or the imaging coordinate system is within an allowable amount.
- the calibration unit 312 may select one of the M calibration matrices calculated in steps S222 to S223 as the calibration matrix actually used by the control device 3 to control the robot 1. For example, the calibration unit 312 may select one of the M calibration matrices with the smallest calibration error as the calibration matrix actually used by the control device 3 to control the robot 1.
- the user or the calibration unit 312 may evaluate the quality of the N pieces of calibration image data IMG_calib using M calibration errors in addition to or instead of the calibration index value. For example, if at least one of the M calibration errors does not exceed a predetermined first tolerance threshold, the user or the calibration unit 312 may evaluate that the N pieces of calibration image data IMG_calib are unlikely to include at least one piece of calibration image data IMG_calib whose quality does not reach the acceptable quality. In this case, the calibration unit 312 may perform a final calibration process using all of the N pieces of calibration image data IMG_calib acquired in step S220. Note that, when the quality of the N pieces of calibration image data IMG_calib is evaluated using M calibration errors, the calibration unit 312 does not need to calculate a calibration index value based on the M calibration errors.
- the calibration image data IMG_calib whose quality does not reach the acceptable quality may include calibration image data IMG_calib that causes a situation in which the calibration index value exceeds the first acceptable threshold.
- a cause for acquiring calibration image data IMG_calib whose quality does not reach the acceptable quality is that a part of the calibration member CP (particularly, a part of the calibration pattern) is not included in the imaging angle of view of the camera 21.
- the calibration image data IMG_calib whose quality does not reach the acceptable quality may include calibration image data IMG_calib that shows an image that does not include a part of the calibration member CP (particularly, a part of the calibration pattern).
- calibration image data IMG_calib whose quality does not reach the acceptable quality is that the brightness of a part of the calibration member CP falls below the lower limit of the dynamic range of the camera 21, resulting in a part of the image shown by the calibration image data IMG_calib being crushed to black.
- calibration image data IMG_calib whose quality does not reach the acceptable quality may include calibration image data IMG_calib showing an image with at least a part crushed to black.
- calibration image data IMG_calib whose quality does not reach the acceptable quality is when the brightness of a part of the calibration member CP exceeds the upper limit of the dynamic range of the camera 21, resulting in a part of the image shown by the calibration image data IMG_calib being crushed to white.
- calibration image data IMG_calib whose quality does not reach the acceptable quality may include calibration image data IMG_calib showing an image with at least a part crushed to white.
- the calibration unit 312 may reacquire N pieces of calibration image data IMG_calib.
- the calibration unit 312 may control the camera 21 to generate new N pieces of calibration image data IMG_calib by re-imaging the calibration member CP N times.
- the calibration unit 312 may perform the accuracy information display process shown in FIG. 8 again (particularly, the process after step S221).
- the calibration unit 312 may newly acquire N pieces of calibration image data IMG_calib based on an instruction from the user received using the input device 34. For example, a user who checks the display screen 3520 shown in FIG. 10 may determine whether or not the calibration index value displayed on the display screen 3520 exceeds a predetermined first tolerance threshold. In addition, the calibration unit 312 may display information on the first tolerance threshold as calibration accuracy information on the display device 35 so that the user can appropriately determine whether or not the calibration index value exceeds the predetermined first tolerance threshold. If the user determines that the calibration index value exceeds the predetermined first tolerance threshold, the user may input an instruction to the control device 3 using the input device 34 to request that N pieces of calibration image data IMG_calib be newly acquired.
- the calibration unit 312 may determine whether or not the calibration index value exceeds the predetermined first tolerance threshold regardless of the presence or absence of a user instruction. If the calibration unit 312 determines that the calibration index value exceeds a predetermined first tolerance threshold, the calibration unit 312 may reacquire N pieces of calibration image data IMG_calib.
- the calibration unit 312 may calculate a suitable position for the camera 21 to image the calibration member CP based on the already acquired N pieces of calibration image data IMG_calib. For example, the calibration unit 312 may calculate a suitable position for the camera 21 to image the calibration member CP so that a calibration matrix with sufficiently high conversion accuracy can be calculated from the newly acquired N pieces of calibration image data IMG_calib.
- the robot control unit 316 may move the robot 1 (particularly the robot arm 12) so that the camera 21 moves to the position calculated by the calibration unit 312, and then the calibration unit 312 may control the camera 21 to image the calibration member CP.
- the calibration unit 312 may display on the display device 35 a position suitable for the camera 21 to capture an image of the calibration member CP. In this case, the user may manually move the camera 21 to the calculated position.
- the calibration unit 312 may calculate a posture suitable for the camera 21 to image the calibration member CP based on the already acquired N pieces of calibration image data IMG_calib. For example, the calibration unit 312 may calculate a posture suitable for the camera 21 to image the calibration member CP so that a calibration matrix with sufficiently high accuracy can be calculated from the newly acquired N pieces of calibration image data IMG_calib. In this case, the robot control unit 316 may move the robot 1 (particularly the robot arm 12) so that the posture of the camera 21 becomes the calculated posture, and then the calibration unit 312 may control the camera 21 to image the calibration member CP.
- the calibration unit 312 may display on the display device 35 a posture suitable for the camera 21 to capture an image of the calibration member CP.
- the user may manually move the camera 21 so that the posture of the camera 21 becomes the calculated posture.
- the calibration unit 312 may calculate a positional relationship between the camera 21 and the calibration member CP suitable for the camera 21 to capture the calibration member CP based on the already acquired N pieces of calibration image data IMG_calib. For example, the calibration unit 312 may calculate a positional relationship between the camera 21 and the calibration member CP suitable for the camera 21 to capture the calibration member CP so that a calibration matrix with sufficiently high accuracy can be calculated from the newly acquired N pieces of calibration image data IMG_calib.
- the robot control unit 316 may move the robot 1 (particularly the robot arm 12) so that the positional relationship between the camera 21 and the calibration member CP becomes the calculated positional relationship, and then the calibration unit 312 may control the camera 21 to capture the calibration member CP.
- the calibration unit 312 may display on the display device 35 a positional relationship between the camera 21 and the calibration member CP that is suitable for the camera 21 to capture an image of the calibration member CP.
- the user may manually move the camera 21 so that the positional relationship between the camera 21 and the calibration member CP becomes the calculated positional relationship.
- the user or the calibration unit 312 may evaluate the quality of the N pieces of calibration image data IMG_calib using M calibration errors in addition to or instead of the calibration index value. For example, if all of the M calibration errors exceed a predetermined first tolerance threshold, the user or the calibration unit 312 may evaluate that there is a possibility that the quality of all of the N pieces of calibration image data IMG_calib does not reach the acceptable quality. In this case, the calibration unit 312 may reacquire N pieces of calibration image data IMG_calib. If the quality of the N pieces of calibration image data IMG_calib is evaluated using M calibration errors, the calibration unit 312 does not need to calculate a calibration index value based on the M calibration errors.
- the calibration image data IMG_calib whose quality does not reach the acceptable quality may include calibration image data IMG_calib that causes a situation in which the calibration error exceeds the second acceptable threshold.
- An example of calibration image data IMG_calib whose quality does not reach the acceptable quality has already been described.
- the second tolerance threshold may be set to an appropriate value that allows a distinction to be made between a state in which a calibration matrix with sufficiently high conversion accuracy can be calculated by the calibration process and a state in which a calibration matrix with sufficiently high conversion accuracy cannot be calculated by the calibration process, based on the calibration error.
- the second tolerance threshold may be a fixed value that cannot be changed, or may be a variable value that can be changed.
- the second tolerance threshold may be set by the user, may be set in advance, or may be set appropriately by the calibration unit 312.
- the calibration unit 312 may exclude at least one calibration image data IMG_calib from at least one of the first and second calibration image data groups used to calculate the one calibration error. In other words, the calibration unit 312 may exclude at least one calibration image data IMG_calib included in at least one of the first and second calibration image data groups as at least one calibration image data IMG_calib whose quality is assumed to not reach the acceptable quality.
- the calibration unit 312 may exclude at least one calibration image data IMG_calib based on an instruction from the user received using the input device 34. For example, a user who checks the display screen 3520 shown in FIG. 10 may determine whether a calibration error displayed on the display screen 3520 exceeds a predetermined second tolerance threshold. In addition, the calibration unit 312 may display information on the second tolerance threshold as calibration accuracy information on the display device 35 so that the user can appropriately determine whether a calibration index value exceeds the predetermined second tolerance threshold.
- the user may input an instruction to the control device 3 using the input device 34 to request the exclusion of at least one calibration image data IMG_calib from at least one of the first and second calibration image data groups used to calculate the calibration error.
- the calibration unit 312 may determine whether a calibration error exceeds a predetermined second tolerance threshold regardless of the presence or absence of a user instruction. If the calibration unit 312 determines that one calibration error exceeds a predetermined second tolerance threshold, the calibration unit 312 may exclude at least one calibration image data IMG_calib from at least one of the first and second calibration image data groups used to calculate the one calibration error.
- the calibration unit 312 may redo the first and second calibration processes in steps S222 and S223 of FIG. 8, respectively, using the first and second calibration image data groups from which at least one calibration image data IMG_calib has been excluded. The calibration unit 312 may then recalculate the calibration error.
- the calibration unit 312 may exclude at least one calibration image data IMG_calib from the first calibration image data group used to calculate one calibration error. That is, the calibration unit 312 may exclude at least one calibration image data IMG_calib included in the first calibration image data group as at least one calibration image data IMG_calib whose quality is assumed to not reach the acceptable quality. In this case, the calibration unit 312 is more likely to be able to exclude at least one calibration image data IMG_calib whose quality is assumed to not reach the acceptable quality, compared to the case where at least one calibration image data IMG_calib included in the second calibration image data group is excluded.
- the calibration unit 312 may redo the first and second calibration processes in steps S222 and S223 of FIG. 8, respectively, using a first calibration image data group from which at least one calibration image data IMG_calib has been excluded and a second calibration image data group from which at least one calibration image data IMG_calib has not been excluded. The calibration unit 312 may then recalculate the calibration error.
- the calibration unit 312 can reduce the number of times the first and second calibration processes are repeated. This is because, compared to the second specific example in which all calibration image data IMG_calib are reacquired, in the third specific example, it is sufficient to exclude at least one calibration image data IMG_calib whose quality is assumed to not reach the acceptable quality. As a result, the calibration unit 312 can shorten the time required for the calibration process.
- the calibration unit 312 may remove at least one calibration image data IMG_calib included in at least one of the first and second calibration image data groups used to calculate one calibration error from the N calibration image data IMG_calib, and then perform a final calibration process using the remaining calibration image data IMG_calib.
- the calibration unit 312 can generate a calibration matrix with sufficiently high conversion accuracy by the final calibration process.
- the calibration unit 312 excludes at least one calibration image data IMG_calib from at least one of the first and second calibration image data groups used to calculate one calibration error.
- the calibration unit 312 may perform a process of excluding at least one calibration image data IMG_calib from at least one of the first and second calibration image data groups used to calculate the calibration error each time a calibration error is calculated.
- the calibration unit 312 may determine whether or not the calculated calibration error exceeds the second tolerance threshold each time a calibration error is calculated.
- the calibration unit 312 may display the calculated calibration error on the display device 35 each time a calibration error is calculated, and the user may determine whether or not the displayed calibration error exceeds the second tolerance threshold.
- the calibration unit 312 may exclude at least one calibration image data IMG_calib from at least one of the first and second calibration image data groups. Thereafter, the calibration unit 312 may perform the first and second calibration processes again to recalculate the calibration error.
- the calibration unit 312 may repeat the calibration accuracy information display process shown in FIG. 8 while excluding at least one calibration image data IMG_calib from at least one of the first and second calibration image data groups used to calculate the calibration error exceeding the second tolerance threshold. For example, in step S220, the calibration unit 312 may acquire N pieces of calibration image data IMG_calib generated in the first period, and perform the processes from step S221 to step S227 using the N pieces of calibration image data IMG_calib. Furthermore, the calibration unit 312 may exclude at least one piece of calibration image data IMG_calib whose quality is assumed to not reach the tolerance quality based on the calibration index value and the calibration error.
- the calibration unit 312 may acquire new N pieces of calibration image data IMG_calib generated in a second period after the first period, and perform the processes from step S221 to step S227 using the N pieces of calibration image data IMG_calib. Thereafter, the calibration unit 312 may repeat the same operation. As a result, the calibration unit 312 may perform the final calibration process using more than N pieces of calibration image data IMG_calib remaining as a result of multiple calibration accuracy information display processes.
- the calibration unit 312 excludes at least one calibration image data IMG_calib from at least one of the first and second calibration image data groups used to calculate the one calibration error.
- the calibration unit 312 may replace at least one calibration image data IMG_calib included in at least one of the first and second calibration image data groups used to calculate the one calibration error with another calibration image data IMG_calib.
- the calibration unit 312 may exclude at least one calibration image data IMG_calib from at least one of the first and second calibration image data groups used to calculate the one calibration error, and newly add another calibration image data IMG_calib different from the excluded calibration image data IMG_calib to at least one of the first and second calibration image data groups used to calculate the one calibration error.
- the other calibration image data IMG_calib different from the excluded calibration image data IMG_calib may be calibration image data IMG_calib obtained separately from step S220.
- the calibration unit 312 can calculate calibration accuracy information (for example, at least one of a calibration index value and a calibration error) that substantially indicates the accuracy of the calibration processing in the process of generating the recipe file 321 before actually starting to move the robot 1 by performing the processes of steps S31 to S37 in FIG. 6.
- calibration accuracy information for example, at least one of a calibration index value and a calibration error
- the user or the calibration unit 312 can evaluate the quality of the calibration image data IMG_calib used to perform the calibration processing.
- the calibration unit 312 can generate a calibration matrix using calibration image data IMG_calib whose quality has reached the acceptable quality without using calibration image data IMG_calib whose quality has not reached the acceptable quality. Therefore, the calibration unit 312 can generate a calibration matrix with high conversion accuracy compared to the case where a calibration matrix is generated using calibration image data IMG_calib whose quality has not reached the acceptable quality.
- the robot control unit 316 can accurately calculate at least one of the position and orientation of the object OBJ in the robot coordinate system from at least one of the image data IMG_2D and IMG_3D using the calibration matrix with high conversion accuracy. Therefore, the robot control unit 316 can accurately move the robot 1 (particularly the robot arm 12, and essentially the end effector 4 and tool center point TCP) based on at least one of the position and orientation of the object OBJ.
- the robot control unit 316 does not move the robot 1 using a calibration matrix whose conversion accuracy is not sufficiently high.
- the robot control unit 316 does not move the robot 1 using a calibration matrix whose conversion accuracy is not sufficiently high.
- there is a low possibility that an error will occur in the movement of the robot 1 due to the low conversion accuracy of the calibration matrix and as a result, there is a low possibility that the robot 1 will collide with the object OBJ (or any other object) due to the low conversion accuracy of the calibration matrix.
- the calibration unit 312 performs the accuracy information display process shown in FIG. 8 in the process of generating the recipe file 321 before actually starting to move the robot 1 by performing the processes from step S31 to step S37 in FIG. 6.
- the calibration unit 312 may perform the accuracy information display process shown in FIG. 8 after actually starting to move the robot 1 by performing the processes from step S31 to step S37 in FIG. 6.
- the mounting positions of at least one of the cameras 21 and 22 relative to the robot 1 may be shifted due to the movement of the robot 1 (particularly, the movement of the robot arm 12).
- the mounting positions of at least one of the cameras 21 and 22 relative to the robot 1 may be shifted due to the vibration of the robot 1 (particularly, the vibration of the robot arm 12).
- the robot 1 starts to actually move by performing the processes of steps S31 to S37 in FIG. 6, at least one of the cameras 21 and 22, which move in accordance with the movement of the robot arm 12, may collide with an object, causing the mounting position of at least one of the cameras 21 and 22 relative to the robot 1 (robot arm 12) to shift.
- the calibration unit 312 may perform the accuracy information display process shown in FIG. 8.
- the calibration unit 312 may display, as the calibration accuracy information, information different from at least one of the above-mentioned calibration index value and calibration error on the display device 35.
- the calibration unit 312 may display information for notifying the user whether or not the calibration process was successful on the display device 35 as calibration accuracy information.
- the calibration unit 312 may display a text message for notifying the user whether or not the calibration process was successful on the display device 35 as calibration accuracy information.
- the calibration unit 312 may display on the display device 35 information for notifying the user whether or not the calibration process performed in steps S222 to S223 of FIG. 8 described above was successful. Specifically, if the calibration error exceeds the second tolerance threshold described above, the calibration unit 312 may determine that the calibration process has not been successful. In this case, the calibration unit 312 may display on the display device 35 information for notifying the user that the calibration process has not been successful. On the other hand, if the calibration error does not exceed the second tolerance threshold described above, the calibration unit 312 may determine that the calibration process has been successful. In this case, the calibration unit 312 may display on the display device 35 information for notifying the user that the calibration process has been successful.
- the calibration unit 312 may display, as calibration accuracy information, on the display device 35, information for prompting the user to perform the calibration process again.
- the calibration unit 312 may display, on the display device 35, a text message for prompting the user to perform the calibration process again.
- the calibration unit 312 may display, as calibration accuracy information, on the display device 35, information for notifying the user that the final calibration process may not be successful even if the calibration process is repeated as is.
- the calibration unit 312 may display, on the display device 35, a text message for notifying the user that the final calibration process may not be successful even if the calibration process is repeated as is.
- the calibration unit 312 may display, as calibration accuracy information, on the display device 35, an image indicated by the calibration image data IMG_calib that is the cause of the unsuccessful calibration process.
- the calibration unit 312 may display on the display device 35 an image indicated by at least one calibration image data IMG_calib included in the first calibration image data group, which is a small number of samples.
- the calibration unit 312 may display on the display device 35, as calibration accuracy information, information that prompts the user to remove the calibration image data IMG_calib that is causing the calibration process to be unsuccessful.
- the calibration unit 312 may display on the display device 35 a text message that prompts the user to remove the calibration image data IMG_calib that is causing the calibration process to be unsuccessful.
- the user may use the input device 34 to input an instruction to the control device 3 requesting reacquisition of the above-mentioned N pieces of calibration image data IMG_calib, exclusion of calibration image data IMG_calib whose quality does not reach the acceptable quality, or replacement of calibration image data IMG_calib whose quality does not reach the acceptable quality, before the accuracy information display process is completed.
- the calibration unit 312 may reacquire the above-mentioned N pieces of calibration image data IMG_calib, exclusion of calibration image data IMG_calib whose quality does not reach the acceptable quality, or replacement of calibration image data IMG_calib whose quality does not reach the acceptable quality, before the accuracy information display process is completed.
- the calibration unit 312 may reacquire the above-mentioned N pieces of calibration image data IMG_calib, exclusion of calibration image data IMG_calib whose quality does not reach the acceptable quality, or replacement of calibration image data IMG_calib whose quality does not reach the acceptable quality, regardless of whether or not a user instruction is given, before the accuracy information display process is completed.
- the calibration unit 312 may display on the display device 35 information for notifying the user whether the series of calibration processes performed in steps S222 and S223 in FIG. 8 described above were successful or not. Specifically, if the calibration index value exceeds the first tolerance threshold described above, the calibration unit 312 may display on the display device 35 information for notifying the user that the series of calibration processes were not successful. On the other hand, if the calibration index value does not exceed the first tolerance threshold described above (and further, if the calibration error does not exceed the second tolerance threshold), the calibration unit 312 may display on the display device 35 information for notifying the user that the series of calibration processes were successful.
- the processing position and orientation setting unit 313 may display a processing position and orientation setting screen 3530 for setting the processing position and orientation information PAI on the display device 35.
- the display signal generation unit 317 may generate a display signal for displaying the processing position and orientation setting screen 3530 on the display device 35 under the control of the processing position and orientation setting unit 313.
- the display signal output unit 318 may output the display signal generated by the display signal generation unit 317 to the display device 35.
- the display device 35 may display the processing position and orientation setting screen 3530 based on the display signal acquired from the display signal output unit 318.
- FIG. 11 An example of the processing position and orientation setting screen 3530 is shown in FIG. 11.
- the processing position and orientation setting screen 3530 is a screen on which an end effector model EM and an object model OM are arranged.
- a three-dimensional space SP3 that can be used as a virtual space is virtually constructed within the processing position and orientation setting screen 3530.
- the end effector model EM and the object model OM are arranged in this three-dimensional space SP3.
- a position within the three-dimensional space SP3 may be represented using a position and orientation setting coordinate system, which is a three-dimensional coordinate system used to indicate a position within the three-dimensional space SP3.
- the position and orientation setting coordinate system may be a three-dimensional coordinate system defined using an X-axis (SP3), a Y-axis (SP3) perpendicular to the X-axis (SP3), and a Z-axis (SP3) perpendicular to the X-axis (SP3) and Y-axis (SP3).
- the X-axis (SP3) may be set to be an axis corresponding to the X-axis of the global coordinate system.
- the X-axis (SP3) may be set to be an axis corresponding to the X-axis of the robot coordinate system.
- the X-axis (SP3) may be an axis set independently of the X-axis of the global coordinate system and the robot coordinate system.
- the Y-axis (SP3) may be set to be an axis corresponding to the X-axis of the global coordinate system.
- the Y-axis (SP3) may be set to be an axis corresponding to the Y-axis of the robot coordinate system.
- the Y-axis (SP3) may be an axis set independently of the Y-axis of the global coordinate system and the robot coordinate system.
- the Z-axis (SP3) may be set to be an axis corresponding to the Z-axis of the global coordinate system.
- the Z-axis (SP3) may be set to be an axis corresponding to the Z-axis of the robot coordinate system.
- the Z-axis (SP3) may be an axis set independently of the Z-axis of the global coordinate system and the robot coordinate system.
- a transformation matrix for converting three-dimensional coordinates in either the global coordinate system (or the robot coordinate system, the same applies hereinafter in this paragraph) or the position and orientation setting coordinate system into three-dimensional coordinates in the other of the global coordinate system and the position and orientation setting coordinate system may be calculated in advance.
- the transformation matrix for converting three-dimensional coordinates in either the global coordinate system or the position and orientation setting coordinate system into three-dimensional coordinates in the other of the global coordinate system and the position and orientation setting coordinate system may be information known to the processing position and orientation setting unit 313.
- the transformation matrix may include a translation transformation matrix that translates the three-dimensional coordinates in either the global coordinate system or the position and orientation setting coordinate system.
- the transformation matrix may include a translation transformation matrix that converts the position of an object in either the global coordinate system or the position and orientation setting coordinate system into the position of the object in the other of the global coordinate system and the position and orientation setting coordinate system.
- the transformation matrix may include a rotation transformation matrix that rotates the three-dimensional coordinates in either the global coordinate system or the position and orientation setting coordinate system.
- the transformation matrix may include a rotation transformation matrix that transforms the orientation of an object in either the global coordinate system or the position and orientation setting coordinate system into the orientation of the object in the other of the global coordinate system or the position and orientation setting coordinate system.
- the processing position and orientation setting unit 313 can use the transformation matrix to transform three-dimensional coordinates in either the global coordinate system or the position and orientation setting coordinate system into three-dimensional coordinates in the other of the global coordinate system or the position and orientation setting coordinate system.
- the end effector model EM is a three-dimensional model of the end effector 4 that is attached to the robot arm 12. Note that the end effector model EM has already been explained, so a detailed explanation will be omitted.
- the object model OM is a three-dimensional model of the object OBJ, which is the target object on which the end effector 4 performs a specified process. Note that the object model OM has already been explained, so a detailed explanation will be omitted.
- end effector model information indicating the end effector model EM and object model information indicating the object model OM may be input to the control device 3.
- a user may input the end effector model information and object model information to the control device 3 using the input device 34.
- the processing position and orientation setting unit 313 may read the end effector model information and the object model information from a recording medium on which the end effector model information and the object model information are recorded.
- the processing position and orientation setting unit 313 may download the end effector model information and the object model information from an external server on which the end effector model information and the object model information are recorded.
- the processing position and orientation setting unit 313 may set the processing position and orientation information PAI based on the positional relationship between the object model OM and the end effector model EM in the three-dimensional space SP3. Specifically, the user uses the input device 34 to input an instruction to the control device 3 to move at least one of the object model OM and the end effector model EM within the processing position and orientation setting screen 3530 (i.e., within the three-dimensional space SP3 within the processing position and orientation setting screen 3530).
- the user uses the input device 34 to input an instruction to the control device 3 to change the relative positional relationship between the object model OM and the end effector model EM within the processing position and orientation setting screen 3530 (i.e., within the three-dimensional space SP3 within the processing position and orientation setting screen 3530).
- the position and orientation setting coordinate system used to indicate a position in the three-dimensional space SP3 may be a coordinate system based on the object model OM.
- the position and orientation setting coordinate system used to indicate a position in the three-dimensional space SP3 may be a coordinate system whose origin is located on the surface or inside the object model OM.
- the position and orientation setting coordinate system used to indicate a position within the three-dimensional space SP3 may be a coordinate system based on the end effector model EM.
- the position and orientation setting coordinate system used to indicate a position within the three-dimensional space SP3 may be a coordinate system whose origin is located on the surface or inside of the end effector model EM.
- FIG. 11 shows an example in which a coordinate system based on the object model OM is used as the position and orientation setting coordinate system.
- the origin of the position and orientation setting coordinate system i.e., the intersection of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3)
- SP3 the origin of the position and orientation setting coordinate system
- the origin of the position and orientation setting coordinate system may be set to a position based on the object model OM (for example, a position on the surface or inside of the end effector model EM).
- FIGS. 12 to 15 which will be described later.
- the processing position and orientation setting unit 313 updates the processing position and orientation setting screen 3530 so that at least one of the object model OM and the end effector model EM moves within the processing position and orientation setting screen 3530 (i.e., within the three-dimensional space SP3) in accordance with the user's instruction.
- the display signal generation unit 317 generates a display signal for displaying on the display device 35 the processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM moves in accordance with the user's instruction.
- the display signal output unit 318 outputs the display signal generated by the display signal generation unit 317 to the display device 35.
- the display device 35 displays the processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM moves in accordance with the user's instruction based on the display signal acquired from the display signal output unit 318.
- the user may move at least one of the object model OM and the end effector model EM so that the positional relationship between the object model OM and the end effector model EM becomes the same as the ideal positional relationship between the object OBJ and the end effector 4 that should be realized when the end effector 4 performs a predetermined process on the object OBJ.
- the user may move at least one of the object model OM and the end effector model EM so that the positional relationship between the object model OM and the end effector model EM becomes the same as the ideal positional relationship between the object OBJ and the end effector 4 that the user desires when the end effector 4 performs a predetermined process on the object OBJ.
- moving at least one of the object model OM and the end effector model EM may be considered equivalent to aligning the object model OM and the end effector model EM.
- the processing position and orientation setting screen 3530 displays an object model OM indicating the three-dimensional shape of the workpiece W, and an end effector model EM.
- the user may move at least one of the object model OM and the end effector model EM so that the positional relationship between the object model OM and the end effector model EM is the same as the ideal positional relationship between the workpiece W and the end effector 4 that should be realized when the end effector 4 performs a holding process on the workpiece W.
- the process position and orientation setting screen 3530 displays the object model OM indicating the three-dimensional shape of the mounting device T from which the workpiece W is to be released, and the end effector model EM.
- the release process is performed, a three-dimensional model obtained by adding a three-dimensional model of the workpiece W held by the end effector 4 to the end effector model EM may be used as the end effector model EM.
- the user may move at least one of the object model OM and the end effector model EM so that the positional relationship between the object model OM and the end effector model EM is the same as the ideal positional relationship between the mounting device T and the end effector 4 that should be realized when the end effector 4 performs a release process to release the workpiece W to the mounting device T.
- the user may move at least one of the object model OM and the end effector model EM so that the posture relationship between the object model OM and the end effector model EM becomes the same as the ideal posture relationship between the object OBJ and the end effector 4 that should be realized when the end effector 4 performs a specified process on the object OBJ.
- the user may move at least one of the object model OM and the end effector model EM so that the posture relationship between the object model OM and the end effector model EM becomes the same as the ideal posture relationship between the object OBJ and the end effector 4 that the user desires when the end effector 4 performs a specified process on the object OBJ.
- moving at least one of the object model OM and the end effector model EM may be considered equivalent to aligning (aligning the posture) the object model OM and the end effector model EM.
- the processing position and orientation setting screen 3530 displays an object model OM showing the three-dimensional shape of the workpiece W, and an end effector model EM.
- the user may move at least one of the object model OM and the end effector model EM so that the orientation relationship between the object model OM and the end effector model EM becomes the same as the ideal orientation relationship between the workpiece W and the end effector 4 that should be realized when the end effector 4 performs a holding process on the workpiece W.
- the process position and orientation setting screen 3530 displays the object model OM indicating the three-dimensional shape of the mounting device T from which the workpiece W is to be released, and the end effector model EM.
- the release process is performed, a three-dimensional model obtained by adding a three-dimensional model of the workpiece W held by the end effector 4 to the end effector model EM may be used as the end effector model EM.
- the user may move at least one of the object model OM and the end effector model EM so that the orientation relationship between the object model OM and the end effector model EM is the same as the ideal orientation relationship between the mounting device T and the end effector 4 that should be realized when the end effector 4 performs a release process to release the workpiece W to the mounting device T.
- the processing position and orientation setting unit 313 may set the processing position and orientation information PAI based on at least one of the positional relationship and the orientation relationship between the object model OM and the end effector model EM changed by the user.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI that includes information regarding the relative position of the end effector model EM with respect to the object model OM as information for calculating a processing target position PTP at which the end effector 4 performs a predetermined processing with respect to the object OBJ.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI that includes information regarding the relative position of the end effector 4 with respect to the object OBJ as information for calculating the processing target position PTP.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI that includes information regarding the relative position of the end effector 4 with respect to the object OBJ as information for calculating the processing target position PTP.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI that includes information regarding the relative orientation of the end effector model EM with respect to the object model OM as information for calculating a processing target orientation PTA in which the end effector 4 performs a predetermined processing with respect to the object OBJ.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI that includes information regarding the relative orientation of the end effector 4 with respect to the object OBJ as information for calculating the processing target orientation PTA.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI that includes information regarding the relative orientation of the end effector 4 with respect to the object OBJ as a reference as information for calculating the processing target orientation PTA.
- the robot control unit 316 moves the robot arm 12 so that the tool center point TCP (or an arbitrary movement reference point MBP) moves to the processing target position PTP.
- the processing position and orientation setting unit 313 may calculate the position of the tool center point TCP from the end effector model EM moved by the user, and set the processing position and orientation information PAI including information on the relative position of the tool center point TCP with respect to the object model OM as information for calculating the processing target position PTP where the end effector 4 performs a predetermined processing on the object OBJ.
- the processing position and orientation setting unit 313 may calculate the position of an arbitrary movement reference point MBP different from the tool center point TCP from the end effector model EM moved by the user, and set the processing position and orientation information PAI including information on the relative position of the arbitrary movement reference point MBP with respect to the object model OM as information for calculating the processing target position PTP where the end effector 4 performs a predetermined processing on the object OBJ.
- the processing position and orientation setting unit 313 may display the processing position and orientation setting screen 3530 including the tool center point TCP determined based on the end effector model EM on the display device 35 in addition to or instead of displaying the processing position and orientation setting screen 3530 including the end effector model EM on the display device 35.
- the user may input an instruction to move the tool center point TCP in addition to or instead of inputting an instruction to move the end effector model EM.
- the processing position and orientation setting unit 313 may update the processing position and orientation setting screen 3530 so that the tool center point TCP moves within the processing position and orientation setting screen 3530 (i.e., within the three-dimensional space SP3) in accordance with the user's instruction.
- the processing position and orientation setting unit 313 may set the processing position and orientation information PAI based on at least one of the positional relationship and the orientation relationship between the object model OM and the tool center point TCP changed by the user.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI that includes information regarding the relative position of the tool center point TCP moved by the user as information for calculating the processing target position PTP.
- the processing target position PTP may include at least one of the position of the end effector 4 along the X-axis of the global coordinate system, the position of the end effector 4 along the Y-axis of the global coordinate system, and the position of the end effector 4 along the Z-axis of the global coordinate system.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI including information regarding at least one of the position of the end effector model EM along the X-axis (SP3), the position of the end effector model EM along the Y-axis (SP3), and the position of the end effector model EM along the Z-axis (SP3) based on the positional relationship between the object model OM and the end effector model EM as information for calculating the processing target position PTP.
- processing position and orientation information PAI including information regarding at least one of the position of the end effector model EM along the X-axis (SP3), the position of the end effector model EM along the Y-axis (SP3), and the position of the end effector model EM along the Z-axis (SP3) based on the positional relationship between the object model OM and the end effector model EM as information for calculating the processing target position PTP.
- the robot control unit 316 may calculate at least one of the positions of the end effector 4 along the X axis of the global coordinate system, the Y axis of the global coordinate system, and the Z axis of the global coordinate system as the processing target position PTP based on at least one of the calculation results of the position and orientation of the object OBJ in the global coordinate system based on at least one of the image data IMG_2D and IMG_3D and the processing position and orientation information PAI. After that, the robot control unit 316 may control the robot 1 so that the end effector 4 moves toward the calculated processing target position PTP.
- the processing target orientation PTA may include at least one of the orientation of the end effector 4 in the rotation direction about the X-axis of the global coordinate system, the orientation of the end effector 4 in the rotation direction about the Y-axis of the global coordinate system, and the orientation of the end effector 4 in the rotation direction about the Z-axis of the global coordinate system.
- the processing position and orientation setting unit 313 may set processing position and orientation information PAI including information regarding at least one of the orientation of the end effector model EM in the rotation direction about the X-axis (SP3), the orientation of the end effector model EM in the rotation direction about the Y-axis (SP3), and the orientation of the end effector model EM in the rotation direction about the Z-axis (SP3) based on the orientation relationship between the object model OM and the end effector model EM as information for calculating the processing target position PTP.
- processing position and orientation information PAI including information regarding at least one of the orientation of the end effector model EM in the rotation direction about the X-axis (SP3), the orientation of the end effector model EM in the rotation direction about the Y-axis (SP3), and the orientation of the end effector model EM in the rotation direction about the Z-axis (SP3) based on the orientation relationship between the object model OM and the end effector model EM as information for calculating the processing target position PTP.
- the robot control unit 316 may calculate at least one of the orientation of the end effector 4 in the rotation direction about the X axis of the global coordinate system, the orientation of the end effector 4 in the rotation direction about the Y axis of the global coordinate system, and the orientation of the end effector 4 in the rotation direction about the Z axis of the global coordinate system (i.e., the orientation of the end effector 4) as the orientation to be processed PTA based on at least one of the calculation results of the position and orientation of the object OBJ in the global coordinate system based on at least one of the image data IMG_2D and IMG_3D and the processing position and orientation information PAI.
- the robot control unit 316 may control the robot 1 so that the end effector 4 takes the calculated orientation to be processed PTA.
- the user may move (translate) at least one of the object model OM and the end effector model EM along at least one of the three axes that form the position and orientation setting coordinate system, which is a three-dimensional coordinate system used to indicate a position within the three-dimensional space SP3.
- the user may move at least one of the object model OM and the end effector model EM along at least one of the X axis (SP3), Y axis (SP3), and Z axis (SP3).
- the user may move at least one of the object model OM and the end effector model EM along any two of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3).
- moving at least one of the object model OM and the end effector model EM along any two of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3) may be considered equivalent to moving at least one of the object model OM and the end effector model EM along a tilt direction that is tilted toward any two of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3).
- FIG. 13 moving at least one of the object model OM and the end effector model EM along any two of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3).
- the user moves the end effector model EM along both the X-axis (SP3) and Y-axis (SP3). That is, in the example shown in Fig. 13, the user moves the end effector model EM along a tilt direction that tilts toward both the X-axis (SP3) and the Y-axis (SP3).
- the X-axis (SP3), the Y-axis (SP3), and the Z-axis (SP3) are written as "X", "Y", and "Z", respectively.
- the user may move at least one of the object model OM and the end effector model EM along all of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3).
- moving at least one of the object model OM and the end effector model EM along all of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3) may be considered equivalent to moving at least one of the object model OM and the end effector model EM along a tilt direction that is tilted along all of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3).
- the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3) are represented as "X", "Y", and "Z", respectively, for the sake of simplicity.
- the user may move at least one of the object model OM and the end effector model EM along any one of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3).
- the user also moves the end effector model EM along the Y-axis (SP3).
- the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3) are written as "X,” "Y,” and "Z,” respectively.
- the user can reduce the time required to align the object model OM and the end effector model EM, compared to when a user performs uniaxial movement, moving at least one of the object model OM and the end effector model EM along one of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3).
- the user needs to perform a single-axis movement to move at least one of the object model OM and the end effector model EM along one of the X-axis (SP3), the Y-axis (SP3), and the Z-axis (SP3), and then perform a single-axis movement to move at least one of the object model OM and the end effector model EM along the other one of the X-axis (SP3), the Y-axis (SP3), and the Z-axis (SP3).
- the user needs to perform a single-axis movement multiple times.
- the processing position and orientation setting unit 313 can set the processing position and orientation information PAI more quickly than when the user performs single-axis movement but does not perform multi-axis movement.
- the user may perform a multi-axis movement and then a single-axis movement. Specifically, the user may first perform a multi-axis movement to roughly change the positional relationship between the object model OM and the end effector model EM. That is, the user may input an instruction to the control device 3 to perform a multi-axis movement so as to roughly change the positional relationship between the object model OM and the end effector model EM.
- the processing position and orientation setting unit 313 may display a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM moves along at least two of the X axis (SP3), the Y axis (SP3), and the Z axis (SP3).
- the display signal generating unit 317 may generate a display signal for displaying a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM moves along at least two of the X axis (SP3), the Y axis (SP3), and the Z axis (SP3). Furthermore, the display device 35 may display a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM moves along at least two of the X-axis (SP3), the Y-axis (SP3), and the Z-axis (SP3) based on the display signal.
- the user may change the positional relationship between the object model OM and the end effector model EM with high accuracy by performing one-axis movement.
- an instruction to perform one-axis movement so as to change the positional relationship between the object model OM and the end effector model EM with high accuracy may be input to the control device 3.
- the processing position and orientation setting unit 313 may display a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM moves along any one of the X-axis (SP3), the Y-axis (SP3), and the Z-axis (SP3).
- the display signal generating unit 317 may generate a display signal for displaying a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM moves along any one of the X axis (SP3), Y axis (SP3), and Z axis (SP3). Furthermore, the display device 35 may display a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM moves along any one of the X axis (SP3), Y axis (SP3), and Z axis (SP3) based on the display signal.
- the amount of movement (translation amount) of at least one of the object model OM and the end effector model EM by the single-axis movement may be less than the amount of movement (translation amount) of at least one of the object model OM and the end effector model EM by the multi-axis movement.
- the user may perform a single-axis movement after performing a multi-axis movement such that the amount of movement (translation amount) of at least one of the object model OM and the end effector model EM by the single-axis movement is less than the amount of movement (translation amount) of at least one of the object model OM and the end effector model EM by the multi-axis movement.
- the user can roughly change the positional relationship between the object model OM and the end effector model EM by performing a multi-axis movement, and then change the positional relationship between the object model OM and the end effector model EM with high precision by performing a single-axis movement.
- the processing position and orientation setting unit 313 can set the processing position and orientation information PAI quickly, while setting the processing position and orientation information PAI that can be used to calculate at least one of the processing target position PTP and the processing target orientation PTA with high accuracy.
- the user may input an instruction to specify the position of at least one of the object model OM and the end effector model EM in addition to or instead of performing a single-axis movement.
- the user may input an instruction to specify the position where at least one of the object model OM and the end effector model EM should be located if a single-axis movement were performed.
- the user may rotate (rotate and move) at least one of the object model OM and the end effector model EM around a rotation axis along at least one of the three axes that form the position and orientation setting coordinate system.
- the user may rotate at least one of the object model OM and the end effector model EM around a rotation axis along at least one of the X axis (SP3), Y axis (SP3), and Z axis (SP3).
- the user may rotate at least one of the object model OM and the end effector model EM around two rotation axes along any two of the X axis (SP3), Y axis (SP3), and Z axis (SP3).
- rotating at least one of the object model OM and the end effector model EM around two rotation axes along any two of the X axis (SP3), Y axis (SP3), and Z axis (SP3) may be considered equivalent to rotating at least one of the object model OM and the end effector model EM around a rotation axis along a tilt direction that tilts toward any two of the X axis (SP3), Y axis (SP3), and Z axis (SP3).
- the user may rotate at least one of the object model OM and the end effector model EM around three rotation axes that are aligned along all of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3).
- rotating at least one of the object model OM and the end effector model EM around three rotation axes that are aligned along all of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3) may be considered equivalent to rotating at least one of the object model OM and the end effector model EM around a rotation axis that is aligned along an inclination direction that is inclined along all of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3).
- the user may rotate at least one of the object model OM and the end effector model EM around two rotation axes along any one of the X axis (SP3), Y axis (SP3), and Z axis (SP3).
- the user can reduce the time required to align (align) the object model OM and the end effector model EM compared to when a user performs single-axis movement in which at least one of the object model OM and the end effector model EM is rotated around a rotation axis along one of the X-axis (SP3), Y-axis (SP3), and Z-axis (SP3).
- the processing position and orientation setting unit 313 can set the processing position and orientation information PAI more quickly than when the user performs single-axis movement but does not perform multi-axis movement.
- the user may perform a multi-axis movement and then a single-axis movement. Specifically, the user may first perform a multi-axis movement to roughly change the attitude relationship between the object model OM and the end effector model EM. That is, the user may input an instruction to the control device 3 to perform a multi-axis movement so as to roughly change the attitude relationship between the object model OM and the end effector model EM.
- the processing position and orientation setting unit 313 may display a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM rotates around at least two rotation axes along at least two of the X axis (SP3), the Y axis (SP3), and the Z axis (SP3). That is, the display signal generating unit 317 may generate a display signal for displaying a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM rotates around at least two rotation axes along at least two of the X axis (SP3), the Y axis (SP3), and the Z axis (SP3).
- the display device 35 may display a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM rotates around at least two rotation axes along at least two of the X axis (SP3), the Y axis (SP3), and the Z axis (SP3) based on the display signal.
- the user may change the orientation relationship between the object model OM and the end effector model EM with high accuracy by performing one-axis movement.
- an instruction to perform one-axis movement so as to change the orientation relationship between the object model OM and the end effector model EM with high accuracy may be input to the control device 3.
- the processing position and orientation setting unit 313 may display a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM becomes vacant around a rotation axis along any one of the X axis (SP3), the Y axis (SP3), and the Z axis (SP3). That is, the display signal generating unit 317 may generate a display signal for displaying a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM rotates around a rotation axis along any one of the X axis (SP3), Y axis (SP3), and Z axis (SP3).
- the display device 35 may display a processing position and orientation setting screen 3530 in which at least one of the object model OM and the end effector model EM rotates around a rotation axis along any one of the X axis (SP3), Y axis (SP3), and Z axis (SP3) based on the display signal.
- SP3 X axis
- SP3 Y axis
- SP3 Z axis
- the amount of movement (rotational movement) of at least one of the object model OM and the end effector model EM by the single-axis movement may be less than the amount of movement (rotational movement) of at least one of the object model OM and the end effector model EM by the multi-axis movement.
- the user may perform a single-axis movement after performing a multi-axis movement such that the amount of movement (rotational movement) of at least one of the object model OM and the end effector model EM by the single-axis movement is less than the amount of movement (rotational movement) of at least one of the object model OM and the end effector model EM by the multi-axis movement.
- the user can roughly change the attitude relationship between the object model OM and the end effector model EM by performing a multi-axis movement, and then change the attitude relationship between the object model OM and the end effector model EM with high precision by performing a single-axis movement.
- the processing position and orientation setting unit 313 can set the processing position and orientation information PAI quickly, while setting the processing position and orientation information PAI that can be used to calculate at least one of the processing target position PTP and the processing target orientation PTA with high accuracy.
- the user may input an instruction to specify the posture of at least one of the object model OM and the end effector model EM in addition to or instead of performing a single-axis movement.
- the user may input an instruction to specify the posture that at least one of the object model OM and the end effector model EM should have if a single-axis movement had been performed.
- one end effector 4 may perform a predetermined process on one object OBJ
- the relative attitude relationship between one end effector 4 and one object OBJ is a first attitude relationship
- one end effector 4 may perform a predetermined process on one object OBJ
- the processing position and attitude setting unit 313 may set multiple processing position and attitude information PAI that can each be used to calculate multiple different processing target attitudes PTA for a combination of one object OBJ and one end effector 4.
- the processing position and orientation information PAI indicates at least one of the position of the end effector model EM relative to the object model OM and the orientation of the end effector model EM relative to the object model OM.
- the position of the end effector model EM relative to the object model OM indicated by the processing position and orientation information PAI is referred to as the set position STP
- the orientation of the end effector model EM relative to the object model OM indicated by the processing position and orientation information PAI is referred to as the set orientation STA.
- the user may move at least one of the object model OM and the end effector model EM so that the positional relationship between the object model OM and the end effector model EM becomes the same as an ideal first positional relationship between the object OBJ and the end effector 4 that should be realized when the end effector 4 performs a predetermined process on the object OBJ.
- the processing position and orientation setting unit 313 may set the processing position and orientation information PAI including information on the first set position STP#1 indicated by the positional relationship between the object model OM and the end effector model EM (i.e., the first positional relationship) as the first processing position and orientation information PAI that can be used to calculate the first processing target position PTP.
- the user may move at least one of the object model OM and the end effector model EM so that the attitude relationship between the object model OM and the end effector model EM becomes the same as an ideal first attitude relationship between the object OBJ and the end effector 4 that should be realized when the end effector 4 performs a predetermined process on the object OBJ.
- the processing position and attitude setting unit 313 may set the processing position and attitude information PAI including information on the first set attitude STA#1 indicated by the attitude relationship between the object model OM and the end effector model EM (i.e., the first attitude relationship) as the first processing position and attitude information PAI that can be used to calculate the first processing target attitude PTA.
- the user may move at least one of the object model OM and the end effector model EM so that the positional relationship between the object model OM and the end effector model EM becomes the same as an ideal second positional relationship between the object OBJ and the end effector 4 that should be realized when the end effector 4 performs a predetermined process on the object OBJ.
- the processing position and orientation setting unit 313 may set the processing position and orientation information PAI including information on the second set position STP#2 indicated by the positional relationship between the object model OM and the end effector model EM (i.e., the second positional relationship) as the second processing position and orientation information PAI that can be used to calculate the second processing target position PTP.
- the user may move at least one of the object model OM and the end effector model EM so that the attitude relationship between the object model OM and the end effector model EM becomes the same as an ideal second attitude relationship between the object OBJ and the end effector 4 that should be realized when the end effector 4 performs a predetermined process on the object OBJ.
- the processing position and attitude setting unit 313 may set the processing position and attitude information PAI including information on the second set attitude STA#2 indicated by the attitude relationship between the object model OM and the end effector model EM (i.e., the second attitude relationship) as the second processing position and attitude information PAI that can be used to calculate the second processing target attitude PTA.
- the processing position and orientation setting unit 313 may set another processing position and orientation information PAI to determine whether or not the end effector 4 can perform a specified processing on the object OBJ when the relative positional relationship between the end effector 4 and the object OBJ is in a second positional relationship when interference occurs between the end effector 4 (or the robot arm 12) and the object OBJ (as a result, the end effector 4 cannot perform a specified processing on the object OBJ) when the relative positional relationship between the end effector 4 and the object OBJ is in a first positional relationship.
- the processing position and orientation setting unit 313 may set another processing position and orientation information PAI in order to determine whether or not the end effector 4 can perform a predetermined process on the object OBJ in a situation where the relative orientation between the end effector 4 and the object OBJ is in a second orientation when interference between the end effector 4 (or the robot arm 12) and the object OBJ occurs under a situation where the relative orientation between the end effector 4 and the object OBJ is in a first orientation (as a result, the end effector 4 cannot perform a predetermined process on the object OBJ).
- the processing position and orientation setting unit 313 may set the second processing position and orientation information PAI in order to determine whether or not the end effector 4 can perform a predetermined process on the object OBJ using the second processing position and orientation information PAI in a situation where interference between the end effector 4 (or the robot arm 12) and the object OBJ occurs when the first processing position and orientation information PAI is used (as a result, the end effector 4 cannot perform a predetermined process on the object OBJ).
- the processing position and orientation setting unit 313 can set multiple pieces of processing position and orientation information PAI relatively easily and quickly. This effect becomes more pronounced as the number of processing position and orientation information PAI set for the combination of the same object OBJ and the same end effector 4 increases.
- the robot 1 may perform a predetermined first processing on the object OBJ using a first end effector 4#1 attached to the robot arm 12, and may also perform a predetermined second processing on the object OBJ using a second end effector 4#2 attached to the robot arm 12.
- the user may move at least one of the first end effector model EM#1, which is a three-dimensional model of the first end effector 4#1, and the object model OM, which is a three-dimensional model of the object OBJ, on the processing position and orientation setting screen 3530.
- the user may move the first end effector model EM#1 to the first setting position STP#11.
- the processing position and orientation setting unit 313 may set the first processing position and orientation information PAI that can be used to calculate the first processing target position PTP#11 where the first end effector 4#1 performs a predetermined process on the object OBJ, based on the first setting position STP#11 indicated by the positional relationship between the first end effector model EM#1 and the object model OM.
- the user in order to set the second processing position and orientation information PAI that can be used to calculate the second processing target position PTP#12 at which the second end effector 4#2 performs a predetermined processing (second processing) on the object OBJ, the user does not need to move at least one of the second end effector model EM#2, which is a three-dimensional model of the second end effector 4#2, and the object model OM, on the processing position and orientation setting screen 3530. In other words, the user does not need to input an instruction to the control device 3 to move at least one of the second end effector model EM#2 and the object model OM.
- the processing position and orientation setting unit 313 may set second processing position and orientation information PAI that can be used to calculate a second processing target position PTP#12 where the second end effector 4#2 performs a predetermined process on the object OBJ, based on end effector shape difference information relating to the difference between the shape of the first end effector model EM#1 and the shape of the second end effector model EM#2 and the first set position STP#11 that has already been set.
- the left side of FIG. 17 shows the first set position STP#11 set based on a user's instruction, together with the positional relationship between the first end effector model EM#1 and the object model OM.
- the positional relationship between the first end effector model EM#1 and the object model OM is determined by the first set position STP#11.
- the processing position and orientation setting unit 313 may replace the first end effector model EM#1 aligned with the object model OM based on the set first set position STP#11 with the second end effector model EM#2 in the three-dimensional space SP3, as shown in the center of FIG. 17.
- the processing position and orientation setting unit 313 may replace the first end effector model EM#1 with the second end effector model EM#2 so that the corresponding predetermined part EBP of the second end effector model EM#2 is located at the same position as the predetermined part EBP of the first end effector model EM#1 (as an example, the connection point with the robot arm 12).
- the processing position and orientation setting unit 313 may move the second end effector model EM#2 in the three-dimensional space SP3 based on the end effector shape difference information.
- the processing position and orientation setting unit 313 may change the positional relationship between the object model OM and the second end effector model EM#2 in the three-dimensional space SP3 based on the end effector shape difference information.
- the user does not need to input an instruction to change the positional relationship between the object model OM and the second end effector model EM#2 to the control device 3. Specifically, as shown on the right side of FIG.
- the processing position and orientation setting unit 313 may move the second end effector model EM#2 by a movement amount ⁇ ED determined based on the end effector shape difference information.
- the processing position and orientation setting unit 313 may move the object model OM by an amount determined based on the end effector shape difference information. This is because when the object model OM moves, the positional relationship between the object model OM and the second end effector model EM#2 is changed, just as when the second end effector model EM#2 moves.
- the second end effector model EM#2 is moved will be explained.
- the processing position and orientation setting unit 313 performs internal calculations to change the positional relationship between the object model OM and the second end effector model EM#2 in the three-dimensional space SP3 based on the end effector shape difference information, but does not need to update the positional relationship between the object model OM and the second end effector model EM#2 in the processing position and orientation setting screen 3530. In other words, the processing position and orientation setting unit 313 does not need to move at least one of the object model OM and the second end effector model EM#2 on the processing position and orientation setting screen 3530.
- the end effector shape difference information may be input by a user using the input device 34.
- the end effector shape difference information may be generated by the processing position and orientation setting unit 313 based on information input by a user using the input device 34 to generate the end effector shape difference information.
- the end effector shape difference information may be generated by the processing position and orientation setting unit 313 based on the first end effector model EM#1 and the second end effector model EM#2.
- the end effector shape difference information may include information on the difference ⁇ EL between the overall length of the first end effector model EM#1 and the overall length of the second end effector model EM#2, as shown in FIG. 17.
- the processing position and orientation setting unit 313 may calculate the movement amount ⁇ ED of the second end effector model EM#2 described above based on the difference ⁇ EL between the overall length of the first end effector model EM#1 and the overall length of the second end effector model EM#2.
- the processing position and orientation setting unit 313 may set the difference ⁇ EL between the overall length of the first end effector model EM#1 and the overall length of the second end effector model EM#2 itself as the movement amount ⁇ ED of the second end effector model EM#2 described above.
- the user may input information regarding the difference ⁇ EL between the overall length of the first end effector model EM#1 and the overall length of the second end effector model EM#2 to the control device 3 using the input device 34.
- the user may input information regarding the overall length of the first end effector model EM#1 and the overall length of the second end effector model EM#2 to the control device 3 using the input device 34.
- the processing position and orientation setting unit 313 may calculate the difference ⁇ EL between the overall length of the first end effector model EM#1 and the overall length of the second end effector model EM#2 based on the information regarding the overall length of the first end effector model EM#1 and the overall length of the second end effector model EM#2 input by the user.
- the processing position and orientation setting unit 313 may calculate the difference ⁇ EL between the overall length of the first end effector model EM#1 and the overall length of the second end effector model EM#2 based on the first end effector model EM#1 and the second end effector model EM#2.
- the end effector shape difference information may include information regarding the difference between the width of the first end effector model EM#1 and the width of the second end effector model EM#2.
- the processing position and orientation setting unit 313 may calculate the movement amount ⁇ ED of the second end effector model EM#2 described above based on the difference between the width of the first end effector model EM#1 and the width of the second end effector model EM#2.
- the processing position and orientation setting unit 313 may set the difference between the width of the first end effector model EM#1 and the width of the second end effector model EM#2 itself as the movement amount ⁇ ED of the second end effector model EM#2 described above.
- the user may input information regarding the difference between the width of the first end effector model EM#1 and the width of the second end effector model EM#2 to the control device 3 using the input device 34.
- the user may input information regarding the width of the first end effector model EM#1 and the width of the second end effector model EM#2 to the control device 3 using the input device 34.
- the processing position and orientation setting unit 313 may calculate the difference between the width of the first end effector model EM#1 and the width of the second end effector model EM#2 based on the information regarding the width of the first end effector model EM#1 and the width of the second end effector model EM#2.
- the processing position and orientation setting unit 313 may calculate the difference between the width of the first end effector model EM#1 and the width of the second end effector model EM#2 based on the first end effector model EM#1 and the second end effector model EM#2.
- the end effector shape difference information may include information regarding the difference between the position of the tool center point TCP of the first end effector model EM#1 and the position of the tool center point TCP of the second end effector model EM#2.
- the processing position and orientation setting unit 313 may calculate the movement amount ⁇ ED of the second end effector model EM#2 described above based on the difference between the position of the tool center point TCP of the first end effector model EM#1 and the position of the tool center point TCP of the second end effector model EM#2. As an example, the processing position and orientation setting unit 313 may set the difference between the position of the tool center point TCP of the first end effector model EM#1 and the position of the tool center point TCP of the second end effector model EM#2 as the movement amount ⁇ ED of the second end effector model EM#2 described above.
- the user may input information regarding the difference between the position of the tool center point TCP of the first end effector model EM#1 and the position of the tool center point TCP of the second end effector model EM#2 to the control device 3 using the input device 34.
- the user may input information regarding the position of the tool center point TCP of the first end effector model EM#1 and the position of the tool center point TCP of the second end effector model EM#2 to the control device 3 using the input device 34.
- the processing position and orientation setting unit 313 may calculate the difference between the position of the tool center point TCP of the first end effector model EM#1 and the position of the tool center point TCP of the second end effector model EM#2 based on the information regarding the position of the tool center point TCP of the first end effector model EM#1 and the position of the tool center point TCP of the second end effector model EM#2.
- the processing position and orientation setting unit 313 may calculate the difference between the position of the tool center point TCP of the first end effector model EM#1 and the position of the tool center point TCP of the second end effector model EM#2 based on the first end effector model EM#1 and the second end effector model EM#2.
- the end effector shape difference information indicating the difference between the shape of the first end effector model EM#1 and the shape of the second end effector model EM#2 may be considered to be information regarding the difference between the shape of the first end effector 4#1 and the shape of the second end effector 4#2.
- the first end effector model EM#1 is a three-dimensional model indicating the three-dimensional shape of the first end effector 4#1
- the second end effector model EM#2 is a three-dimensional model indicating the three-dimensional shape of the second end effector 4#2.
- the processing position and orientation setting unit 313 may set second processing position and orientation information PAI that can be used to calculate the second processing target position PTP#12 based on the positional relationship between the object model OM and the second end effector model EM#2 in the three-dimensional space SP3.
- the processing position and orientation setting unit 313 may set, as the second processing position and orientation information PAI that can be used to calculate the second processing target position PTP#12, processing position and orientation information PAI that includes information indicating, as the second setting position STP#12, the position of the tool center point TCP indicated by the positional relationship between the object model OM and the second end effector model EM#2 changed based on the end effector shape difference information.
- the processing position and orientation setting unit 313 may set the processing position and orientation information PAI including information about the second set position STP#12 (see FIG. 17) indicated by the positional relationship between the object model OM changed based on the end effector shape difference information and the second end effector model EM#2 as the second processing position and orientation information PAI that can be used to calculate the second processing target position PTP#12.
- the user when the end effector 4 attached to the robot arm 12 is replaced, the user inputs an instruction to the control device 3 to change the positional relationship between the end effector model EM corresponding to any one of the multiple end effectors 4 attached to the robot arm 12 and the object model OM, while not having to input an instruction to the control device 3 to change the positional relationship between the end effector model EM corresponding to the remaining of the multiple end effectors 4 attached to the robot arm 12 and the object model OM.
- the time required to set multiple processing position and orientation information PAI corresponding to the multiple end effectors 4 is also reduced.
- This effect is particularly beneficial when multiple different processing position and orientation information PAI are set for the combination of the same object OBJ and the same end effector 4 as described in the first modified example.
- the more processing position and orientation information PAI that the processing position and orientation setting unit 313 needs to set the more beneficial this effect becomes.
- the end effector 4 may perform a predetermined processing on a plurality of different objects OBJ.
- the robot 1 may use the end effector 4 to perform a predetermined first processing on a first object OBJ#1 and a predetermined second processing on a second object OBJ#1.
- the user may move at least one of the end effector model EM, which is a three-dimensional model of the end effector 4, and the first object model OM#1, which is a three-dimensional model of the first object OBJ#1, on the processing position and orientation setting screen 3530.
- the user may move the end effector model EM to the first setting position STP#21.
- the processing position and orientation setting unit 313 may set a plurality of pieces of processing position and orientation information PAI that can each be used to calculate a first processing target position PTP#21 where the end effector 4 performs a predetermined process on the first object OBJ#1 based on the positional relationship between the end effector model EM and the first object model OM#1.
- the user in order to set the second processing position and orientation information PAI that can be used to calculate the second processing target position PTP#22 where the end effector 4 performs a predetermined process on the second object OBJ#2, the user does not need to move at least one of the end effector model EM and the second object model OM#2, which is a three-dimensional model of the second object OBJ#2, on the processing position and orientation setting screen 3530. In other words, the user does not need to input an instruction to move at least one of the end effector model EM and the second object model OM#2 to the control device 3.
- the processing position and orientation setting unit 313 may set the second processing position and orientation information PAI that can be used to calculate the second processing target position PTP#22 where the end effector 4 performs a predetermined process on the second object OBJ#2, based on the object shape difference information related to the difference between the shape of the first object model OM#1 and the shape of the second object model OM#2 and the first setting position STP#21.
- the left side of FIG. 18 shows the first set position STP#21 set based on a user's instruction, together with the positional relationship between the end effector model EM and the first object model OM#1.
- the positional relationship between the end effector model EM and the first object model OM#1 is determined by the first set position STP#21.
- the processing position and orientation setting unit 313 may replace the first object model OM#1 aligned with the end effector model EM based on the set first processing target position PTP#21 with the second object model OM#2 in the three-dimensional space SP3, as shown in the center of FIG. 18.
- the processing position and orientation setting unit 313 may replace the first object model OM#1 with the second object model OM#2 so that the corresponding specified part OBP of the second object model OM#2 is located at the same position as the specified part OBP of the first object model OM#1.
- the processing position and orientation setting unit 313 may move the second object model OM#2 in the three-dimensional space SP3 based on the object shape difference information. That is, the processing position and orientation setting unit 313 may change the positional relationship between the end effector model EM and the second object model OM#2 in the three-dimensional space SP3 based on the object shape difference information. In this case, the user does not need to input an instruction to change the positional relationship between the end effector model EM and the second object model OM#2 to the control device 3. Specifically, as shown on the right side of FIG. 18, the processing position and orientation setting unit 313 may move the second object model OM#2 by a movement amount ⁇ OD determined based on the object shape difference information.
- the processing position and orientation setting unit 313 may move the end effector model EM by a movement amount determined based on the object shape difference information in addition to or instead of moving the second object model OM#2. This is because when the end effector model EM moves, the positional relationship between the end effector model EM and the second object model OM#2 changes, just as when the second object model OM#2 moves. For ease of explanation, the following explanation will be given using an example in which the second object model OM#2 is moved.
- the processing position and orientation setting unit 313 performs internal calculations to change the positional relationship between the second object model OM#2 and the end effector model EM in the three-dimensional space SP3 based on the object shape difference information, but does not need to update the positional relationship between the second object model OM#2 and the end effector model EM in the processing position and orientation setting screen 3530. In other words, the processing position and orientation setting unit 313 does not need to move at least one of the second object model OM#2 and the end effector model EM on the processing position and orientation setting screen 3530.
- the object shape difference information may be input by the user using the input device 34.
- the object shape difference information may be generated by the processing position and orientation setting unit 313 based on information input by the user using the input device 34 to generate the object shape difference information.
- the object shape difference information may be generated by the processing position and orientation setting unit 313 based on the first object model OM#1 and the second object model OM#2.
- the object shape difference information may include information on the difference ⁇ OL between the overall length of the first object model OM#1 and the overall length of the second object model OM#2, as shown in FIG. 18.
- the processing position and orientation setting unit 313 may calculate the movement amount ⁇ ED of the second object model OM#2 described above based on the difference ⁇ OL between the overall length of the first object model OM#1 and the overall length of the second object model OM#2.
- the processing position and orientation setting unit 313 may set the difference ⁇ OL between the overall length of the first object model OM#1 and the overall length of the second object model OM#2 itself as the movement amount ⁇ OD of the second object model OM#2 described above.
- the user may input information regarding the difference ⁇ L between the total length of the first object model OM#1 and the total length of the second object model OM#2 to the control device 3 using the input device 34.
- the user may input information regarding the total length of the first object model OM#1 and the total length of the second object model OM#2 to the control device 3 using the input device 34.
- the processing position and orientation setting unit 313 may calculate the difference ⁇ OL between the total length of the first object model OM#1 and the total length of the second object model OM#2 based on the information regarding the total length of the first object model OM#1 and the total length of the second object model OM#2 input by the user.
- the processing position and orientation setting unit 313 may calculate the difference ⁇ OL between the total length of the first object model OM#1 and the total length of the second object model OM#2 based on the first object model OM#1 and the second object model OM#2.
- the object shape difference information may include information regarding the difference between the width of the first object model OM#1 and the width of the second object model OM#2.
- the processing position and orientation setting unit 313 may calculate the movement amount ⁇ ED of the second object model OM#2 described above based on the difference between the width of the first object model OM#1 and the width of the second object model OM#2.
- the processing position and orientation setting unit 313 may set the difference between the width of the first object model OM#1 and the width of the second object model OM#2 itself as the movement amount ⁇ ED of the second object model OM#2 described above.
- the user may input information regarding the difference between the width of the first object model OM#1 and the width of the second object model OM#2 to the control device 3 using the input device 34.
- the user may input information regarding the width of the first object model OM#1 and the width of the second object model OM#2 to the control device 3 using the input device 34.
- the processing position and orientation setting unit 313 may calculate the difference between the width of the first object model OM#1 and the width of the second object model OM#2 based on the information regarding the width of the first object model OM#1 and the width of the second object model OM#2.
- the processing position and orientation setting unit 313 may calculate the difference between the width of the first object model OM#1 and the width of the second object model OM#2 based on the first object model OM#1 and the second object model OM#2.
- the reference position of the object OBJ may change.
- the reference position of the object OBJ may be determined depending on the shape of the object OBJ.
- the reference position of the object OBJ may include the position of the datum of the object OBJ.
- the datum of the object OBJ may include at least one of the datum point and the datum surface of the object OBJ.
- the reference position of the object OBJ may include the reference position where the end effector 4 performs a predetermined processing. Therefore, the object shape difference information may include information regarding the difference between the reference position of the first object model OM#1 and the reference position of the second object model OM#2.
- the processing position and orientation setting unit 313 may calculate the movement amount ⁇ ED of the second object model OM#2 described above based on the difference between the reference position of the first object model OM#1 and the reference position of the second object model OM#2. As an example, the processing position and orientation setting unit 313 may set the difference between the reference position of the first object model OM#1 and the reference position of the second object model OM#2 as the movement amount ⁇ ED of the second object model OM#2 described above.
- the user may input information regarding the difference between the reference position of the first object model OM#1 and the reference position of the second object model OM#2 to the control device 3 using the input device 34.
- the user may input information regarding the reference position of the first object model OM#1 and the reference position of the second object model OM#2 to the control device 3 using the input device 34.
- the processing position and orientation setting unit 313 may calculate the difference between the reference position of the first object model OM#1 and the reference position of the second object model OM#2 based on the information regarding the reference position of the first object model OM#1 and the reference position of the second object model OM#2.
- the processing position and orientation setting unit 313 may calculate the difference between the reference position of the first object model OM#1 and the reference position of the second object model OM#2 based on the first object model OM#1 and the second object model OM#2.
- the object shape difference information indicating the difference between the shape of the first object model OM#1 and the shape of the second object model OM#2 may be considered to be information regarding the difference between the shape of the first object OBJ#1 and the shape of the second object OBJ#2.
- the first object model OM#1 is a three-dimensional model indicating the three-dimensional shape of the first object OBJ#1
- the second object model OM#2 is a three-dimensional model indicating the three-dimensional shape of the second object OBJ#2.
- the processing position and orientation setting unit 313 may set the processing position and orientation information PAI including information about the second set position STP#22 (see FIG. 18) indicated by the positional relationship between the second object model OM#2 changed based on the object shape difference information and the end effector model EM as the second processing position and orientation information PAI that can be used to calculate the second processing target position PTP#22.
- the user when the end effector 4 performs a predetermined process on a plurality of different objects OBJ, the user inputs an instruction to the control device 3 to change the positional relationship between the object model OM corresponding to one of the plurality of objects OBJ and the end effector model EM, while not having to input an instruction to the control device 3 to change the positional relationship between the object model OM corresponding to the remaining of the plurality of objects OBJ and the end effector model EM.
- This reduces the user's effort when setting a plurality of pieces of processing position and orientation information PAI corresponding to the plurality of objects OBJ.
- the time required to set a plurality of pieces of processing position and orientation information PAI corresponding to the plurality of objects OBJ is also reduced.
- This effect is particularly beneficial when a plurality of different processing position and orientation information PAI is set for a combination of the same object OBJ and the same end effector 4 as described in the first modified example.
- the more processing position and orientation information PAI that the processing position and orientation setting unit 313 needs to set the more beneficial this effect becomes.
- Fig. 19 is a flowchart showing the flow of the path generation process.
- the wait position PW refers to at least one of the pre-wait position PW_pre and the post-wait position PW_post.
- the path generating unit 315 may perform the path generating process described below to generate an arbitrary movement path of the tool center point TCP in addition to or instead of the movement path MRr of the tool center point TCP between the fine approach position PF and the pre-wait position PW_pre or the post-wait position PW_post.
- the route generation unit 315 may display a route generation screen 3550 used to generate the travel route MRr on the display device 35 (step S251). Specifically, the display signal generation unit 317 may generate a display signal for displaying the route generation screen 3550 on the display device 35 under the control of the route generation unit 315. Thereafter, the display signal output unit 318 may output the display signal generated by the display signal generation unit 317 to the display device 35. As a result, the display device 35 may display the route generation screen 3550 based on the display signal acquired from the display signal output unit 318.
- FIG. 20 An example of the path generation screen 3550 is shown in FIG. 20.
- the path generation screen 3550 is a screen on which an end effector model EM, a robot model RM, and a peripheral object model SM are arranged.
- a three-dimensional space SP5 that can be used as a virtual space is virtually constructed within the path generation screen 3550.
- the end effector model EM, the robot model RM, and the peripheral object model SM are arranged in this three-dimensional space SP5.
- at least one of the end effector model EM, the robot model RM, and the peripheral object model SM does not have to be arranged on the path generation screen 3550.
- a position within the three-dimensional space SP5 may be represented using a path generation coordinate system, which is a three-dimensional coordinate system used to indicate a position within the three-dimensional space SP5.
- the path generation coordinate system may be a three-dimensional coordinate system defined using an X-axis (SP5), a Y-axis (SP5) perpendicular to the X-axis (SP5), and a Z-axis (SP5) perpendicular to the X-axis (SP5) and Y-axis (SP5).
- the X-axis (SP5) may be set to be an axis corresponding to the X-axis of the global coordinate system.
- the X-axis (SP5) may be set to be an axis corresponding to the X-axis of the robot coordinate system.
- the X-axis (SP5) may be an axis set independently of the X-axis of the global coordinate system and the robot coordinate system.
- the Y-axis (SP5) may be set to be an axis corresponding to the X-axis of the global coordinate system.
- the Y-axis (SP5) may be set to be an axis corresponding to the Y-axis of the robot coordinate system.
- the Y-axis (SP5) may be an axis set independently of the Y-axis of the global coordinate system and the robot coordinate system.
- the Z-axis (SP5) may be set to be an axis corresponding to the Z-axis of the global coordinate system.
- the Z-axis (SP5) may be set to be an axis corresponding to the Z-axis of the robot coordinate system.
- the Z-axis (SP5) may be an axis set independently of the Z-axis of the global coordinate system and the robot coordinate system.
- a transformation matrix for converting three-dimensional coordinates in either the global coordinate system (or the robot coordinate system, the same applies hereinafter in this paragraph) or the path generation coordinate system into three-dimensional coordinates in the other of the global coordinate system and the path generation coordinate system may be calculated in advance.
- the transformation matrix for converting three-dimensional coordinates in either the global coordinate system or the path generation coordinate system into three-dimensional coordinates in the other of the global coordinate system and the path generation coordinate system may be information known to the path generation unit 315.
- the transformation matrix may include a translation transformation matrix that translates the three-dimensional coordinates in either the global coordinate system or the path generation coordinate system.
- the transformation matrix may include a translation transformation matrix that converts the position of an object in either the global coordinate system or the path generation coordinate system into the position of the object in the other of the global coordinate system and the path generation coordinate system.
- the transformation matrix may include a rotation transformation matrix that rotates the three-dimensional coordinates in either the global coordinate system or the path generation coordinate system.
- the transformation matrix may include a rotation transformation matrix that transforms the orientation of an object in either the global coordinate system or the path generation coordinate system into the orientation of the object in the other of the global coordinate system or the path generation coordinate system.
- the path generation unit 315 can use the transformation matrix to transform three-dimensional coordinates in either the global coordinate system or the path generation coordinate system into three-dimensional coordinates in the other of the global coordinate system or the path generation coordinate system.
- the end effector model EM is a three-dimensional model of the end effector 4 that is attached to the robot arm 12. Note that the end effector model EM has already been explained, so a detailed explanation will be omitted.
- the robot model RM is a three-dimensional model of the robot 1. Since the robot model RM has already been explained, a detailed explanation of it will be omitted.
- the surrounding object model SM is a three-dimensional model of the surrounding objects located around the robot 1. Note that the surrounding object model SM has already been explained, so a detailed explanation will be omitted.
- three peripheral objects are located around the robot 1, and three peripheral object models SM corresponding to the three peripheral objects are arranged on the path generation screen 3550.
- multiple peripheral object models SM corresponding to the multiple peripheral objects may be arranged on the path generation screen 3550.
- a combined model generated by combining at least two of the multiple peripheral object models SM corresponding to the multiple peripheral objects may be arranged as the peripheral object model SM on the path generation screen 3550.
- a single combined model generated by combining all of the multiple peripheral object models SM corresponding to the multiple peripheral objects may be arranged as the peripheral object model SM on the path generation screen 3550.
- the positions of the end effector model EM, the robot model RM, and the peripheral object model SM in the virtual three-dimensional space SP5 constructed in the path generation screen 3550 may correspond to the actual positions of the end effector 4, the robot 1, and the peripheral objects, respectively.
- the relative positional relationship between the end effector model EM, the robot model RM, and the peripheral object model SM in the virtual three-dimensional space SP5 may correspond to the actual relative positional relationship between the end effector 4, the robot 1, and the peripheral objects.
- the relative positional relationship between the end effector model EM, the robot model RM, and the peripheral object model SM in the virtual three-dimensional space SP5 may be substantially the same as the actual relative positional relationship between the end effector 4, the robot 1, and the peripheral objects.
- the end effector model EM, the robot model RM, and the peripheral object model SM may be placed in the three-dimensional space SP5 based on information regarding the actual positions of the end effector 4, the robot 1, and the peripheral objects.
- the end effector model EM, the robot model RM, and the peripheral object model SM may be positioned in the three-dimensional space SP5 based on information regarding the relative positional relationship between the end effector 4, the robot 1, and the peripheral object.
- the end effector model EM, robot model RM, and peripheral object model SM may be placed in the three-dimensional space SP5 based on a user's instruction input to the control device 3 using the input device 34.
- the user may input an instruction specifying the placement positions of the end effector model EM, robot model RM, and peripheral object model SM in the three-dimensional space SP5 to the control device 3 using the input device 34.
- the path generation unit 315 may place the end effector model EM, robot model RM, and peripheral object model SM at the placement positions specified by the user in the three-dimensional space SP5 based on the user's instruction.
- the end effector model EM, the robot model RM, and the peripheral object model SM may be placed in the three-dimensional space SP5 by the path generation unit 315 regardless of whether or not there is a user's instruction.
- the path generation unit 315 may acquire information regarding the actual placement positions of the end effector 4, the robot 1, and the peripheral objects.
- the path generation unit 315 may acquire image data generated by a camera capable of capturing images of the end effector 4, the robot 1, and the peripheral objects as information regarding the placement positions of the end effector 4, the robot 1, and the peripheral objects.
- At least one of the cameras 21 and 22 may be used as a camera capable of capturing images of the end effector 4, the robot 1, and the peripheral objects, or a camera other than the cameras 21 and 22 may be used.
- the path generation unit 315 may calculate the placement positions at which the end effector model EM, the robot model RM, and the peripheral object model SM should be placed in the three-dimensional space SP5 based on the acquired information. The path generation unit 315 may then place the end effector model EM, the robot model RM, and the peripheral object model SM at the calculated placement positions.
- the control device 3 may be input with end effector model information indicating the end effector model EM displayed on the path generation screen 3550, robot model information indicating the robot model RM displayed on the path generation screen 3550, and peripheral object model information indicating the peripheral object model SM displayed on the path generation screen 3550.
- end effector model information indicating the end effector model EM displayed on the path generation screen 3550
- robot model information indicating the robot model RM displayed on the path generation screen 3550
- peripheral object model information indicating the peripheral object model SM displayed on the path generation screen 3550.
- a user may input the end effector model information, robot model information, and peripheral object model information to the control device 3 using the input device 34.
- the path generation unit 315 may read the end effector model information, robot model information, and peripheral object model information from a recording medium on which the end effector model information, robot model information, and peripheral object model information are recorded.
- the processing position and orientation setting unit 313 may download the end effector model information, robot model information, and peripheral object model information from an external server on which the end effector model information, robot model information, and peripheral object model information are recorded.
- the end effector model information may include information regarding the type of the end effector model EM.
- the information regarding the type of the end effector model EM may include information regarding at least one of the identification number and model number of the end effector model EM.
- the end effector model information may include information regarding the position of the end effector model EM.
- the information regarding the position of the end effector model EM may include information regarding the position of the end effector model EM in a direction along a predetermined translation axis.
- the end effector model information may include information regarding the attitude of the end effector model EM.
- the information regarding the attitude of the end effector model EM may include information regarding the position of the end effector model EM in the rotation direction around a predetermined rotation axis (i.e., the attitude of the end effector model EM around the predetermined rotation axis). Note that the information regarding the attitude of the end effector model EM may be considered to be included in the information regarding the position of the end effector model EM.
- the end effector model information may include information regarding at least one of the magnification rate and the reduction rate of the end effector model EM.
- the path generating unit 315 may place an end effector model EM in a virtual three-dimensional space SP5 in the path generation screen 3550 based on the end effector model information.
- the path generating unit 315 may place an end effector model EM of a type indicated by the end effector model information in the three-dimensional space SP5.
- the path generating unit 315 may place the end effector model EM in the three-dimensional space SP5 at a position indicated by the end effector model information.
- the path generating unit 315 may place the end effector model EM in the three-dimensional space SP5 in a posture indicated by the end effector model information.
- the path generating unit 315 may place an end effector model EM having a size according to the enlargement ratio or reduction ratio indicated by the end effector model information in the three-dimensional space SP5.
- the robot model information may include information regarding the type of the robot model RM.
- the information regarding the type of the robot model RM may include information regarding at least one of the identification number and model number of the robot model RM.
- the robot model information may include information regarding the position of the robot model RM.
- the information regarding the position of the robot model RM may include information regarding the position of the robot model RM in a direction along a predetermined translation axis.
- the robot model information may include information regarding the posture of the robot model RM.
- the information regarding the posture of the robot model RM may include information regarding the position of the robot model RM in the rotation direction around a predetermined rotation axis (i.e., the posture of the robot model RM around the predetermined rotation axis). Note that the information regarding the posture of the robot model RM may be considered to be included in the information regarding the position of the robot model RM.
- the robot model information may include information regarding at least one of the enlargement ratio and reduction ratio of the robot model RM.
- the path generation unit 315 may place a robot model RM in a virtual three-dimensional space SP5 in the path generation screen 3550 based on the robot model information.
- the path generation unit 315 may place a robot model RM of a type indicated by the robot model information in the three-dimensional space SP5.
- the path generation unit 315 may place the robot model RM at a position indicated by the robot model information in the three-dimensional space SP5.
- the path generation unit 315 may place the robot model RM in a posture indicated by the robot model information in the three-dimensional space SP5.
- the path generation unit 315 may place a robot model RM having a size according to the enlargement ratio or reduction ratio indicated by the robot model information in the three-dimensional space SP5.
- the peripheral object model information may include information on the type of the peripheral object model SM.
- the information on the type of the peripheral object model SM may include information on at least one of the identification number and model number of the peripheral object model SM.
- the peripheral object model information may include information on the position of the peripheral object model SM.
- the information on the position of the peripheral object model SM may include information on the position of the peripheral object model SM in a direction along a predetermined translation axis.
- the peripheral object model information may include information on the orientation of the peripheral object model SM.
- the information on the orientation of the peripheral object model SM may include information on the position of the peripheral object model SM in the rotation direction around a predetermined rotation axis (i.e., the orientation of the peripheral object model SM around the predetermined rotation axis). Note that the information on the orientation of the peripheral object model SM may be considered to be included in the information on the position of the peripheral object model SM.
- the peripheral object model information may include information on at least one of the en
- the route generating unit 315 may place a peripheral object model SM in a virtual three-dimensional space SP5 in the route generation screen 3550 based on the peripheral object model information.
- the route generating unit 315 may place a peripheral object model SM of a type indicated by the peripheral object model information in the three-dimensional space SP5.
- the route generating unit 315 may place the peripheral object model SM at a position indicated by the peripheral object model information in the three-dimensional space SP5.
- the route generating unit 315 may place the peripheral object model SM in the three-dimensional space SP5 in a posture indicated by the peripheral object model information.
- the route generating unit 315 may place a peripheral object model SM having a size according to the enlargement ratio or reduction ratio indicated by the peripheral object model information in the three-dimensional space SP5.
- a movement start point MS may further be arranged in the virtual three-dimensional space SP5 constructed in the path generation screen 3550.
- the movement start point MS indicates the start point of the movement path MRr.
- the movement start point MS indicates the position where the tool center point TCP starts moving.
- the movement start point MS may indicate the position where any movement reference point MBP different from the tool center point TCP starts moving.
- the movement start point MS may indicate at least one of the waiting position PW and the fine approach position PF described above.
- the initial position of the movement start point MS may be specified by the user using the input device 34. In other words, the user may input an instruction to specify the initial position of the movement start point MS to the control device 3 using the input device 34.
- the path generation unit 315 may place the movement start point MS at a position specified by the user in the three-dimensional space SP5 based on the user's instruction.
- the initial position of the movement start point MS may be specified by the path generation unit 315.
- the path generating unit 315 may set the initial position of the movement start point MS based on at least one default value of the waiting position PW and the fine approach position PF described above.
- the path generating unit 315 may set the position of the tool center point TCP when the robot arm 12 is in the initial state as the initial position of the movement start point MS.
- the path generating unit 315 may place the movement start point MS at a position specified by the path generating unit 315 within the three-dimensional space SP5.
- the user or the path generation unit 315 may set the initial orientation of the end effector model EM at the movement start point MS.
- the user or the path generation unit 315 may set the initial orientation of the end effector model EM when the tool center point TCP is located at the movement start point MS.
- setting the initial orientation of the end effector model EM at the movement start point MS may be considered equivalent to setting the initial orientation of the end effector 4 at the start of movement.
- a movement end point ME may further be arranged in the virtual three-dimensional space SP5 constructed in the path generation screen 3550.
- the movement end point ME indicates the end point of the movement path MRr.
- the movement end point ME indicates the position where the tool center point TCP ends its movement.
- the movement start point MS may indicate the position where an arbitrary movement reference point MBP different from the tool center point TCP ends its movement.
- the movement end point ME may indicate at least one of the waiting position PW and the fine approach position PF described above.
- the initial position of the movement end point ME may be specified by the user using the input device 34. In other words, the user may input an instruction to specify the initial position of the movement end point ME to the control device 3 using the input device 34.
- the path generation unit 315 may place the movement end point ME at a position specified by the user in the three-dimensional space SP5 based on the user's instruction.
- the initial position of the movement end point ME may be specified by the path generation unit 315.
- the path generating unit 315 may set the initial position of the movement end point ME based on at least one default value of the waiting position PW and the fine approach position PF described above. For example, when the above-mentioned holding process is performed, the path generating unit 315 may set a position near the workpiece W held by the holding process as the initial position of the movement end point ME.
- the path generating unit 315 may set a position near the placement device T from which the workpiece W is released by the release process as the initial position of the movement end point ME.
- the path generating unit 315 may place the movement end point ME at a position specified by the path generating unit 315 in the three-dimensional space SP5.
- the user or the path generation unit 315 may set the initial orientation of the end effector model EM at the movement end point ME.
- the user or the path generation unit 315 may set the initial orientation of the end effector model EM when the tool center point TCP is located at the movement end point ME.
- setting the initial orientation of the end effector model EM at the movement end point ME may be considered equivalent to setting the initial orientation of the end effector 4 at the end of the movement.
- a moving via point MI may be further arranged in the virtual three-dimensional space SP5 constructed in the path generation screen 3550.
- the moving via point MI indicates a position on the moving path MRr between the start point of the moving path MRr and the end point of the moving path MRr.
- the moving via point MI indicates a position through which the tool center point TCP passes.
- the moving start point MS may indicate a position through which an arbitrary moving reference point MBP different from the tool center point TCP passes.
- the initial position of the moving via point MI may be specified by the user using the input device 34.
- the user may input an instruction to specify the initial position of the moving via point MI to the control device 3 using the input device 34.
- the path generation unit 315 may place the moving via point MI at a position specified by the user in the three-dimensional space SP5 based on the user's instruction.
- the initial position of the moving via point MI may be specified by the path generation unit 315.
- the path generation unit 315 may place the movement via point MI at a position specified by the path generation unit 315 within the three-dimensional space SP5.
- the moving via point MI may typically be a point at which the moving direction of the tool center point TCP changes.
- the moving via point MI may be a point that satisfies the condition that the moving direction of the tool center point TCP moving toward the moving via point MI is different from the moving direction of the tool center point TCP moving away from the moving via point MI.
- the user or the path generation unit 315 may set the initial orientation of the end effector model EM at the movement via point MI.
- the user or the path generation unit 315 may set the initial orientation of the end effector model EM when the tool center point TCP is located at the movement via point MI.
- setting the initial orientation of the end effector model EM at the movement via point MI may be considered equivalent to setting the initial orientation of the end effector 4 during movement.
- a single movement via point MI may be placed within the three-dimensional space SP5.
- a plurality of movement via points MI may be placed within the three-dimensional space SP5.
- Figure 20 shows an example in which a single movement via point MI is placed within the three-dimensional space SP5.
- the path generating unit 315 then generates a movement path MRr (step S252). Specifically, the path generating unit 315 generates a movement path MRr from the movement start point MS to the movement end point ME via the movement via point MI. In particular, the path generating unit 315 may generate a movement path MRr in which no interference occurs between at least one of the end effector model EM, the robot model RM, and the peripheral object model SM based on the relative positional relationship between the end effector model EM, the robot model RM, and the peripheral object model SM in the three-dimensional space SP5.
- the path generating unit 315 may place the end effector model EM, the robot model RM, and the peripheral object model SM in a predetermined virtual space (e.g., three-dimensional space SP5) and move the end effector model EM and the robot model RM in the virtual space in accordance with the movement of the end effector 4 and the robot 1 when the robot 1 is assumed to be driven, thereby generating a movement path MRr in which no interference occurs between at least one of the end effector model EM, the robot model RM, and the peripheral object model SM.
- the path generating unit 315 may generate a movement path MRr in which no interference occurs between at least one of the end effector model EM, the robot model RM, and the peripheral object model SM.
- step S252 the path generation unit 315 performs internal calculations to move the end effector model EM and the robot model RM within the three-dimensional space SP5, but does not need to update the end effector model EM and the robot model RM within the path generation screen 3550. In other words, the path generation unit 315 does not need to move the end effector model EM and the robot model RM on the path generation screen 3550.
- the peripheral object model SM used when generating the movement path MRr may be different from the peripheral object model SM used when generating the movement path MRf. At least one of the multiple peripheral object models SM used when generating the movement path MRr may be different from at least one of the multiple peripheral object models SM used when generating the movement path MRf. This is because the position of the robot 1 in the global coordinate system when the robot 1 is moving along the movement path MRr is typically different from the position of the robot 1 in the global coordinate system when the robot 1 is moving along the movement path MRf.
- the peripheral objects located around the robot 1 when the robot 1 is moving along the movement path MRr may be different from the peripheral objects located around the robot 1 when the robot 1 is moving along the movement path MRf.
- At least one of the multiple peripheral objects located around the robot 1 when the robot 1 is moving along the movement path MRr may be different from at least one of the multiple peripheral objects located around the robot 1 when the robot 1 is moving along the movement path MRf. Therefore, the peripheral object model SM used when generating the movement path MRr may be different from the peripheral object model SM used when generating the movement path MRf.
- the peripheral object model SM used when generating the movement path MRr may be different from the peripheral object model SM used when generating the movement path MRf.
- at least one of the multiple peripheral objects located around the robot 1 when the robot 1 moves along the movement path MRr is different from at least one of the multiple peripheral objects located around the robot 1 when the robot 1 moves along the movement path MRf
- at least one of the multiple peripheral object models SM used when generating the movement path MRr may be different from at least one of the multiple peripheral object models SM used when generating the movement path MRf.
- the peripheral object model SM used when generating the movement path MRr may be the same as the peripheral object model SM used when generating the movement path MRf. At least one of the multiple peripheral object models SM used when generating the movement path MRr may be the same as at least one of the multiple peripheral object models SM used when generating the movement path MRf. For example, if the peripheral objects located around the robot 1 when the robot 1 is moving along the movement path MRr are the same as the peripheral objects located around the robot 1 when the robot 1 is moving along the movement path MRf, the peripheral object model SM used when generating the movement path MRr may be the same as the peripheral object model SM used when generating the movement path MRf.
- At least one of the multiple peripheral objects located around the robot 1 when the robot 1 is moving along the movement path MRr is the same as at least one of the multiple peripheral objects located around the robot 1 when the robot 1 is moving along the movement path MRf
- at least one of the multiple peripheral object models SM used when generating the movement path MRr may be the same as at least one of the multiple peripheral object models SM used when generating the movement path MRf.
- the path generating unit 315 may use a path planning method to generate a movement path MRr from the movement start point MS to the movement end point ME via the movement via point MI.
- a path planning method is a random sampling-based path planning method that generates a path by randomly setting nodes through which the tool center point TCP should pass.
- the path generating unit 315 may first randomly set multiple nodes through which the tool center point TCP will pass.
- setting each node may be considered to be essentially equivalent to setting the position of the end effector model EM when the tool center point TCP is located at each node.
- the path generating unit 315 may calculate the attitude of the end effector model EM when the tool center point TCP is located at each node.
- the path generating unit 315 may calculate the position and posture of the robot model RM when the tool center point TCP is located at each node by back calculation from the posture of the end effector model EM when the tool center point TCP is located at each node. This is because the position and posture of the end effector 4 are determined depending on the drive amount of each link 121 of the robot 1 (particularly, the robot arm 12) to which the end effector 4 is attached, and when the drive amount of each link 121 of the robot 1 (particularly, the robot arm 12) is determined, the position and posture of the robot 1 (particularly, the robot arm 12) are also determined.
- the path generating unit 315 determines whether or not interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM during the period when the tool center point TCP passes through a plurality of nodes in order. If interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM, the path generation unit 315 may change the position of at least one of the multiple nodes, may delete at least one of the multiple nodes, or may add at least one new node.
- the path generation unit 315 may then repeat calculations based on the path planning method to minimize the path in which the tool center point TCP passes through the multiple nodes in order while satisfying the condition that no interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM. As a result, the movement path MRr is generated.
- the path generation unit 315 may generate a movement path MRr that does not satisfy the condition that no interference occurs between at least one of the end effector model EM, robot model RM, and peripheral object model SM.
- the path generating unit 315 may generate a series of paths including the movement path MRr of the end effector 4 between the fine approach position PF and the processing target position PTP, in addition to the movement path MRr of the end effector 4 between the fine approach position PF and the pre-wait position PW_pre or the post-wait position PW_post, as the movement path MRr.
- the path generating unit 315 may generate the movement path MRf of the end effector 4 between the fine approach position PF and the processing target position PTP, which is generated by the robot control unit 316 based on at least one of the image data IMG_2D and IMG_3D, without using the image data IMG_2D and IMG_3D.
- the path generating unit 315 may use a path planning method to generate a movement path MRr including a movement path MRr that does not cause interference with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM.
- the route generation unit 315 displays the movement route MRr generated in step S252 on the display device 35 (step S253). That is, the route generation unit 315 displays a route generation screen 3550 including the movement route MRr generated in step S252 on the display device 35 (step S253).
- the display signal generation unit 317 generates a display signal for displaying the route generation screen 3550 including the movement route MRr generated in step S252 on the display device 35.
- the display signal output unit 318 outputs the display signal generated by the display signal generation unit 317 to the display device 35.
- the display device 35 displays the route generation screen 3550 including the movement route MRr generated in step S252 based on the display signal acquired from the display signal output unit 318.
- the path generation unit 315 may display a path generation screen 3550 including the movement path MRf of the end effector 4 between the fine approach position PF and the processing target position PTP in step S253.
- FIG. 21 shows an example in which the movement path MRr is displayed on the path generation screen 3550 together with the end effector model EM, the robot model RM, and the surrounding object model SM. While the movement path MRr is displayed on the path generation screen 3550, at least one of the end effector model EM, the robot model RM, and the surrounding object model SM does not necessarily have to be displayed.
- the path generation process shown in FIG. 19 is performed before the robot 1 actually starts moving by performing the processes of steps S31 to S37 in FIG. 6. That is, the path generation process shown in FIG. 19 is performed without actually moving the robot 1. In other words, the path generation process shown in FIG. 19 is performed without actually moving the robot 1 based on a movement path MRr that is generated in advance before the robot 1 is actually moved by the path generation process shown in FIG. 19.
- the movement path MRr displayed on the path generation screen 3550 may be considered to show the movement trajectory of the tool center point TCP (or any movement reference point MBP) that moves as the robot 1 is driven, when it is assumed that the robot 1 is driven so that the tool center point TCP (or any movement reference point MBP) moves along the generated movement path MRr.
- the movement path MRr displayed on the path generation screen 3550 may be considered to show the movement trajectory of the tool center point TCP (or any movement reference point MBP) that moves with the driving of the robot 1 in the simulation when a simulation is performed in which the robot 1 is driven so that the tool center point TCP (or any movement reference point MBP) moves along the generated movement path MRr.
- the movement path MRr displayed on the path generation screen 3550 does not need to show the movement trajectory of the tool center point TCP (or any movement reference point MBP) that actually moves along the generated movement path MRr.
- the movement path of the tool center point TCP may be read as the movement path of the any movement reference point MBP.
- Figure 21 shows an example in which, in step S253 of Figure 19, the path generation unit 315 displays the movement path MRr of the tool center point TCP generated in step S252 of Figure 19.
- Figure 21 shows an example in which, in step S253 of Figure 19, the path generation unit 315 displays the movement trajectory of the tool center point TCP that moves as the robot 1 is driven when the robot 1 is driven so that the tool center point TCP moves along the movement path MRr generated in step S252 of Figure 19.
- the path generating unit 315 may display on the display device 35 a movement trajectory of an arbitrary movement point MP that moves with the driving of the robot 1 when the robot 1 is driven so that the tool center point TCP moves along the movement path MRr generated in step S252 in addition to or instead of the movement path MRr generated in step S252 of Fig. 19 (i.e., the movement trajectory of the tool center point TCP).
- the path generating unit 315 may display on the display device 35 a movement trajectory of an arbitrary movement point MP that moves with the driving of the robot 1 when it is assumed that the robot 1 is driven so that the tool center point TCP moves along the generated movement path MRr.
- the path generating unit 315 may display on the display device 35 a movement trajectory of an arbitrary movement point MP that moves with the driving of the robot 1 in the simulation. Furthermore, the movement trajectory of the moving point MP displayed on the route generation screen 3550 does not have to show the movement trajectory that the moving point MP actually moved along.
- At least a part of the robot 1 moves as the robot 1 is driven, at least a part of the robot 1 may be used as the moving point MP. At least one point of the robot 1 that is included in at least a part of the robot 1 may be used as the moving point MP. As one example, any point on the surface of the robot 1 may be used as the moving point MP. As another example, any point inside the robot 1 may be used as the moving point MP. Specifically, since at least a part of the robot arm 12 moves as the robot 1 is driven, at least a part of the robot arm 12 may be used as the moving point MP. At least one point of the robot arm 12 that is included in at least a part of the robot arm 12 may be used as the moving point MP.
- any point on the surface of the robot arm 12 may be used as the moving point MP.
- any point inside the robot arm 12 may be used as the moving point MP.
- at least a portion of the end effector 4 attached to the robot 1 moves as the robot 1 moves, at least a portion of the end effector 4 may be used as the moving point MP.
- At least one point of the end effector 4 that is included in at least a portion of the end effector 4 may be used as the moving point MP.
- an arbitrary point on the surface of the end effector 4 may be used as the moving point MP.
- an arbitrary point inside the end effector 4 may be used as the moving point MP.
- the tool center point TCP also moves as the robot 1 is driven.
- the tool center point TCP may be considered to be an example of a moving point MP.
- the tool center point TCP may also be considered to be a point determined based on the end effector 4. In this case, any point determined based on the end effector 4 may be used as the moving point MP.
- the tool center point TCP may also be considered to be a point determined based on the robot 1 (particularly, the robot arm 12) to which the end effector 4 is attached. In this case, any point determined based on the robot 1 (particularly, the robot arm 12) may be used as the moving point MP.
- the above-mentioned movement start point MS may indicate the start point of the movement trajectory of the arbitrary moving point MP.
- the posture of the end effector 4 at the movement start point MS may mean the posture of the end effector 4 when the arbitrary moving point MP is located at the movement start point MS.
- the above-mentioned movement end point ME may indicate the end point of the movement trajectory of the arbitrary moving point MP.
- the posture of the end effector 4 at the movement end point ME may mean the posture of the end effector 4 when the arbitrary moving point MP is located at the movement end point ME.
- the above-mentioned movement via point MI may indicate the position through which the arbitrary moving point MP passes.
- the posture of the end effector 4 at the movement via point MI may mean the posture of the end effector 4 when the arbitrary moving point MP is located at the movement via point MI.
- the moving reference point MBP exists as a term similar to the moving point MP whose moving trajectory is displayed on the path generation screen 3550.
- the moving reference point MBP is mainly used as a moving object (i.e., a control object) used when controlling the driving of the robot 1.
- the moving reference point MBP may be considered to be an element different from the moving point MP, which is a display object whose moving trajectory is displayed on the path generation screen 3550.
- the moving reference point MBP is a point that moves with the driving of the robot 1.
- the moving reference point MBP may be considered to be an example of the moving point MP. For this reason, as shown in FIG.
- displaying the moving path MRr (moving trajectory) of the tool center point TCP used as the moving reference point MBP may be considered to be equivalent to displaying the moving trajectory of a moving point that coincides with the moving reference point MBP.
- Displaying the moving trajectory of a moving point different from the tool center point TCP used as the moving reference point MBP may be considered to be equivalent to displaying the moving trajectory of a moving point different from the moving reference point MBP.
- the path generation unit 315 may calculate the movement trajectory of the arbitrary moving point MP based on the generated movement path MRr, and display the calculated movement trajectory. Alternatively, the path generation unit 315 may calculate the movement trajectory of the arbitrary moving point MP without using the generated movement path MRr, and display the calculated movement trajectory.
- the route generating unit 315 may display the moving point MP moving along the movement trajectory in addition to or instead of the movement trajectory of the arbitrary moving point MP. That is, the route generating unit 315 may perform an animation display in which the moving point MP moves along the movement trajectory in the route generation screen 3550. That is, the route generating unit 315 may display the movement trajectory of the arbitrary moving point MP using the movement of the arbitrary moving point MP.
- the route generation unit 315 may display the movement trajectories of multiple different moving points MP.
- the route generation unit 315 may display a route generation screen 3550 including the movement trajectory of a first moving point MP and the movement trajectory of a second moving point MP that is different from the first moving point MP.
- the path generating unit 315 may display the movement trajectory of at least a part of a moving body (specifically, a moving body that is an object having a size) that moves with the driving of the robot 1. For example, the path generating unit 315 may display the movement trajectory of a part of a moving body that moves with the driving of the robot 1. For example, the path generating unit 315 may display the movement trajectory of the entire moving body that moves with the driving of the robot 1. As an example, as shown in FIG. 22, the path generating unit 315 may display the movement trajectory of at least a part of an end effector 4 that moves with the driving of the robot 1. As another example, the path generating unit 315 may display the movement trajectory of at least a part of the robot 1 that moves with the driving of the robot 1.
- the path generating unit 315 may display the movement trajectory of at least a part of the robot arm 12 that moves with the driving of the robot 1.
- the path generating unit 315 may display the movement trajectory of at least a part of a link 121 that moves with the driving of the robot 1.
- the path generation unit 315 may display the movement trajectory of at least a part of the moving body that includes the multiple different moving points MP as the movement trajectory of the multiple different moving points MP.
- the path generation unit 315 may display the movement trajectory of the multiple different moving points MP included in at least a part of the moving body as the movement trajectory of the multiple different moving points MP.
- the path generating unit 315 may display at least a part of the moving body moving along the movement trajectory in addition to or instead of the movement trajectory of at least a part of the moving body. That is, the path generating unit 315 may display an animation of at least a part of the moving body moving along the movement trajectory in the path generation screen 3550. That is, the path generating unit 315 may display the movement trajectory of at least a part of the moving body using the movement of at least a part of the moving body. As an example, as shown in FIG. 22, the path generating unit 315 may display an animation of at least a part of the end effector 4 moving as the robot 1 is driven. The path generating unit 315 may display a movement trajectory of.
- the path generating unit 315 may display an animation of at least a part of the robot 1 moving as the robot 1 is driven.
- the path generating unit 315 may display an animation of at least a part of the robot arm 12 moving as the robot 1 is driven.
- the path generating unit 315 may display an animation of at least a part of the link 121 moving as the robot 1 is driven.
- the path generation unit 315 may display at least a part of the moving body including the multiple different moving points MP as a movement trajectory of the multiple different moving points MP.
- the path generation unit 315 may display the multiple different moving points MP included in at least a part of the moving body as a movement trajectory of the multiple different moving points MP.
- the path generation unit 315 may perform an animation display that moves at least a part of the moving body including the multiple different moving points MP.
- the path generation unit 315 may perform an animation display that moves the multiple different moving points MP included in at least a part of the moving body.
- the path generating unit 315 may display at least a part of the moving object moving along the movement trajectory by using an afterimage of at least a part of the moving object.
- FIG. 22 shows an example in which the path generating unit 315 displays the end effector 4 (specifically, the end effector model EM) that moves as the robot 1 is driven by using an afterimage of the moving end effector 4 (specifically, the end effector model EM). In this case, the visibility of the movement trajectory of at least a part of the moving object is improved.
- the movement trajectory may mean a continuous movement trajectory.
- the route generation unit 315 may display a route generation screen 3550 including the movement trajectory of the movement point MP as a continuous movement trajectory.
- the movement trajectory may mean a discrete movement trajectory including multiple partial trajectories, each of which indicates a part of the movement trajectory.
- the route generation unit 315 may display a route generation screen 3550 including the movement trajectory of the movement point MP as a discrete movement trajectory including multiple partial trajectories, each of which indicates a part of the movement trajectory.
- the route generation unit 315 judges whether or not the movement route MRr generated in step S252 is appropriate (step S254). In other words, the route generation unit 315 judges whether or not the movement route MRr displayed on the route generation screen 3550 in step S253 is appropriate (step S254).
- the route generation unit 315 may determine whether the travel route MRr generated in step S252 is appropriate based on a user instruction input to the control device 3 using the input device 34.
- the user instruction input to the control device 3 using the input device 34 may include information regarding the user's determination result regarding whether the travel route MRr generated in step S252 is appropriate.
- the route generation unit 315 may determine whether the travel route MRr generated in step S252 is appropriate without using a user instruction.
- the user or the path generating unit 315 may determine that the generated movement path MRr is inappropriate when the length of the generated movement path MRr is longer than a predetermined allowable length. For example, the user or the path generating unit 315 may determine that the generated movement path MRr is appropriate when the length of the generated movement path MRr is shorter than a predetermined allowable length. For example, the user or the path generating unit 315 may determine that the generated movement path MRr is inappropriate when the generated movement path MRr is too far away from the peripheral object model SM.
- a state in which the movement path MRr is too far away from the peripheral object model SM may mean a state in which the distance between the movement path MRr and the peripheral object model SM (typically, the distance between the movement path MRr and the peripheral object model SM when the movement path MRr and the peripheral object model SM are the furthest apart) is longer than a predetermined distance threshold.
- the user or the path generating unit 315 may determine whether the generated movement path MRr is appropriate based on the degree of interference with at least one of the end effector 4, the robot 1, and the surrounding objects when it is assumed that the tool center point TCP moves along the generated movement path MRr.
- the path generating unit 315 when generating the movement path MRr in step S252, the path generating unit 315 generates a movement path MRr that satisfies the condition that interference does not occur with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM in principle. In other words, the path generating unit 315 generates a movement path MRr that satisfies the condition that interference does not occur with at least one of the end effector 4, the robot 1, and the peripheral objects when it is assumed that the tool center point TCP moves along the generated movement path MRr.
- the path generating unit 315 may not be able to generate a movement path MRr that satisfies the condition that interference does not occur.
- the user or the path generating unit 315 may determine whether the generated movement path MRr is appropriate based on the degree of interference with at least one of the end effector 4, the robot 1, and the peripheral objects when it is assumed that the tool center point TCP moves along the generated movement path MRr.
- the path generating unit 315 may display interference information regarding interference with at least one of the end effector 4, the robot 1, and the surrounding objects on the display device 35 to assist the user in determining whether the generated movement path MRr is appropriate. Specifically, the path generating unit 315 may display a path generation screen 3550 including interference information regarding interference with at least one of the end effector 4, the robot 1, and the surrounding objects on the display device 35.
- interference with at least one of the end effector 4, the peripheral object, and the robot 1 may mean interference with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM.
- interference between at least a part of the end effector 4 and at least a part of the robot 1 may mean interference between at least a part of the end effector model EM and at least a part of the robot model RM.
- interference between at least a part of the end effector model EM and at least a part of the robot model RM may be considered to be equivalent to interference between at least a part of the end effector 4 and at least a part of the robot 1.
- interference between at least a part of the end effector 4 and at least a part of the peripheral object may mean interference between at least a part of the end effector model EM and at least a part of the peripheral object model SM. That is, in the following description, interference between at least a part of the end effector model EM and at least a part of the peripheral object model SM may be considered to be equivalent to interference between at least a part of the end effector 4 and at least a part of the peripheral object.
- interference between at least a part of the robot 1 and at least a part of the peripheral object may mean interference between at least a part of the robot model RM and at least a part of the peripheral object model SM. That is, in the following description, interference between at least a part of the robot model RM and at least a part of the peripheral object model SM may be considered to be equivalent to interference between at least a part of the robot 1 and at least a part of the peripheral object.
- interference between a first part of the robot 1 and a second part of the robot 1 may mean interference between a first part of the robot model RM and a second part of the robot model RM. That is, in the following description, interference between a first part of the robot model RM and a second part of the robot model RM may be considered to be equivalent to interference between a first part of the robot 1 and a second part of the robot 1.
- the path generating unit 315 may perform a simulation of driving the robot 1 so that the tool center point TCP moves along the generated movement path MRr, using the end effector model EM, the robot model RM, and the surrounding object model SM. That is, the path generating unit 315 may perform a simulation of moving the robot 1 and the end effector 4 according to (in other words, based on) the generated movement path MRr, using the end effector model EM, the robot model RM, and the surrounding object model SM.
- the path generating unit 315 may perform a simulation of moving the end effector model EM, the robot model RM, and the surrounding object model SM according to (in other words, based on) the generated movement path MRr. Thereafter, the path generating unit 315 may generate interference information regarding interference with at least one of the end effector 4, the robot 1, and the surrounding object, based on the result of the simulation.
- the path generating unit 315 may determine whether interference occurs between at least one of the end effector model EM, the robot model RM, and the peripheral object model SM based on the results of the simulation. In other words, the path generating unit 315 may determine whether interference occurs between at least one of the end effector 4, the robot 1, and the peripheral object, assuming that the robot 1 and the end effector 4 are moved according to (in other words, based on) the generated movement path MRr. In other words, the path generating unit 315 may determine whether interference occurs between at least one of the robot 1 moving according to (in other words, based on) the generated movement path MRr, the end effector 4 moving according to (in other words, based on) the generated movement path MRr, and the peripheral object. Thereafter, the path generating unit 315 may generate interference information based on the determination result of whether interference occurs between at least one of the end effector model EM, the robot model RM, and the peripheral object model SM.
- the path generating unit 315 may perform a simulation of driving the robot 1 so that the tool center point TCP moves along the generated movement path MRr after displaying the path generation screen 3550 including the generated movement path MRr on the display device 35.
- the path generating unit 315 may perform a simulation of driving the robot 1 so that the tool center point TCP moves along the generated movement path MRr before displaying the path generation screen 3550 including the generated movement path MRr on the display device 35.
- the path generating unit 315 may generate interference information regarding interference of at least one of the end effector 4, the robot 1, and the surrounding objects before displaying the path generation screen 3550 including the generated movement path MRr on the display device 35.
- the path generating unit 315 may display the path generation screen 3550 including the generated movement path MRr and the generated interference information on the display device 35.
- the path generation unit 315 may perform a simulation of driving the robot 1 so that the moving point MP moves along the movement trajectory displayed on the path generation screen 3550, using the end effector model EM, the robot model RM, and the peripheral object model SM.
- the path generation unit 315 may perform a simulation of moving the robot 1 and the end effector 4 according to (in other words, based on) the movement trajectory displayed on the path generation screen 3550, using the end effector model EM, the robot model RM, and the peripheral object model SM.
- the path generation unit 315 may perform a simulation of moving the end effector model EM, the robot model RM, and the peripheral object model SM according to (in other words, based on) the movement trajectory displayed on the path generation screen 3550.
- the path generation unit 315 may then generate interference information regarding interference with at least one of the end effector 4, the robot 1, and the surrounding objects based on the results of the simulation.
- the path generating unit 315 may determine whether interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM based on the results of the simulation. In other words, the path generating unit 315 may determine whether interference occurs with at least one of the end effector 4, the robot 1, and the peripheral object, assuming that the robot 1 and the end effector 4 are moved according to (in other words, based on) the movement trajectory displayed on the path generating screen 3550.
- the path generating unit 315 may determine whether interference occurs with at least one of the robot 1 moving according to (in other words, based on) the movement trajectory displayed on the path generating screen 3550, the end effector 4 moving according to (in other words, based on) the movement trajectory displayed on the path generating screen 3550, and the peripheral object. After that, the path generating unit 315 may generate interference information based on the determination result of whether interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM.
- the path generating unit 315 may perform a simulation of driving the robot 1 so that the moving point MP moves along the moving path displayed on the path generating screen 3550 after displaying the path generation screen 3550 including the moving path of the moving point MP on the display device 35.
- the path generating unit 315 may perform a simulation of driving the robot 1 so that the moving point MP moves along the moving path displayed on the path generation screen 3550 before displaying the path generation screen 3550 including the moving path of the moving point MP on the display device 35.
- the path generating unit 315 may generate interference information regarding interference with at least one of the end effector 4, the robot 1, and the surrounding objects before displaying the path generation screen 3550 including the moving path of the moving point MP on the display device 35.
- the path generating unit 315 may display the path generation screen 3550 including the generated moving path of the moving point MP and the generated interference information on the display device 35.
- the path generation unit 315 may display the interference information in any manner as long as the user can understand the interference information. For example, the path generation unit 315 may display the interference information using a text message. For example, the path generation unit 315 may display the interference information using an icon. For example, the path generation unit 315 may display the interference information using an illustration.
- the interference information may include information indicating whether interference will occur between at least one of the end effector model EM, the robot model RM, and the surrounding object model SM.
- the interference information may include information indicating whether interference will occur between at least one of the end effector 4, the robot 1, and the surrounding object.
- the path generation unit 315 may display a display object 3551 indicating information indicating whether interference will occur between at least one of the end effector model EM, the robot model RM, and the surrounding object model SM, as the interference information.
- the path generation unit 315 may display a display object 3551 indicating information indicating whether interference will occur between the end effector 4, the robot 1, and the surrounding object, as the interference information.
- the interference information may include information about an interference model.
- An interference model is at least one of the end effector model EM, the robot model RM, and the surrounding object model SM, and means a model in which interference is occurring. For example, when interference occurs between at least a part of the end effector model EM and at least a part of the robot model RM, each of the end effector model EM and the robot model RM becomes an interference model, and the interference information may include at least one of information about the end effector model EM that interferes with at least a part of the robot model RM and information about the robot model RM that interferes with at least a part of the end effector model EM.
- the interference information may include at least one of information about the end effector 4 that interferes with at least a part of the robot 1 and information about the robot 1 that interferes with at least a part of the end effector 4.
- each of the end effector model EM and the peripheral object model SM may be an interference model
- the interference information may include at least one of information on the end effector model EM interfering with at least a part of the peripheral object model SM and information on the peripheral object model SM interfering with at least a part of the end effector model EM.
- the interference information may include at least one of information on the end effector 4 interfering with at least a part of the peripheral object and information on the peripheral object interfering with at least a part of the end effector 4.
- each of the robot model RM and the peripheral object model SM may be an interference model
- the interference information may include at least one of information on the robot model RM interfering with at least a part of the peripheral object model SM and information on the peripheral object model SM interfering with at least a part of the robot model RM.
- the interference information may include at least one of information about the robot 1 interfering with at least a part of the peripheral object and information about the peripheral object interfering with at least a part of the robot 1.
- the robot model RM becomes an interference model
- the interference information may include at least one of information about the first part of the robot model RM interfering with the second part of the robot model RM and information about the second part of the robot model RM interfering with the first part of the robot model RM.
- the interference information may include at least one of information about the first part of the robot 1 interfering with the second part of the robot 1 and information about the second part of the robot 1 interfering with the first part of the robot 1.
- the path generating unit 315 may display a display object 3551 indicating an interference model as interference information. That is, as shown in FIG. 23, the path generating unit 315 may display a display object 3551 indicating an interfering object, which is at least one of the end effector 4, the robot 1, and the surrounding objects, where interference is occurring, as interference information. In the example shown in FIG. 23, the path generating unit 315 displays a display object 3551 indicating the robot arm 12 interfering with the storage box CB of the workpiece W, which is an example of a surrounding object, and the surrounding object (storage box CB) interfering with the robot arm 12 as interference information. That is, in the example shown in FIG. 23, the path generating unit 315 displays a display object 3551 indicating the robot model RM interfering with the surrounding object model SM of the storage box CB and the surrounding object model SM interfering with the robot model RM as interference information, as interference models.
- the path generation unit 315 displays the name of the interference model as information related to the interference model. In other words, the path generation unit 315 displays the name of the interfering object as information related to the interfering object.
- the path generation unit 315 may display the information related to the interference model in any manner as long as the user can identify the interference model. In other words, the path generation unit 315 may display the information related to the interfering object in any manner as long as the user can identify the interfering object.
- the path generation unit 315 may display the information related to the interference model using identification information (e.g., model number) of the interference model.
- the path generation unit 315 may display the information related to the interfering object using identification information (e.g., model number) of the interfering object.
- the interference information may include information on the position of the interference model. That is, the interference information may include information on the position of the interfering object. In particular, the interference information may include information on the position of the part of the interference model where interference occurs. That is, the interference information may include information on the position of the part of the interfering object where interference occurs.
- FIG. 23 shows an example in which interference information is displayed indicating that the tip of the robot arm 12, which is the first interfering object, interferes with the storage box CB, which is the second interfering object, and that the right side surface of the storage box CB, which is the second interfering object, interferes with the robot arm 12, which is the first interfering object.
- FIG. 23 shows an example in which interference information is displayed indicating that the tip of the robot arm 12, which is the first interfering object, interferes with the storage box CB, which is the second interfering object, and that the right side surface of the storage box CB, which is the second interfering object, interferes with the robot arm 12, which is the first
- FIG. 23 shows an example in which interference information is displayed indicating that a model part corresponding to the tip of the robot arm 12 in the robot model RM, which is the first interference model, interferes with the peripheral object model SM (peripheral object model SM indicating the storage box CB), which is the second interference model, and that a model part corresponding to the right side surface of the storage box CB in the peripheral object model SM, which is the second interference model, interferes with the robot model RM, which is the first interference model.
- the peripheral object model SM peripheral object model SM indicating the storage box CB
- the path generating unit 315 may display the interference model such that the display mode of the part of the interference model where interference occurs is different from the display mode of the other parts of the interference model. For example, when interference occurs between at least a part of the end effector model EM and at least a part of the robot model RM, the path generating unit 315 may display the end effector model EM such that the display mode of a part of the model part of the end effector model EM that interferes with the robot model RM is different from the display mode of other part of the model part of the end effector model EM that does not interfere with the robot model RM.
- the path generating unit 315 may display the robot model RM such that the display mode of a part of the model part of the robot model RM that interferes with the end effector model EM is different from the display mode of other part of the model part of the robot model RM that does not interfere with the end effector model EM.
- the path generating unit 315 may display the end effector model EM such that a display mode of a part of the model portion of the end effector model EM that interferes with the peripheral object model SM is different from a display mode of a part of the other model portion of the end effector model EM that does not interfere with the peripheral object model SM.
- the path generating unit 315 may display the peripheral object model SM such that a display mode of a part of the model portion of the peripheral object model SM that interferes with the end effector model EM is different from a display mode of a part of the other model portion of the peripheral object model SM that does not interfere with the end effector model EM.
- the path generating unit 315 may display the robot model RM such that the display mode of a part of the model part of the robot model RM that interferes with the peripheral object model SM is different from the display mode of the other part of the model part of the robot model RM that does not interfere with the peripheral object model SM.
- the path generating unit 315 may display the peripheral object model SM such that the display mode of a part of the model part of the peripheral object model SM that interferes with the robot model RM is different from the display mode of the other part of the model part of the peripheral object model SM that does not interfere with the robot model RM.
- the robot model RM may be displayed such that the display mode of the first and second parts of the robot model RM is different from the display mode of the other model parts of the robot model RM.
- the display mode may include a display color.
- the path generating unit 315 may display the interference model such that the display color of the part of the interference model where interference occurs is different from the display color of the other parts of the interference model.
- the path generating unit 315 may display the end effector model EM such that the display color of the model part of the end effector model EM that corresponds to the tip of the robot arm 12 is different from the display color of the other model parts of the end effector model EM.
- FIG. 23 the path generating unit 315 may display the end effector model EM such that the display color of the model part of the end effector model EM that corresponds to the tip of the robot arm 12 is different from the display color of the other model parts of the end effector model EM.
- the path generating unit 315 may display the peripheral object model SM such that the color of the model part of the peripheral object model SM that corresponds to the right side surface of the storage box CB is different from the display color of the other model parts of the peripheral object model SM. In this case, the user can intuitively identify which part of the interference model is causing interference.
- step S254 if the result of the judgment in step S254 is that the generated travel route MRr is not appropriate (step S254: No), the user inputs an instruction to change the travel route MRr to the control device 3 using the input device 34 (step S255).
- the user may perform an operation to input an instruction to change the travel route MRr on the route generation screen 3550.
- the user may input an instruction to change the travel route MRr via the route generation screen 3550.
- the route generation unit 315 acquires the user's instruction to change the travel route MRr using the input device 34 (step S255).
- the user may input an instruction to change the position of the movement start point MS in the virtual three-dimensional space SP5 constructed in the route generation screen 3550 as an instruction to change the movement route MRr.
- the user may input an instruction to change the position of the movement start point MS by using the input device 34 to perform an operation to move (e.g., drag) the movement start point MS on the route generation screen 3550.
- the movement start point MS indicates the movement start point of the movement trajectory of the movement point MP different from the tool center point TCP.
- the user may input an instruction to change the position of the movement start point MS (i.e., the position of the movement start point of the movement trajectory of the movement point MP different from the tool center point TCP), or may input an instruction to change the position of the movement start point of the movement path MRr of the tool center point TCP.
- the user may input an instruction to change the position of the movement start point of a desired movement path different from the movement start point MS displayed on the path generation screen 3550.
- the user may input an instruction to change the position of the movement end point ME within the three-dimensional space SP5 as an instruction to change the movement route MRr.
- the user may input an instruction to change the position of the movement end point ME by using the input device 34 to perform an operation to move (e.g., drag) the movement end point ME on the route generation screen 3550.
- the movement end point ME indicates the movement end point of the movement trajectory of the movement point MP other than the tool center point TCP.
- the user may input an instruction to change the position of the movement end point ME (i.e., the position of the movement end point of the movement trajectory of the movement point MP other than the tool center point TCP), or may input an instruction to change the position of the movement end point of the movement path MRr of the tool center point TCP.
- the user may input an instruction to change the position of the movement end point of a desired movement path other than the movement end point ME displayed on the path generation screen 3550.
- the user may input an instruction to change the position of the movement via point MI within the three-dimensional space SP5 as an instruction to change the movement route MRr.
- the user may input an instruction to change the position of the movement via point MI by using the input device 34 to perform an operation to move (e.g., drag) the movement via point MI on the route generation screen 3550.
- the movement via point MI indicates the movement via point of the movement trajectory of the movement point MP other than the tool center point TCP.
- the user may input an instruction to change the position of the movement via point MI (i.e., the position of the movement via point of the movement trajectory of the movement point MP other than the tool center point TCP), or may input an instruction to change the position of the movement via point of the movement path MRr of the tool center point TCP.
- the user may input an instruction to change the position of the movement via point of a desired movement path other than the movement via point MI displayed on the path generation screen 3550.
- the user may input an instruction to add at least one movement via point MI to the three-dimensional space SP5 as an instruction to change the movement route MRr.
- the user may input an instruction to delete at least one movement via point MI from the three-dimensional space SP5 as an instruction to change the movement route MRr.
- the user may input an instruction to change the position of the end effector model EM in the three-dimensional space SP5 as an instruction to change the movement path MRr.
- the user may input an instruction to change the position of the end effector model EM by using the input device 34 to move (e.g., drag) the end effector model EM on the path generation screen 3550.
- an instruction to change the position of at least one of the above-mentioned movement start point MS, movement end point ME, and movement via point MI may be considered to be substantially equivalent to an instruction to change the position of the end effector model EM in the three-dimensional space SP5.
- the user may input an instruction to change the posture of the end effector model EM in the three-dimensional space SP5 as an instruction to change the movement path MRr.
- the user may input an instruction to change the posture of the end effector model EM by performing an operation to move (e.g., rotate) the end effector model EM on the path generation screen 3550 using the input device 34.
- the user may input an instruction to change the posture of the end effector 4 at the movement start point MS in the three-dimensional space SP5 as an instruction to change the movement path MRr.
- the user may input an instruction to change the posture of the end effector 4 at the movement end point ME in the three-dimensional space SP5 as an instruction to change the movement path MRr.
- the user may input an instruction to change the posture of the end effector 4 at the movement via point MI in the three-dimensional space SP5 as an instruction to change the movement path MRr.
- the user may input an instruction to change the position of the robot model RM in the three-dimensional space SP5 as an instruction to change the movement path MRr.
- the user may input an instruction to change the position of the robot model RM by using the input device 34 to move (e.g., drag) the robot model RM on the path generation screen 3550.
- an instruction to change the position of at least one of the above-mentioned movement start point MS, movement end point ME, and movement via point MI may be considered to be substantially equivalent to an instruction to change the position of the robot model RM in the three-dimensional space SP5.
- the user may input an instruction to change the posture of the robot model RM in the three-dimensional space SP5 as an instruction to change the movement path MRr.
- the user may input an instruction to change the posture of the robot model RM by using the input device 34 to perform an operation to move (e.g., rotate) the robot model RM on the path generation screen 3550.
- the user may input an instruction to change the position of the peripheral object model SM in the three-dimensional space SP5 as an instruction to change the travel route MRr.
- the user may input an instruction to change the position of the peripheral object model SM by using the input device 34 to move (e.g., drag) the peripheral object model SM on the route generation screen 3550.
- the position of the peripheral object model SM in the three-dimensional space SP5 may correspond to the actual position of the peripheral object. Therefore, when the position of the peripheral object model SM is changed, the user may change the position of the actual peripheral object accordingly.
- the user may input an instruction to change the attitude of the peripheral object model SM in the three-dimensional space SP5 as an instruction to change the movement route MRr.
- the user may input an instruction to change the attitude of the peripheral object model SM by using the input device 34 to move (e.g., rotate) the peripheral object model SM on the route generation screen 3550.
- the attitude of the peripheral object model SM in the three-dimensional space SP5 may correspond to the actual attitude of the peripheral object. Therefore, when the attitude of the peripheral object model SM is changed, the user may change the attitude of the actual peripheral object accordingly.
- the user may input an instruction to change the type of the end effector model EM as an instruction to change the movement path MRr.
- the user may input an instruction to change the end effector model EM for generating the movement path MRr from a first type of end effector model EM corresponding to a first type of end effector 4 to a second type of end effector model EM corresponding to a second type of end effector 4 different from the first type, as an instruction to change the movement path MRr.
- the movement path MRr may be changed due to the change in the type of the end effector model EM.
- the movement path MRr which was not able to satisfy the condition that interference does not occur among at least one of the end effector model EM, the robot model RM, and the peripheral object model SM before at least one change such as the size and shape of the end effector model EM, may satisfy the condition that interference does not occur among at least one of the end effector model EM, the robot model RM, and the peripheral object model SM due to at least one change such as the size and shape of the end effector model EM.
- the user may also change the type of the end effector 4 attached to the robot 1.
- the user may input an instruction to change the type of the robot model RM as an instruction to change the movement path MRr. That is, the user may input an instruction to change the robot model RM for generating the movement path MRr from a first type of robot model RM corresponding to a first type of robot 1 to a second type of robot model RM corresponding to a second type of robot 1 different from the first type as an instruction to change the movement path MRr.
- the movement path MRr may be changed due to the change in the type of the robot model RM.
- the movement path MRr could not satisfy the condition that at least one of the end effector model EM, the robot model RM, and the peripheral object model SM does not interfere with each other, there is a possibility that due to the change in at least one of the size and shape of the robot model RM, the movement path MRr may satisfy the condition that at least one of the end effector model EM, the robot model RM, and the peripheral object model SM does not interfere with each other.
- the type of robot model RM is changed, the user may also change the type of robot 1 to which the end effector 4 is attached.
- the user may input an instruction to change the type of the peripheral object model SM as an instruction to change the moving path MRr. That is, the user may input an instruction to change the peripheral object model SM for generating the moving path MRr from a first type of peripheral object model SM corresponding to a first type of peripheral object to a second type of peripheral object model SM corresponding to a second type of peripheral object different from the first type as an instruction to change the moving path MRr.
- the moving path MRr may be changed due to the change in the type of the peripheral object model SM.
- the moving path MRr could not satisfy the condition that at least one of the end effector model EM, the robot model RM, and the peripheral object model SM does not interfere with each other, there is a possibility that due to the change in at least one of the size and shape of the peripheral object model SM, the moving path MRr may satisfy the condition that at least one of the end effector model EM, the robot model RM, and the peripheral object model SM does not interfere with each other.
- the type of the peripheral object model SM is changed, the user may also change the type of the actual peripheral objects located around the robot 1.
- the user may input an instruction to change the movement path MRr based on the interference information displayed on the path generation screen 3550.
- the user may input an instruction to change the movement path MRr based on the interference information displayed on the path generation screen 3550 so that interference does not occur with at least one of the end effector model EM, the robot model RM, and the surrounding object model SM.
- the user may input an instruction to change the movement path MRr based on the interference information displayed on the path generation screen 3550 so that interference does not occur with at least one of the end effector 4, the robot 1, and the surrounding object, assuming that the robot 1 is driven so that the tool center point TCP moves based on the movement path MRr.
- the route generation unit 315 changes the movement route MRr based on the user's instruction acquired in step S255 (step S256).
- the route generation unit 315 may change the generated movement route MRr.
- the route generation unit 315 may change at least a part of the movement route MRr so as to reflect the changes based on the user's instruction in the generated movement route MRr.
- the route generation unit 315 may change the movement route MRr by performing the same process as when generating the movement route MRr in step S252 (for example, performing a process based on a random-based route planning method).
- the route generation unit 315 may newly generate a new movement route MRr equivalent to the changed movement route MRr.
- the process of changing the movement route MRr in step S256 may mean at least one of a process of changing the generated movement route MRr and a process of newly generating a new movement route MRr equivalent to the changed movement route MRr. Furthermore, the process of generating a new movement route MRr corresponding to the changed movement route MRr may be the same as the process of generating the movement route MRr in step S252.
- the route generation unit 315 may repeat the processes from step S253 onwards. That is, the route generation unit 315 displays the movement route MRr changed in step S256 (step S253), determines whether the movement route MRr changed in step S256 is appropriate (step S254), and if the movement route MRr changed in step S256 is not appropriate, may change the movement route MRr again (step S256).
- step S254 if the result of the judgment in step S254 is that the generated travel route MRr is appropriate (step S254: Yes), it is assumed that there is little need to change the travel route MRr. In this case, the route generation unit 315 may end the route generation process shown in FIG. 19.
- the path generating unit 315 performs the path generating process shown in FIG. 19 before the robot 1 actually starts moving by performing the processes from step S31 to step S37 in FIG. 6. That is, the path generating unit 315 performs the path generating process shown in FIG. 19 without actually moving the robot 1 in order to generate the movement path MRr. In other words, the path generating unit 315 starts and ends the path generating process shown in FIG. 19 without moving the robot 1.
- moving the robot 1 may include moving at least a part of the robot arm 12.
- Moving the robot 1 may include moving at least one link 121 included in the robot arm 12.
- Moving the robot 1 may include moving the base 11. If the base 11 functions as an automatic guided vehicle or is placed on the automatic guided vehicle, moving the robot 1 may include moving the automatic guided vehicle.
- the path generating unit 315 can generate a movement path MRr of the tool center point TCP before the robot 1 actually starts moving, and can display the movement path MRr of the tool center point TCP on the display device 35. Therefore, by performing the processing from step S31 to step S37 in FIG. 6, the user can determine whether the generated movement path MRr is appropriate before the robot 1 actually starts moving. As a result, if it is determined that the generated movement path MRr is not appropriate, the path generating unit 315 can regenerate the movement path MRr.
- the path generating unit 315 can generate an appropriate movement path MRr without the tool center point TCP moving (i.e., the robot 1 moving) along the inappropriate movement path MRr.
- the path generator 315 can generate an appropriate movement path MRr without causing interference between the end effector 4 and a peripheral object or part of the robot 1 due to movement of the tool center point TCP along an inappropriate movement path MRr.
- the route generation unit 315 does not have to perform steps S254, S255, and S256 in FIG. 19. In other words, while the route generation unit 315 displays the travel route MRr generated in step S252, it does not have to determine whether the travel route MRr is appropriate, it does not have to determine whether the travel route MRr is appropriate, and it does not have to change the travel route MRr. The user does not have to input an instruction to change the travel route MRr to the control device 3 using the input device 34.
- performing the path generation process shown in FIG. 19 without moving the robot 1 may be considered equivalent to performing the path generation process shown in FIG. 19 while the robot 1 is stopped.
- the state in which the robot 1 is stopped may include a state in which the robot 1 is stopped before the robot 1 actually starts moving by performing the processing of steps S31 to S37 in FIG. 6.
- the state in which the robot 1 is stopped may include a state in which the robot 1 is stopped after the robot 1 actually starts moving by performing the processes of steps S31 to S37 in FIG. 6.
- the robot control unit 316 may control the robot 1 so that the tool center point TCP moves along the movement path MRr generated by the path generation unit 315 by performing the processes of steps S31 to S37 in FIG. 6. Thereafter, the robot control unit 316 may stop the robot 1 after the tool center point TCP has finished moving along the movement path MRr. Alternatively, the robot control unit 316 may stop the robot 1 while the tool center point TCP is moving along the movement path MRr. Thereafter, the path generation unit 315 may start the path generation process shown in FIG. 19. That is, the path generation unit 315 may start the path generation process shown in FIG. 19 in order to generate a new movement path MRr.
- the end effector 4 attached to the robot arm 12 may be replaced as necessary.
- the robot 1 may perform a predetermined first process on the object OBJ using a first end effector 4#11 attached to the robot arm 12. Thereafter, the end effector 4 attached to the robot arm 12 may be replaced from the first end effector 4#11 to a second end effector 4#12. Thereafter, the robot 1 may perform a predetermined second process on the object OBJ using the second end effector 4#12 attached to the robot arm 12.
- the path generating unit 315 may generate a movement path MRr (first movement path MRr#1) of the tool center point TCP when the first end effector 4#11 is attached to the robot arm 12 by performing the path generating process shown in FIG. 19.
- the path generating unit 315 may generate a movement path MRr (second movement path MRr#2) of the tool center point TCP when the second end effector 4#12 is attached to the robot arm 12 without performing the path generating process shown in FIG. 19.
- FIG. 24(a) shows the first end effector 4#11 attached to the robot arm 12 in a situation where the tool center point TCP moves along the first movement path MRr#1, using the first end effector model EM#11, which is a three-dimensional model of the first end effector 4#11.
- FIG. 24(b) an example will be described in which, when the tool center point TCP reaches position P1 on the first movement path MRr#1, the end effector 4 attached to the robot arm 12 is replaced from the first end effector 4#11 to the second end effector 4#12.
- the path generating unit 315 may generate the second movement path MRr#2 based on the generated first movement path MRr#1 and a second end effector model EM#12 that is a three-dimensional model of the second end effector 4#12.
- the path generating unit 315 may perform a simulation to drive the robot 1 so that the tool center point TCP of the second end effector 4#12 moves along the generated first movement path MRr#1, and based on the results of the simulation, generate a second movement path MRr#2 that extends from the movement start point MS of the first movement path MRr#1 to the movement end point ME of the first movement path MRr#1.
- the path generating unit 315 may perform a simulation to move the second end effector model EM#12 so that the tool center point TCP of the second end effector 4#12 moves along the generated first movement path MRr#1, and based on the results of the simulation, generate a second movement path MRr#2 that runs from the movement start point MS of the first movement path MRr#1 to the movement end point ME of the first movement path MRr#1.
- the path generating unit 315 may generate the second movement path MRr#2 based on a part of the first movement path MRr#1 and the second end effector model EM#12, instead of the entire first movement path MRr#1.
- the tool center point TCP corresponding to the first end effector 4#11 has moved a part of the first movement path MRr#1, which is a moved path portion MRrp1-3 from the movement start point MS to the position P1, while it has not yet moved another part of the first movement path MRr#1, which is an unmoved path portion MRrp1-4 from the position P1 to the movement end point ME.
- the path generating unit 315 may generate the second movement path MRr#2 based on the untraveled path portion MRrp1-4 and the second end effector model EM#12.
- the path generating unit 315 may perform a simulation of driving the robot 1 so that the tool center point TCP of the second end effector 4#12 moves along the untraveled path portion MRrp1-4, and generate the second movement path MRr#2 from the position P1 on the first movement path MRr#1 to the movement end point ME of the first movement path MRr#1 based on the result of the simulation.
- the path generation unit 315 may perform a simulation to move the second end effector model EM#12 so that the tool center point TCP of the second end effector 4#12 moves along the unmoved path portion MRrp1-4, and based on the results of the simulation, generate a second movement path MRr#2 that extends from a position P1 on the first movement path MRr#1 to the movement end point ME of the first movement path MRr#1.
- FIG. 24(b) An example of a second movement path MRr#2 from a position P1 on the first movement path MRr#1 to the movement end point ME of the first movement path MRr#1 is shown in FIG. 24(b).
- the second movement path MRr#2 from the position P1 to the movement end point ME is different from the first movement path MRr#1 from the position P1 to the movement end point ME.
- a part of the second movement path MRr#2 from the position P1 to the movement end point ME may be the same as a part of the first movement path MRr#1 from the position P1 to the movement end point ME.
- the entire second movement path MRr#2 from the position P1 to the movement end point ME may be the same as the entire first movement path MRr#1 from the position P1 to the movement end point ME.
- FIG. 24(b) shows an example of generating a second movement path MRr#2 that extends from position P1 to the movement end point ME of the first movement path MRr#1, the same can be said when a second movement path MRr#2 is generated that extends from the movement start point MS of the first movement path MRr#1 to the movement end point ME of the first movement path MRr#1.
- the path generation unit 315 may calculate a non-interference space in which no interference occurs between at least one of the first end effector model EM#11, the robot model RM, and the surrounding object model SM. Alternatively, the path generation unit 315 may calculate a non-interference space in which no interference occurs between at least one of the first end effector model EM#11, the robot model RM, and the surrounding object model SM based on the first movement path MRr#1.
- the non-interference space may be a partial space in the virtual space.
- the non-interference space may be a space that satisfies the condition that no interference occurs between at least one of the first end effector model EM#11, the robot model RM, and the surrounding object model SM when the end effector model EM (in this example, the first end effector model EM#11) moves within the non-interference space.
- the path generating unit 315 may calculate a non-interference space by arranging the first end effector model EM#11 at each position in the virtual space and repeating the operation of determining whether or not interference occurs between at least one of the first end effector model EM#11, the robot model RM, and the peripheral object model SM at multiple positions in the virtual space.
- the path generating unit 315 may calculate the non-interference space in parallel with the generation of the first movement path MRr#1, may calculate the non-interference space before the generation of the first movement path MRr#1, or may calculate the non-interference space after the generation of the first movement path MRr#1. In this case, the path generating unit 315 may generate the second movement path MRr#2 based on the second end effector 4#12 so that the second end effector model EM#12 moves in the non-interference space.
- the path generating unit 315 may generate the second movement path MRr#2 based on the first end effector model EM#11 in addition to the first movement path MRr#1 and the second end effector model EM#12. For example, the path generating unit 315 may generate the second movement path MRr#2 such that the second end effector model EM#12 moves through a non-interference space in which no interference occurs between at least one of the first end effector model EM#11, the robot model RM, and the surrounding object model SM, while taking into account the difference between the shape of the first end effector model EM#11 and the shape of the second end effector model EM#12.
- the path generating unit 315 may change (in other words, convert) a first non-interference space in which at least one of the first end effector model EM#11, the robot model RM, and the peripheral object model SM does not interfere with each other into a second non-interference space in which at least one of the second end effector model EM#12, the robot model RM, and the peripheral object model SM does not interfere with each other, based on a difference between the shape of the first end effector model EM#11 and the shape of the second end effector model EM#12, and generate a second movement path MRr#2 so that the second end effector model EM#12 moves through the second non-interference space.
- the path generating unit 315 may offset the generated first movement path MRr#1 based on a difference between the shape of the first end effector model EM#11 and the shape of the second end effector model EM#12, to generate the offset first movement path MRr#1 as the second movement path MRr#2.
- the path generating unit 315 may offset the generated first movement path MRr#1 by the difference between the overall length of the first end effector model EM#11 and the overall length of the second end effector model EM#12, and generate the offset first movement path MRr#1 as the second movement path MRr#2.
- the path generating unit 315 may generate the second movement path MRr#2 based on the surrounding object model SM in addition to the first movement path MRr#1 and the second end effector model EM#12.
- the path generating unit 315 may generate the second movement path MRr#2 based on the robot model RM in addition to the first movement path MRr#1 and the second end effector model EM#12.
- the path generating unit 315 may generate the second movement path MRr#2 based on at least one of the robot model RM and the surrounding object model SM in addition to the first movement path MRr#1 and the second end effector model EM#12.
- the path generating unit 315 may determine whether interference occurs with at least one of the second end effector model EM#12, the robot model RM, and the peripheral object model SM in order to generate the second movement path MRr#2. Specifically, the path generating unit 315 may perform a simulation of driving the robot 1 so that the tool center point TCP of the second end effector 4#12 moves along the generated first movement path MRr#1, and may determine whether interference occurs with at least one of the second end effector model EM#12, the robot model RM, and the peripheral object model SM based on the result of the simulation.
- the path generation unit 315 may perform a simulation to move the second end effector model EM#12 and the robot model RM so that the tool center point TCP of the second end effector 4#12 moves along the generated first movement path MRr#1, and determine whether or not interference has occurred between at least one of the second end effector model EM#12, the robot model RM, and the surrounding object model SM based on the results of the simulation.
- the path generating unit 315 may use the first movement path MRr#1 as the second movement path MRr#2 as is.
- the path generation unit 315 may generate a second movement path MRr#2 so that interference between the second end effector model EM#12 and the peripheral object model SM does not occur.
- the path generation unit 315 may generate a movement path including the modified first movement path MRr#1 as the second movement path MRr#2 by modifying (in other words, changing) at least a portion of the first movement path MRr#1 so that interference between the second end effector model EM#12 and the peripheral object model SM does not occur.
- the route generating unit 315 can generate the second movement route MRr#2 based on the already generated first movement route MRr#1.
- the route generating unit 315 can generate the second movement route MRr#2 by modifying the already generated first movement route MRr#1.
- the route generating unit 315 can use the already generated first movement route MRr#1 as the second movement route MRr#2 as it is. Therefore, compared to the case where the second movement route MRr#2 is newly generated from scratch without being based on the first movement route MRr#1, the route generating unit 315 can reduce the processing load for generating the second movement route MRr#2. Furthermore, compared to the case where the second movement route MRr#2 is newly generated from scratch without being based on the first movement route MRr#1, the route generating unit 315 can reduce the time required to generate the second movement route MRr#2.
- the path generation unit 315 may modify the interference path portion MRrp1-2, which is a path portion of the first movement path MRr#1 that causes interference with at least one of the second end effector model EM#12, the robot model RM, and the surrounding object model SM.
- the interference path portion MRrp1-2 is a path that satisfies the condition that interference occurs with at least one of the second end effector model EM#12, the robot model RM, and the surrounding object model SM when the tool center point TCP of the second end effector 4#12 moves along the interference path portion MRrp1-2.
- the path generation unit 315 may modify the non-interference path portion MRrp1-1, which is a path portion of the first movement path MRr#1 that does not cause interference with at least one of the second end effector model EM#12, the robot model RM, and the surrounding object model SM.
- the non-interference path portion MRrp1-1 is a path that satisfies the condition that no interference occurs with at least one of the second end effector model EM#12, the robot model RM, and the peripheral object model SM when the tool center point TCP of the second end effector 4#12 moves along the non-interference path portion MRrp1-1.
- the second movement path MRr#2 may include the uncorrected non-interference path portion MRrp1-1 and the corrected interference path portion MRrp1-2.
- the path generation unit 315 can reduce the processing load for generating the second movement path MRr#2 compared to the case where the second movement path MRr#2 is generated by changing the entire first movement path MRr#1.
- the path generation unit 315 can reduce the time required to generate the second movement path MRr#2 compared to the case where the second movement path MRr#2 is generated by changing the entire first movement path MRr#1.
- the second end effector 4#12 moves according to (in other words, based on) the unmoved path portion MRrp1-4 along which the tool center point TCP corresponding to the first end effector 4#11 of the first movement path MRr#1 has not yet moved.
- the second end effector 4#12 moves so that the tool center point TCP corresponding to the second end effector 4#12 moves along the unmoved path portion MRrp1-4.
- the second end effector 4#12 moves so that the tool center point TCP corresponding to the second end effector 4#12 moves from the position P1 to the movement end point ME.
- the second end effector 4#12 does not have to move according to (in other words, based on) the moved path portion MRrp1-3 to which the tool center point TCP corresponding to the first end effector 4#11 of the first movement path MRr#1 has already moved.
- the second end effector 4#12 does not have to move so that the tool center point TCP corresponding to the second end effector 4#12 moves on the moved path portion MRrp1-3.
- the second end effector 4#12 does not have to move so that the tool center point TCP corresponding to the second end effector 4#12 moves from the movement start point MS to the position P1.
- the route generating unit 315 may modify at least a part of the untraveled route portion MRrp1-4, while not modifying the traveled route portion MRrp1-3.
- the route generating unit 315 may modify at least a part of the route portion corresponding to the above-mentioned interference route portion MRrp1-2 among the untraveled route portion MRrp1-4.
- the second travel route MRr#2 may include the unmodified traveled route portion MRrp1-3 and the untraveled route portion MRrp1-4, at least a part of which has been modified.
- the route generating unit 315 can reduce the processing load for generating the second travel route MRr#2, compared to the case where the second travel route MRr#2 is generated by changing the entire first travel route MRr#1.
- the route generation unit 315 can reduce the time required to generate the second movement route MRr#2.
- the path generating unit 315 may generate the second movement path MRr#2 after the end effector 4 attached to the robot arm 12 is replaced from the first end effector 4#11 to the second end effector 4#12. In other words, the path generating unit 315 may generate the second movement path MRr#2 after the second end effector 4#12 is attached to the robot arm 12.
- the path generating unit 315 may generate the second movement path MRr#2 in advance before the end effector 4 attached to the robot arm 12 is replaced from the first end effector 4#11 to the second end effector 4#12. In other words, the path generating unit 315 may generate the second movement path MRr#2 in advance before the second end effector 4#12 is attached to the robot arm 12. In this case, the path generating unit 315 may generate the second movement path MRr#2 after the first end effector 4#11 attached to the robot arm 12 starts moving according to the first movement path MRr#1.
- the path generating unit 315 may generate the second movement path MRr#2 before the first end effector 4#11 attached to the robot arm 12 starts moving according to the first movement path MRr#1.
- the first movement path MRr#1 may usually be generated before the first end effector 4#11 attached to the robot arm 12 starts moving. This is because the first end effector 4#11 cannot start moving unless the first movement path MRr#1 has already been generated.
- the path generating unit 315 may generate the second moving path MRr#2 in the above-described manner. For example, even when the end effector 4 attached to the robot arm 12 is replaced after the tool center point TCP has completed moving along the first moving path MRr#1, the path generating unit 315 may generate the second moving path MRr#2 based on the already-generated first moving path MRr#1 and the second end effector model EM#12, similar to the case when the end effector 4 attached to the robot arm 12 is replaced when the tool center point TCP reaches position P1 on the first moving path MRr#1.
- the path generating unit 315 may generate the second movement path MRr#2 by using the above-described non-interference space. For example, even when the end effector 4 attached to the robot arm 12 is replaced after the tool center point TCP has completed moving along the first movement path MRr#1, the path generating unit 315 may generate the second movement path MRr#2 based on the difference between the shape of the first end effector model EM#11 and the shape of the second end effector model EM#12 as described above.
- the path generating unit 315 may generate the second movement path MRr#2 based on the already generated first movement path MRr#1, the second end effector model EM#12, and at least one of the robot model RM and the surrounding object model SM, in the same way as when the end effector 4 attached to the robot arm 12 is replaced.
- both the first movement path MRr#1 and the second movement path MRr#2 indicate the movement path of the tool center point TCP used as the movement reference point MBP.
- the first movement path MRr#1 may indicate the movement path of the first movement reference point MBP
- the second movement path MRr#2 may indicate the movement path of the second movement reference point MBP different from the first movement reference point MBP.
- the path generating unit 315 may generate the first movement path MRr#1, which is the movement path of the first movement reference point MBP when the first end effector 4#11 is attached to the robot arm 12, by performing the path generating process shown in FIG. 19.
- the path generating unit 315 may generate a second movement path MRr#2, which is the movement path MRr of the second movement reference point MBP when the second end effector 4#12 is attached to the robot arm 12, based on the first movement path MRr#1 that has already been generated and the second end effector model EM#12, without performing the path generation process shown in FIG. 19.
- the path generating unit 315 when the end effector 4 attached to the robot arm 12 is replaced, the path generating unit 315 generates the second movement path MRr#2 based on the already generated first movement path MRr#1.
- the robot 1 to which the end effector 4 is attached may be replaced.
- the first robot 1#1 to which the end effector 4 is attached may perform a predetermined process on the object OBJ.
- the end effector 4 may then be detached from the first robot 1#1 and attached to the second robot 1#2.
- the second robot 1#2 to which the end effector 4 is attached may then perform a predetermined process on the object OBJ.
- the path generating unit 315 may generate a movement path MRr (first movement path MRr#11) of the tool center point TCP of the end effector 4 attached to the first robot 1#1 by performing the path generating process shown in Fig. 19.
- the path generating unit 315 may generate a movement path MRr (second movement path MRr#12) of the tool center point TCP of the end effector 4 attached to the second robot 1#2 without performing the path generating process shown in Fig. 19.
- the path generating unit 315 may generate the second movement path MRr#12 based on the generated first movement path MRr#11 and the second robot model RM#2 which is a three-dimensional model of the second robot model #2.
- the operation of generating the second movement path MRr#12 based on the already generated first movement path MRr#11 and the second robot model RM#2 may be the same as the operation of generating the second movement path MRr#2 based on the already generated first movement path MRr#1 and the second end effector model EM#12 described above.
- the operation of generating the second movement path MRr#12 based on the already generated first movement path MRr#11 and the second robot model RM#2 may be the same as the operation of generating the second movement path MRr#2 based on the already generated first movement path MRr#1 and the second end effector model EM#12 described above, except that the object to be replaced is the robot 1 rather than the end effector 4.
- the route generation unit 315 can reduce the processing load for generating the second movement route MRr#12, compared to when the second movement route MRr#12 is generated from scratch without being based on the first movement route MRr#11. Furthermore, the route generation unit 315 can reduce the time required to generate the second movement route MRr#12, compared to when the second movement route MRr#12 is generated from scratch without being based on the first movement route MRr#11.
- step S25 of Fig. 6 the control device 3 performs the path generation processing shown in Fig. 19 using the path generation unit 315 to generate a movement path MRr of the tool center point TCP that moves by the rough movement operation.
- the control device 3 generates the movement path MRr using the end effector model EM, the robot model RM, and the peripheral object model SM so that interference does not occur among at least one of the end effector model EM, the robot model RM, and the peripheral object model SM.
- the control device 3 generates the movement path MRr so that interference does not occur among at least one of the end effector 4, the robot 1, and the peripheral objects when it is assumed that the robot 1 is actually driven.
- the control device 3 uses the robot control unit 316 to generate a movement path MRf for the tool center point TCP that moves by the fine movement operation.
- the control device 3 when generating the movement path MRf, the control device 3 generates the movement path MRf such that interference does not occur with at least one of the end effector 4, the robot 1, and the surrounding objects, just as when generating the movement path MRr.
- the control device 3 uses the end effector model EM, the robot model RM, and the surrounding object model SM to generate the movement path MRf such that interference does not occur with at least one of the end effector model EM, the robot model RM, and the surrounding object model SM.
- each of the movement paths MRr and MRf may be considered to be a part of the movement path from the waiting position PW to the processing target position PTP.
- Each of the movement paths MRr and MRf may be considered to be a part of the entire movement path along which the end effector 4 must move in order to perform a specified processing on the object OBJ (e.g., the workpiece W).
- the movement path MRr may be considered to be a movement path farther from the object OBJ than the movement path MRf.
- the movement path MRr may be considered to be a movement path farther from the processing target position PTP than the movement path MRf. It may be considered to be a movement path farther from the object OBJ than the movement path MRf.
- the movement route MRf may be considered to be closer to the processing target position PTP than the movement route MRr.
- the control device 3 may use an extended end effector model EM_ex instead of the end effector model EM.
- the extended end effector model EM_ex is a three-dimensional model obtained by extending (in other words, enlarging) the end effector model EM.
- the extended end effector model EM_ex may be a three-dimensional model obtained by extending the entire end effector model EM.
- the extended end effector model EM_ex may be a three-dimensional model obtained by extending a part of the end effector model EM.
- the extended end effector model EM_ex may be a three-dimensional model obtained by extending at least a part of the end effector model EM.
- the end effector model EM and the extended end effector model EM_ex may both be considered to be three-dimensional models based on the three-dimensional model of the end effector 4.
- the extension of the end effector model EM may mean an increase in the volume of at least a portion of the end effector model EM.
- the extended end effector model EM_ex may be considered to be a three-dimensional model obtained by increasing the volume of at least a portion of the end effector model EM.
- the three-dimensional model obtained by increasing the volume of at least a portion of the end effector model EM may be considered to be equivalent to the three-dimensional model of the end effector 4 with at least a portion of the increased volume.
- the volume of the end effector model EM may literally mean the volume of the end effector model EM that is solid.
- the end effector model EM is a surface model, which is a three-dimensional model that represents the end effector 4 using a surface that corresponds to at least a part of the surface of the end effector 4
- the volume of the end effector model EM may mean the volume of the space enclosed by the surface represented by the end effector model EM.
- "moving each part of the surface in the desired expansion direction" as described in the next paragraph may be considered equivalent to increasing the volume of the end effector model EM, which is a surface model.
- Expanding the end effector model EM may mean moving each part of the surface of the end effector model EM along a desired expansion direction.
- expanding the end effector model EM may mean moving each part of the surface of the end effector model EM along the normal to the surface of the end effector model EM at each part.
- the extended end effector model EM_ex may mean a three-dimensional model defined using each part after movement.
- Expansion of the end effector model EM may mean moving each part of the surface of the end effector model EM by a desired expansion amount exa.
- the extended end effector model EM_ex may mean a three-dimensional model defined using each part after movement.
- the expansion amount exa may be a fixed value.
- the expansion amount exa may be a changeable variable value.
- the control device 3 may generate each of the movement paths MRr and MRf that satisfy the condition that no interference occurs between at least one of the extended end effector model EM_ex, the robot model RM, and the peripheral object model SM. In this case, the control device 3 can more appropriately prevent the occurrence of interference (particularly actual contact) of the end effector 4 during the period when the robot 1 is actually moving, compared to when the extended end effector model EM_ex is not used. Specifically, as shown in the upper part of FIG.
- the control device 3 may generate each of the movement paths MRr and MRf such that the distance between the extended end effector model EM_ex and each of the robot model RM and the peripheral object model SM is longer than a predetermined allowable distance AD2.
- the end effector 4 moves so that the distance between the end effector 4 and each of the robot 1 and the peripheral object is longer than the sum of the predetermined allowable distance AD1 and the extension amount exa.
- the control device 3 may generate each of the movement paths MRr and MRf so that the distance between the end effector model EM and each of the robot model RM and the peripheral object model SM is longer than the predetermined allowable distance AD2.
- the end effector 4 moves so that the distance between the end effector 4 and each of the robot 1 and the peripheral object is longer than the predetermined allowable distance AD1.
- the control device 3 can more appropriately prevent the occurrence of interference with the end effector 4 compared to when the extended end effector model EM_ex is not used. In other words, the control device 3 can more appropriately prevent the occurrence of interference between the robot 1 and each of the peripheral objects and the end effector 4.
- the control device 3 can reduce the possibility of interference (particularly actual contact) between the peripheral object and the end effector 4.
- the control device 3 places the peripheral object model SM at a position corresponding to the actual position of the peripheral object (in this case, the position assumed by the control device 3 as the actual position of the peripheral object) in the virtual space, and then generates each of the movement paths MRr and MRf that satisfy the condition that no interference occurs between at least one of the end effector model EM, the robot model RM, and the peripheral object model SM.
- the peripheral object model SM may be located at a position different from the position corresponding to the actual position of the peripheral object in the virtual space.
- the expansion amount exa may be set by the user.
- the control device 3 may control the display device 35 to display a display screen that the user can operate to set the expansion amount exa.
- the control device 3 may control the display device 35 to display a route generation screen 3550 that includes display objects that the user can operate to set the expansion amount exa.
- the user may use the input device 34 to input an instruction to set the expansion amount exa to the control device 3 via a display screen (e.g., the route generation screen 3550) displayed on the display device 35.
- the expansion amount exa may be set by the control device 3 (e.g., at least one of the path generating unit 315 and the robot control unit 316).
- the control device 3 may set the expansion amount exa capable of preventing interference between the end effector 4 and a peripheral object based on the characteristics (e.g., at least one of size and shape) of the peripheral object that may interfere with the end effector 4.
- the control device 3 may set the expansion amount exa capable of preventing interference between the end effector 4 and a peripheral object based on the characteristics (e.g., at least one of size and shape) of the peripheral object model SM that may interfere with the end effector model EM.
- control device 3 may set the expansion amount exa capable of preventing interference between the end effector 4 and the robot 1 based on the characteristics (e.g., at least one of size and shape) of the robot 1 that may interfere with the end effector 4.
- control device 3 may set an expansion amount exa that can prevent interference between the end effector 4 and the robot 1 based on the characteristics (e.g., at least one of the size and shape) of the robot model RM that may interfere with the end effector 4.
- the expansion amount exa used when generating the movement path MRr is different from the expansion amount exa used when generating the movement path MRf. That is, the control device 3 may use a first expansion amount exa1 as the expansion amount exa used when generating the movement path MRr, and a second expansion amount exa2 different from the first expansion amount exa1 as the expansion amount exa used when generating the movement path MRf.
- the expansion amount exa used when generating the movement path MRr and the expansion amount exa used when generating the movement path MRf may be the same.
- control device 3 may use a first expansion amount exa1 as the expansion amount exa used when generating the movement path MRr, and a second expansion amount exa2 the same as the first expansion amount exa1 as the expansion amount exa used when generating the movement path MRf.
- the expansion amount exa used when generating the movement path MRr may be greater than the expansion amount exa used when generating the movement path MRf.
- the expansion amount exa used when generating the movement path MRf may be less than the expansion amount exa used when generating the movement path MRr.
- the control device 3 may use a first expansion amount exa1 that is greater than the second expansion amount exa2 as the expansion amount exa used when generating the movement path MRr, and use a second expansion amount exa2 that is less than the first expansion amount exa1 as the expansion amount exa used when generating the movement path MRf.
- the control device 3 may use a first expansion amount exa1 that is greater than a second expansion amount exa2 described below when generating a movement path MRr for bringing the end effector 4 closer to the storage box CB in which the workpiece W is stored.
- the control device 3 may use a second expansion amount exa2 that is less than the first expansion amount exa1 described above when generating a movement path MRf for bringing the end effector 4 closer to the workpiece W so as to actually hold the workpiece W.
- the control device 3 may use a relatively large first extension amount exa1 when generating a movement path MRr for bringing the end effector 4 within a certain distance of the pallet PLT on which the workpiece W is to be placed (released).
- the control device 3 may use a relatively small second extension amount exa2 when generating a movement path MRf for bringing the end effector 4 even closer to the vicinity of the pallet PLT so as to actually place the workpiece W.
- the control device 3 may use a relatively large first extension amount exa1 when generating a movement path MRr for bringing the end effector 4 within a certain distance of the second workpiece W into which the first workpiece W is to be fitted.
- the control device 3 may use a relatively small second extension amount exa2 when generating a movement path MRf for bringing the end effector 4 close to the vicinity of the second workpiece W so as to actually fit the first workpiece W into the second workpiece W.
- the control device 3 since the first expansion amount exa1 is greater than the second expansion amount exa2, as shown in FIG. 28(a), the control device 3 generates the movement path MRr using the extended end effector model EM_ex with a large expansion amount exa.
- the interval between the end effector model EM and at least one of the robot model RM and the peripheral object model SM is wider by the expansion amount exa, compared to when the movement path MRr is generated based on the normal end effector model EM.
- the control device 3 can appropriately prevent interference with the end effector 4 moving along the movement path MRr.
- the control device 3 when generating the movement path MRr, the control device 3 generates the movement path MRr using an extended end effector model EM_ex with a large extension amount exa in order to prioritize preventing interference between the end effector 4 moving along the movement path MRr.
- Figure 28 (a) shows an example in which the movement path MRr is a movement path from the outside of the storage box CB to the inside of the storage box CB.
- the control device 3 may generate the movement path MRr using an extended end effector model EM_ex with a large extension amount exa in order to prioritize preventing interference between the end effector 4 moving along the movement path MRr and the storage box CB.
- the control device 3 may generate the movement path MRr using an extended end effector model EM_ex having a large extension amount exa in order to prioritize prevention of interference between the end effector 4 moving along the movement path MRr and the side wall of the storage box CB.
- the control device 3 may generate the movement path MRr using an extended end effector model EM_ex having a large extension amount exa in order to prioritize prevention of interference between the end effector 4 moving along the movement path MRr and the side wall of the storage box CB.
- the control device 3 may generate the movement path MRr using an extended end effector model EM_ex with a large extension amount exa in order to prioritize preventing interference between the end effector 4 moving along the movement path MRr and the side wall of the storage box CB.
- the control device 3 since the second expansion amount exa2 is smaller than the first expansion amount exa1, as shown in FIG. 28(b), the control device 3 generates a movement path MRf using an extended end effector model EM_ex with a small expansion amount exa.
- the movement path MRf is a movement path for moving the tool center point TCP to a processing target position PTP where the end effector 4 performs a predetermined processing on the object OBJ. Therefore, during the period when the end effector 4 is moving along the movement path MRf, there is a high possibility that a peripheral object is located near the end effector 4.
- control device 3 may not be able to generate a movement path MRf that satisfies the condition that no interference occurs between at least one of the extended end effector model EM_ex, the robot model RM, and the peripheral object model SM. For example, as shown in FIG.
- the control device 3 when generating a movement path MRf in which the end effector 4 holds a workpiece W arranged at a corner of a storage box CB, which is an example of a peripheral object, the control device 3 may not be able to generate a movement path MRf that satisfies the condition that the distance between the extended end effector model EM_ex and the peripheral object model SM, which is a three-dimensional model of the storage box CB, is longer than a predetermined allowable distance AD2. For example, as shown in FIG.
- the control device 3 may not be able to generate a movement path MRf that satisfies the condition that the distance between the extended end effector model EM_ex and the two peripheral object models SM, which are three-dimensional models of the second workpiece W#2 and the third workpiece W#3 regarded as peripheral objects, is longer than a predetermined allowable distance AD2.
- the end effector 4 may not be able to perform a predetermined process on the object OBJ.
- a second expansion amount exa2 smaller than the first expansion amount exa1 is used in order to prevent the end effector 4 from interfering with the movement path MRf and ensure an opportunity to generate the movement path MRf.
- the smaller the expansion amount exp the higher the possibility that it is determined that the distance between the extended end effector model EM_ex and the peripheral object model SM is longer than the predetermined allowable distance AD2. Therefore, the control device 3 can appropriately ensure the opportunity to generate a movement path MRf that satisfies the condition that no interference occurs with the extended end effector model EM_ex.
- the second extension amount exa2 may be zero.
- the control device 3 can maximize the chances of generating a movement path MRf that satisfies the condition that interference with the extended end effector model EM_ex does not occur.
- the second extension amount exa2 is zero, if the condition that interference with the extended end effector model EM_ex does not occur is satisfied, the distance between the end effector model EM and the surrounding object model SM will still be longer than the predetermined allowable distance AD2. Therefore, the control device 3 can appropriately generate a movement path MRf that can appropriately prevent interference with the end effector 4.
- the control device 3 may change the allowable distance AD2 used as a threshold value. Specifically, the control device 3 may change the allowable distance AD2 so that the allowable distance AD2 used when generating the movement path MRr is different from the allowable distance AD2 used when generating the movement path MRf. As an example, the control device 3 may change the allowable distance AD2 so that the allowable distance AD2 used when generating the movement path MRr is greater than the allowable distance AD2 used when generating the movement path MRf.
- control device 3 may change the allowable distance AD2 so that the allowable distance AD2 used when generating the movement path MRf is smaller than the allowable distance AD2 used when generating the movement path MRr. As a result, it is possible to enjoy the same effect as that which can be enjoyed when the expansion amount exa is changed.
- the part of the end effector model EM that is extended when generating the movement path MRr may be different from the part of the end effector model EM that is extended when generating the movement path MRf.
- the control device 3 may use an extended end effector model EM_ex obtained by extending the entire end effector model EM to generate either the movement path MRr or MRf, and as shown in FIG. 29(b), the control device 3 may use an extended end effector model EM_ex obtained by extending a part of the end effector model EM to generate the other of the movement paths MRr or MRf.
- FIG. 29(a) the control device 3 may use an extended end effector model EM_ex obtained by extending a part of the end effector model EM to generate the other of the movement paths MRr or MRf.
- the control device 3 may use an extended end effector model EM_ex obtained by extending a first model portion of the end effector model EM to generate one of the movement paths MRr and MRf, and as shown in FIG. 29(d), the control device 3 may use an extended end effector model EM_ex obtained by extending a second model portion of the end effector model EM that is different from the first model portion to generate the other of the movement paths MRr and MRf.
- the control device 3 can change the path generation conditions used to generate a part of the entire movement path along which the tool center point TCP should move when the end effector 4 performs a predetermined process on the workpiece W, for each of the multiple movement paths obtained by dividing the entire movement path.
- the control device 3 can change the path generation conditions as necessary in the process of generating the entire movement path defined by one recipe file 321.
- the control device 3 can change the path generation conditions so that the path generation conditions used to generate the movement path MRr and the path generation conditions used to generate the movement path MRf are different.
- Each of the multiple movement paths obtained by dividing the entire movement path may be referred to as a partial path.
- each of the movement paths MRr and MRf may be referred to as a partial path.
- the control device 3 can flexibly generate an optimal movement path for each position on the entire movement path.
- path generation conditions include the above-mentioned expansion amount exa and at least one of the parts of the end effector model EM to be expanded.
- the path generation conditions may include conditions other than the above-mentioned expansion amount exa and at least one of the parts of the end effector model EM to be expanded.
- the path generation conditions may include the position of the robot model RM in the virtual space.
- the path generation conditions may include the attitude of the robot model RM in the virtual space.
- the path generation conditions may include the position of the peripheral object model SM in the virtual space.
- the path generation conditions may include the attitude of the peripheral object model SM in the virtual space.
- the path generation conditions may include the initial position of the end effector model EM in the virtual space.
- the path generation conditions may include the initial attitude of the end effector model EM in the virtual space.
- the path generation conditions may include the movement start point MS.
- the path generation conditions may include the movement end point ME.
- the route generation condition may include a travel via point MI.
- the type of route generation condition may be considered to be the route generation condition for the entire travel route.
- the control device 3 uses the extended end effector model EM_ex generated by extending the end effector model EM when generating each of the movement paths MRr and MRf.
- the control device 3 may use an extended robot model generated by extending the robot model RM in addition to or instead of the extended end effector model EM_ex when generating each of the movement paths MRr and MRf.
- the process of extending the robot model RM may be the same as the process of extending the end effector model EM.
- the extension amount exa of the extended robot model used when generating the movement path MRr may be different from the extension amount exa of the extended robot model used when generating the movement path MRf.
- the extension amount exa of the extended robot model used when generating the movement path MRr may be greater than the extension amount exa of the extended robot model used when generating the movement path MRf.
- the part of the robot model RM extended when generating the movement path MRr may be different from the part of the robot model RM extended when generating the movement path MRf.
- both the robot model RM and the extended robot model may be considered to be three-dimensional models based on the three-dimensional model of the robot 1.
- the control device 3 may use an extended peripheral object model generated by extending the peripheral object model SM in addition to or instead of at least one of the extended end effector model EM_ex and the extended robot model.
- the process of extending the peripheral object model SM may be the same as the process of extending the end effector model EM.
- the extension amount exa of the extended peripheral object model used when generating the movement path MRr may be different from the extension amount exa of the extended peripheral object model used when generating the movement path MRf.
- the extension amount exa of the extended peripheral object model used when generating the movement path MRr may be greater than the extension amount exa of the extended peripheral object model used when generating the movement path MRf.
- the part of the peripheral object model SM that is extended when generating the movement path MRr may be different from the part of the peripheral object model SM that is extended when generating the movement path MRf.
- both the peripheral object model SM and the extended peripheral object model may be considered to be three-dimensional models based on three-dimensional models of peripheral objects.
- the control device 3 uses the extended end effector model EM_ex generated by extending the end effector model EM.
- extending the end effector model EM may be considered as an example of modifying the end effector model EM.
- the control device 3 may use a modified end effector model generated by modifying the end effector model EM in addition to or instead of the extended end effector model EM_ex.
- the control device 3 may use a modified end effector model generated by modifying the size of the end effector model EM.
- the control device 3 may use a modified end effector model generated by modifying the shape of the end effector model EM.
- extending the robot model RM may be considered to be an example of a modification generated by modifying the robot model RM.
- the control device 3 may use a modified robot model generated by modifying the robot model RM in addition to or instead of the extended robot model.
- the control device 3 may use a modified robot model generated by modifying the size of the robot model RM.
- the control device 3 may use a modified robot model generated by modifying the shape of the robot model RM.
- the control device 3 may use a modified peripheral object model generated by modifying the peripheral object model sM in addition to or instead of the extended peripheral object model.
- the control device 3 may use a modified peripheral object model generated by modifying the size of the peripheral object model SM.
- the control device 3 may use a modified peripheral object model generated by modifying the shape of the peripheral object model SM.
- the control device 3 generates two movement paths, MRr and MRf, as at least a part of the entire movement path along which the tool center point TCP should move when the end effector 4 performs a predetermined process on the workpiece W.
- the control device 3 divides the entire movement path along which the tool center point TCP should move when the end effector 4 performs a predetermined process on the workpiece W into two movement paths (specifically, movement paths MRr and MRf) and generates the two movement paths.
- the control device 3 may divide the entire movement path along which the tool center point TCP should move when the end effector 4 performs a predetermined process on the workpiece W into three or more movement paths and generate the three or more movement paths.
- control device 3 may divide the movement path MRr into two or more movement paths and generate the two or more movement paths and the movement path MRf.
- control device 3 may divide the movement path MRf into two or more movement paths and generate the two or more movement paths MRf and the movement path MRr.
- control device 3 may divide the entire movement path along which the tool center point TCP must move when the end effector 4 performs a predetermined process on the workpiece W into three movement paths, including a movement path from the waiting position PW to a first approach position that is closer to the object OBJ than the waiting position PW, a movement path from the first approach position to a second approach position that is closer to the object OBJ than the first approach position, and a movement path from the second approach position to a processing target position PTP that is closer to the object OBJ than the second approach position, and generate the three or more movement paths and the movement path MRf.
- control device 3 may divide the entire movement path along which the tool center point TCP must move when the end effector 4 performs a specified process on the workpiece W into three or more movement paths, and generate the three or more movement paths.
- the control device 3 performs the path generation process shown in FIG. 19 using the path generation unit 315 in step S25 of FIG. 6 to generate the movement path MRr of the tool center point TCP that moves by the rough movement operation.
- the control device 3 may generate the movement path MRr using a method different from the method of performing the path generation process shown in FIG. 19.
- the control device 3 may generate the movement path MRr using a method based on teaching with a teaching pendant.
- the control device 3 may generate the movement path MRr using a method based on direct teaching.
- the control device 3 may generate the movement path MRr using a method based on image data acquired from an imaging device that images the robot 1 from a position that allows a bird's-eye view of the robot 1.
- control device 3 may generate the movement path MRr using a method different from the method of performing the path generation process shown in FIG. 19.
- the path generation unit 315 performs the path generation process shown in Fig. 19 before the robot 1 actually starts moving by performing the processes from step S31 to step S37 in Fig. 6. In other words, the path generation unit 315 generates the entire movement path MRr before the robot 1 actually starts moving by performing the processes from step S31 to step S37 in Fig. 6.
- the path generating unit 315 may generate a part of the movement path MRr in advance by performing at least a part of the path generating process shown in Fig. 19 (e.g., the process of step S252) in step S25 of Fig. 6 before the robot 1 actually starts moving by performing the processes of steps S31 to S37 of Fig. 6.
- the path generating unit 315 may generate another part of the movement path MRr by performing at least a part of the path generating process shown in Fig. 19 (e.g., the process of step S252).
- the robot control unit 316 may generate the other part of the movement path MRr by performing at least a part of the path generating process shown in Fig. 19 instead of the path generating unit 315.
- the robot 1 moves the first tool center point TCP along the movement path MRr from the pre-wait position PW_pre to the fine approach position PF, then the tool center point TCP moves along the movement path MRf from the first fine approach position PF to the processing target position PTP, then the end effector 4 performs a specified processing at the processing target position PTP, then the tool center point TCP moves along the movement path MRf from the processing target position PTP to a second fine approach position PF that is the same as or different from the first fine approach position PF, and then moves along the movement path MRr from the second fine approach position PF to the post-wait position PW_post.
- the path generating unit 315 may generate in advance a movement path MRr from the pre-wait position PW_pre to the first fine approach position PF by performing at least a part of the path generating process shown in Fig. 19 in step S25 of Fig. 6 before the robot 1 actually starts moving by performing the processes of steps S31 to S37 of Fig. 6.
- the path generating unit 315 does not need to generate in advance a movement path MRr from the second fine approach position PF to the post-wait position PW_post before the robot 1 actually starts moving so that the tool center point TCP moves along the movement path MRr from the pre-wait position PW_pre to the first fine approach position PF.
- the path generating unit 315 may generate the movement path MRr from the second fine approach position PF to the post-standby position PW_post by performing at least a part of the path generating processing shown in Fig. 19. For example, while the robot 1 is moving so that the tool center point TCP moves along the movement path MRr from the pre-standby position PW_pre to the first fine approach position PF, the path generating unit 315 may generate the movement path MRr from the second fine approach position PF to the post-standby position PW_post.
- the path generating unit 315 may generate a movement path MRr from the second fine approach position PF to the post-waiting position PW_post.
- the path generating unit 315 may generate a movement path MRr from the second fine approach position PF to the post-waiting position PW_post.
- the path generating unit 315 may acquire at least one of the image data IMG_2D and IMG_3D from the imaging device 2, calculate at least one of the positions and orientations of the surrounding objects based on at least one of the image data IMG_2D and IMG_3D, and generate a portion of the movement path MRr based on at least one of the positions and orientations of the surrounding objects in addition to the end effector model EM, the robot model RM, and the surrounding object model SM.
- at least one of the cameras 21 and 22 may capture an image of the surrounding objects.
- the path generating unit 315 can calculate at least one of the position and the orientation of the peripheral object based on at least one of the image data IMG_2D and IMG_3D.
- the process of calculating at least one of the position and the orientation of the peripheral object based on at least one of the image data IMG_2D and IMG_3D may be the same as the process of calculating at least one of the position and the orientation of the object OBJ based on at least one of the image data IMG_2D and IMG_3D, so a detailed description thereof will be omitted.
- the path generating unit 315 may set at least one of the position and the orientation of the peripheral object model SM in the predetermined virtual space based on at least one of the calculation results of the position and the orientation of the peripheral object.
- the path generating unit 315 may set at least one of the position and the orientation of the peripheral object model SM in the predetermined virtual space by adjusting at least one of the position and the orientation of the peripheral object model SM in the predetermined virtual space based on at least one of the calculation results of the position and the orientation of the peripheral object.
- at least one of the position and orientation of the peripheral object model SM in the predetermined virtual space approaches at least one of the actual position and actual orientation of the peripheral object.
- at least one of the position and orientation of the peripheral object model SM in the predetermined virtual space may match at least one of the actual position and actual orientation of the peripheral object.
- the path generating unit 315 may generate a part of the movement path MRr based on the end effector model EM, the robot model RM, and the peripheral object model SM in which at least one of the position and orientation is adjusted based on the calculation result of at least one of the position and orientation of the peripheral object.
- the path generating unit 315 can appropriately generate a movement path MRr that satisfies the condition that no interference occurs between at least one of the end effector 4, the robot 1, and the peripheral object.
- the path generation unit 315 may acquire image data IMG_3D from the imaging device 2 and generate a point cloud model of the surrounding objects (or other types of three-dimensional models such as polygon models, mesh models, or solid models, the same as in the third modified example of the path generation process below) based on the image data IMG_3D.
- the path generation unit 315 may generate the above-mentioned three-dimensional position data WSD based on the image data IMG_3D and generate a point cloud model of the surrounding objects based on the three-dimensional position data WSD.
- the path generation unit 315 may perform a crop process on the image data IMG_3D to extract an image portion in which the surrounding objects are reflected from the image indicated by the image data IMG_3D, and generate a point cloud model of the surrounding objects based on the image data IMG_3D after the crop process.
- the path generating unit 315 may use the generated point cloud model of the peripheral object as the peripheral object model SM. That is, the path generating unit 315 may use a three-dimensional model based on the three-dimensional position data WSD as the peripheral object model SM.
- the path generating unit 315 may place the peripheral object model SM, which is a point cloud model of the peripheral object generated based on the image data IMG_3D, in a predetermined virtual space.
- the point cloud model includes a plurality of points associated with three-dimensional coordinates in a global coordinate system.
- the path generating unit 315 may convert the three-dimensional coordinates of each of the plurality of points included in the point cloud model into three-dimensional coordinates in the virtual space, and then place each of the plurality of points included in the point cloud model at a position indicated by the three-dimensional coordinates obtained by the conversion in the virtual space, thereby placing the peripheral object model SM, which is a point cloud model, in the virtual space.
- the path generating unit 315 may generate a part of the movement path MRr based on the end effector model EM, the robot model RM, and the peripheral object model SM, which is a point cloud model of the peripheral object generated based on the image data IMG_3D.
- the path generating unit 315 can appropriately generate the movement path MRr that satisfies the condition that no interference occurs between at least one of the end effector 4, the robot 1, and the peripheral object.
- the path generation unit 315 may obtain at least one of the image data IMG_2D and IMG_3D from the imaging device 2, calculate at least one of the positions and orientations of the surrounding objects based on at least one of the image data IMG_2D and IMG_3D, and generate a part of the movement path MRr based on at least one of the positions and orientations of the surrounding objects in addition to the end effector model EM, the robot model RM, and the surrounding object model SM.
- the path generation unit 315 may obtain the image data IMG_3D from the imaging device 2, generate a point cloud model of the surrounding objects as the surrounding object model SM based on the image data IMG_3D, and generate a part of the movement path MRr based on the end effector model EM, the robot model RM, and the surrounding object model SM, which is the point cloud model of the surrounding objects generated based on the image data IMG_3D.
- the robot control unit 316 may calculate at least one of the positions and orientations of the peripheral objects based on at least one of the image data IMG_2D and IMG_3D, as in the case of generating the movement path MRr described above.
- the robot control unit 316 may calculate at least one of the positions and orientations of the peripheral objects based on at least one of the image data IMG_2D and IMG_3D generated by capturing images of the peripheral objects by at least one of the cameras 21 and 22 that move with the movement of the robot 1.
- the robot control unit 316 may generate the movement path MRf based on at least one of the positions and orientations of the peripheral objects in addition to at least one of the positions and orientations of the end effector model EM, the robot model RM, the peripheral object model SM, and the object OBJ.
- the robot control unit 316 may determine a processing target position PTP based on the object OBJ in the global coordinate system based on at least one of the calculation results of the position and orientation of the object OBJ in the global coordinate system and the processing position and orientation information PAI generated by the processing position and orientation setting unit 313 described above (i.e., the position of the end effector 4 based on the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ in a coordinate system different from the global coordinate system). Thereafter, the robot control unit 316 may generate a movement path MRf for moving the tool center point TCP so that the tool center point TCP approaches the processing target position PTP.
- the robot control unit 316 may generate a movement path MRf that satisfies at least one condition that no interference occurs among the end effector model EM, the robot model RM, and a peripheral object model SM in which at least one of the position and orientation is adjusted based on at least one of the calculation results of the position and orientation of the peripheral object. Even in this case, since at least one of the actual position and actual posture of the peripheral object is reflected in the peripheral object model SM, the robot control unit 316 can appropriately generate a movement path MRf that satisfies the condition that no interference occurs between at least one of the end effector 4, the robot 1, and the peripheral object.
- the robot control unit 316 may sequentially calculate at least one of the position and orientation of the peripheral object based on at least one of the image data IMG_2D and IMG_3D. Thereafter, the robot control unit 316 may sequentially generate (in other words, sequentially update) the movement path MRf based on at least one of the position and orientation of the end effector model EM, the robot model RM, the peripheral object model SM, and the object OBJ, as well as at least one of the position and orientation of the peripheral object that is sequentially calculated.
- the robot control unit 316 may generate a point cloud model of the peripheral objects based on the image data IMG_3D, and use the generated point cloud model of the peripheral objects as the peripheral object model SM.
- the robot control unit 316 may generate a point cloud model of the peripheral objects based on the image data IMG_3D generated by the camera 22 moving with the movement of the robot 1 capturing images of the peripheral objects, and use the generated point cloud model of the peripheral objects as the peripheral object model SM.
- the path generation unit 315 may place the peripheral object model SM, which is the point cloud model of the peripheral objects generated based on the image data IMG_3D, in a predetermined virtual space.
- the point cloud model includes a plurality of points associated with three-dimensional coordinates in the global coordinate system.
- the path generating unit 315 may convert the three-dimensional coordinates of each of the multiple points included in the point cloud model into three-dimensional coordinates in the virtual space, and then, in the virtual space, place each of the multiple points included in the point cloud model at a position indicated by the three-dimensional coordinates obtained by the conversion, thereby placing the peripheral object model SM, which is a point cloud model, in the virtual space.
- the robot control unit 316 may generate a movement path MRf based on at least one of the position and the posture of the object OBJ, based on the end effector model EM, the robot model RM, and the peripheral object model SM, which is a point cloud model of the peripheral objects generated based on the image data IMG_3D.
- the robot control unit 316 may determine a processing target position PTP based on the object OBJ in the global coordinate system based on at least one of the calculation results of the position and orientation of the object OBJ in the global coordinate system and the processing position and orientation information PAI generated by the processing position and orientation setting unit 313 described above (i.e., the position of the end effector 4 based on the object OBJ when the end effector 4 performs a predetermined processing on the object OBJ in a coordinate system different from the global coordinate system). Thereafter, the robot control unit 316 may generate a movement path MRf for moving the tool center point TCP so that the tool center point TCP approaches the processing target position PTP.
- the robot control unit 316 may generate a movement path MRf that satisfies at least one condition that no interference occurs among the end effector model EM, the robot model RM, and a peripheral object model SM in which at least one of the position and orientation is adjusted based on at least one of the calculation results of the position and orientation of the peripheral object. Even in this case, because at least one of the actual position and actual orientation of the peripheral object is reflected in the peripheral object model SM, the robot control unit 316 can appropriately generate a movement path MRf that satisfies the condition that no interference occurs between at least one of the end effector 4, the robot 1, and the peripheral object.
- the robot control unit 316 may sequentially generate (update) a point cloud model of the peripheral object based on the image data IMG_3D, and sequentially use the sequentially generated point cloud model of the peripheral object as the peripheral object model SM.
- the robot control unit 316 may sequentially generate (in other words, sequentially update) the movement path MRf based on the end effector model EM, the robot model RM, and the peripheral object model SM, which is the point cloud model of the peripheral object that is sequentially generated (updated) based on the image data IMG_3D.
- the robot control unit 316 may acquire multiple image data IMG_3D generated by the imaging device 2 capturing images of the peripheral object multiple times, and generate a point cloud model of the peripheral object based on the multiple image data IMG_3D.
- the robot control unit 316 may use the point cloud model of the peripheral object generated based on the multiple image data IMG_3D as the peripheral object model SM to generate a movement path MRf that satisfies the condition that no interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM.
- the operation of generating a point cloud model of a peripheral object based on multiple image data IMG_3D will be described later as a first modified example of the fine movement operation.
- the end effector 4 is moving near the surrounding object.
- the end effector 4 performs the holding process of holding the above-mentioned workpiece W
- the end effector 4 is moving near the inner wall of the storage box CB (see FIG. 3(c) above) that defines the storage space in which the workpiece W is stored.
- the robot control unit 316 can appropriately generate a movement path MRf that satisfies the condition that no interference occurs between at least one of the end effector 4, the robot 1, and the surrounding object (e.g., the storage box CB).
- the robot control unit 316 does not need to use the extended end effector model EM_ex described in the second modified example of the path generation process to generate the movement path MRf.
- the robot control unit 316 may generate the movement path MRf using the end effector model EM (i.e., the extended end effector model EM_ex with the extension amount set to zero).
- the robot control unit 316 can generate a movement path MRf that satisfies the condition that no interference occurs between at least one of the end effector 4, the robot 1, and the surrounding object (e.g., the storage box CB).
- multiple objects OBJ may be reflected in the images represented by the image data IMG_2D and IMG_3D.
- the imaging device 2 may capture images of at least multiple objects OBJ to generate image data IMG_2D and IMG_3D, each of which represents an image in which multiple objects OBJ are reflected.
- an object different from the multiple objects OBJ for example, a peripheral object different from the object OBJ, such as a storage box CB
- a peripheral object different from the object OBJ such as a storage box CB
- the imaging device 2 may capture an object different from the multiple objects OBJ in addition to the multiple objects OBJ.
- one of the multiple objects OBJ may be selected as the object OBJ on which the end effector 4 should perform a predetermined process, and at least one object OBJ different from the one object OBJ among the multiple objects OBJ may be regarded as a peripheral object.
- the one object OBJ selected as the object OBJ on which the end effector 4 should perform a predetermined process is referred to as a target object (processing execution object) OBJ_tgt
- the at least one object OBJ regarded as a peripheral object is referred to as a non-target object OBJ_ntgt.
- a non-target object OBJ_ntgt that was not selected as the target object OBJ_tgt in the first period may be selected as the target object OBJ_tgt for which the end effector 4 should perform a predetermined process in a second period different from the first period.
- the non-target object OBJ_ntgt may be considered as an object OBJ that is a candidate for the end effector 4 to perform a predetermined process.
- the target object OBJ_tgt for which the end effector 4 should perform a predetermined process in the first period may be considered as a non-target object OBJ_ntgt in the second period different from the first period.
- the robot control unit 316 After that, after the target object OBJ_tgt is selected, the robot control unit 316 generates a movement path MRf leading to a processing target position PTP for which the end effector 4 performs a predetermined process on the target object OBJ_tgt.
- both the target object OBJ_tgt and the non-target object OBJ_ntgt are objects OBJ for which the end effector 4 performs a predetermined process at a desired timing. Therefore, the shape of the target object OBJ_tgt and the shape of the non-target object OBJ_ntgt may be identical. The size of the target object OBJ_tgt and the size of the non-target object OBJ_ntgt may be identical.
- the state in which "the shape of the target object OBJ_tgt and the shape of the non-target object OBJ_ntgt are identical" may include the state in which "the shape of the target object OBJ_tgt and the shape of the non-target object OBJ_ntgt are completely identical.”
- the state where "the shape of the target object OBJ_tgt and the shape of the non-target object OBJ_ntgt are the same” may include a state where "the shape of the target object OBJ_tgt and the shape of the non-target object OBJ_ntgt are not completely the same, but the difference between the two is so small that they may be considered to be substantially the same.”
- the state where "the size of the target object OBJ_tgt and the size of the non-target object OBJ_ntgt are the same” may include a state where "the size of the target object OBJ_tgt and the size of the non-target object OBJ
- the robot control unit 316 may adjust at least one of the positions and orientations of the peripheral object model SM (specifically, a non-target object model OM_ntgt which is a three-dimensional model of at least one non-target object OBJ_ntgt) in a specified virtual space based on the calculation results of at least one of the positions and orientations of at least one non-target object OBJ_ntgt (i.e., peripheral object).
- a non-target object model OM_ntgt which is a three-dimensional model of at least one non-target object OBJ_ntgt
- the robot control unit 316 may calculate at least one of the positions and orientations of the non-target object OBJ_ntgt together with at least one of the positions and orientations of the target object OBJ_tgt.
- the robot control unit 316 may select the target object OBJ_tgt and calculate at least one of the position and orientation of the target object OBJ_tgt, and then newly acquire at least one of image data IMG_2D and IMG_3D generated by the imaging device 2 capturing an image of at least one non-target object OBJ_ntgt, and calculate at least one of the position and orientation of the non-target object OBJ_ntgt based on at least one of the newly acquired image data IMG_2D and IMG_3D.
- the target object OBJ_tgt may or may not be reflected in an image indicated by at least one of the image data IMG_2D and IMG_3D used to calculate at least one of the position and orientation of the non-target object OBJ_ntgt.
- image indicated by at least one of the image data IMG_2D and IMG_3D used to calculate at least one of the position and orientation of the non-target object OBJ_ntgt all of one or more non-target objects OBJ_ntgt not selected as the target object OBJ_tgt may be reflected, or some of the multiple non-target objects OBJ_ntgt not selected as the target object OBJ_tgt may be reflected.
- the robot control unit 316 may adjust at least one of the position and orientation of the non-target object model OM_ntgt in a predetermined virtual space based on at least one of the position and orientation of at least one non-target object OBJ_ntgt.
- the control device 3 may adjust the position of the non-target object model OM_ntgt in a predetermined virtual space so that the position of the non-target object model OM_ntgt in the predetermined virtual space becomes a position set based on the position of at least one non-target object OBJ_ntgt.
- control device 3 may adjust the position of the non-target object model OM_ntgt in a predetermined virtual space so that the position of the non-target object model OM_ntgt in the predetermined virtual space becomes a position calculated by converting the position of at least one non-target object OBJ_ntgt into a position in the predetermined virtual space.
- control device 3 may place the non-target object model OM_ntgt in the predetermined virtual space at a position set based on the position of at least one non-target object OBJ_ntgt.
- control device 3 may adjust the orientation of the non-target object model OM_ntgt in the predetermined virtual space so that the orientation of the non-target object model OM_ntgt in the predetermined virtual space becomes an orientation set based on the position of at least one non-target object OBJ_ntgt.
- control device 3 may adjust the orientation of the non-target object model OM_ntgt in a predetermined virtual space so that the orientation of the non-target object model OM_ntgt in the predetermined virtual space becomes an orientation calculated by converting the orientation of at least one non-target object OBJ_ntgt into an orientation in the predetermined virtual space.
- control device 3 may place the non-target object model OM_ntgt in the predetermined virtual space in an orientation set based on the orientation of at least one non-target object OBJ_ntgt.
- the robot control unit 316 may adjust at least one of the positions and orientations of the multiple non-target object models OM_ntgt in the predetermined virtual space based on at least one of the positions and orientations of the multiple non-target objects OBJ_ntgt.
- the robot control unit 316 may generate the movement path MRf so that interference between at least one of the end effector model EM, the robot model RM, and at least one non-target object model OM_ntgt does not occur.
- the robot control unit 316 can generate the movement path MRf that satisfies the condition that interference between at least one of the end effector 4, the robot 1, and the non-target object OBJ_ntgt does not occur.
- the robot control unit 316 may generate a point cloud model of at least one non-target object OBJ_ntgt based on image data IMG_3D generated by capturing an image of at least one non-target object OBJ_ntgt. Note that the target object OBJ_tgt may or may not be reflected in the image indicated by the image data IMG_3D used to generate the point cloud model of the non-target object OBJ_ntgt.
- the image indicated by the image data IMG_3D used to generate a point cloud model of the non-target object OBJ_ntgt may include all of one or more non-target objects OBJ_ntgt that were not selected as the target object OBJ_tgt, or may include some of the multiple non-target objects OBJ_ntgt that were not selected as the target object OBJ_tgt.
- the robot control unit 316 may generate a point cloud model of the target object OBJ_tgt and the non-target object OBJ_ntgt based on the image data IMG_3D, and delete the model portion corresponding to the target object OBJ_tgt from the generated point cloud model.
- the robot control unit 316 may perform a crop process on the image data IMG_3D to extract an image portion in which the non-target object OBJ_ntgt is captured from the image represented by the image data IMG_3D, and generate a point cloud model of the non-target object OBJ_ntgt based on the image data IMG_3D on which the crop process has been performed. After that, the robot control unit 316 may use the generated point cloud model of at least one non-target object OBJ_ntgt as at least one non-target object model OM_ntgt.
- the robot control unit 316 may generate a movement path MRf so that interference between at least one of the end effector model EM, the robot model RM, and at least one non-target object model OM_ntgt does not occur.
- the robot control unit 316 can generate a movement path MRf that satisfies the condition that interference between at least one of the end effector 4, the robot 1, and the non-target object OBJ_ntgt does not occur.
- the robot control unit 316 may generate a movement path MRf based on a first peripheral object model SM (i.e., a non-target object model OM_ntgt) corresponding to the non-target object OBJ_ntgt and a second peripheral object model SM corresponding to a peripheral object different from the non-target object OBJ_ntgt.
- a first peripheral object model SM i.e., a non-target object model OM_ntgt
- SM i.e., a non-target object model OM_ntgt
- SM i.e., a second peripheral object model SM corresponding to a peripheral object different from the non-target object OBJ_ntgt.
- the robot control unit 316 may generate a movement path MRf that satisfies the condition that no interference occurs with the first peripheral object model SM and the second peripheral object model SM.
- the robot control unit 316 may obtain a plurality of image data IMG_3D generated by the imaging device 2 capturing an image of the non-target object OBJ_ntgt a plurality of times, and generate a point cloud model of the non-target object OBJ_ntgt based on the plurality of image data IMG_3D.
- the robot control unit 316 may generate a movement path MRf that satisfies the condition that no interference occurs with at least one of the end effector model EM, the robot model RM, and the peripheral object model SM (non-target object model OM_ntgt) by using the point cloud model of the non-target object OBJ_ntgt generated based on the plurality of image data IMG_3D as the non-target object model OM_ntgt.
- the operation of generating a point cloud model of the non-target object OBJ_ntgt (peripheral object) based on the plurality of image data IMG_3D will be described later as a first modified example of the fine movement operation.
- the route generation unit 315 may acquire at least one of the image data IMG_2D and IMG_3D from the imaging device 2 in order to generate a travel route MRr by performing the route generation processing shown in FIG. 19. Thereafter, the route generation unit 315 may perform the above-mentioned matching processing (for example, at least one of the 2D matching processing and the 3D matching processing) based on at least one of the image data IMG_2D and IMG_3D. Thereafter, the route generation unit 315 may display information regarding the result of the matching processing on the display device 35. That is, the route generation unit 315 may display information regarding the result of the matching processing on the route generation screen 3550 used to generate the travel route MRr.
- the route generation unit 315 may display information regarding the result of the matching processing on the route generation screen 3550 used to generate the travel route MRr.
- the matching process selects one object OBJ (i.e., target object OBJ_tgt) on which the end effector 4 is to perform a predetermined process.
- the path generation unit 315 may display information about the target object OBJ_tgt on the path generation screen 3550.
- the imaging device 2 it is preferable that the imaging device 2 is located in a position where it can capture an image of the target object OBJ_tgt while the path generation process shown in FIG. 19 is being performed.
- the target object OBJ_tgt is included in the imaging angle of view of the imaging device 2 (i.e., the imaging angle of view of at least one of the cameras 21 and 22) while the path generation process shown in FIG. 19 is being performed.
- the path generating unit 315 may display a target object model OM_tgt (i.e., the object model OM described above), which is a three-dimensional model representing the target object OBJ_tgt, as information related to the target object OBJ_tgt.
- the path generating unit 315 may place the target object model OM_tgt in a position corresponding to the position of the target object OBJ_tgt calculated by the matching process in a virtual three-dimensional space SP5 constructed in the path generating screen 3550.
- the path generating unit 315 may place the target object model OM_tgt in a posture corresponding to the posture of the target object OBJ_tgt calculated by the matching process in a virtual three-dimensional space SP5 constructed in the path generating screen 3550.
- the user can appropriately grasp at least one of the actual position and posture of the target object OBJ_tgt.
- the user can input instructions to the control device 3 to change the movement path MRr while taking into account at least one of the actual position and orientation of the target object OBJ_tgt.
- the path generation screen 3550 displayed in the fourth modified example of the path generation process includes at least one of the end effector model EM, the robot model RM, and the surrounding object model SM in addition to the target object model OM_tgt.
- the path generation screen 3550 displayed in the fourth modified example of the path generation process does not have to include at least one of the end effector model EM, the robot model RM, and the surrounding object model SM.
- the target object model OM_tgt may be a CAD model of the target object OBJ_tgt.
- the path generation unit 315 may use the three-dimensional model used as a template in the above-mentioned matching process as the target object model OM_tgt.
- the path generating unit 315 may generate a point cloud model (or other types of three-dimensional models such as polygon models, mesh models or solid models, the same below in the fourth variant of the path generating process) of the target object OBJ_tgt based on image data IMG_3D generated by capturing an image of the target object OBJ_tgt. Specifically, the path generating unit 315 may generate three-dimensional position data WSD based on the image data IMG_3D. In this case, the three-dimensional position data WSD may be data indicating the three-dimensional positions of each of multiple points of the target object OBJ_tgt.
- the path generating unit 315 may generate data indicating a point cloud model of the target object OBJ_tgt as the three-dimensional position data WSD.
- the path generating unit 315 may generate three-dimensional position data WSD (e.g., depth image data) different from the data indicating the point cloud model of the target object OBJ_tgt, and generate the point cloud model of the target object OBJ_tgt based on the three-dimensional position data WSD.
- the point cloud model of the target object OBJ_tgt may be regarded as information based on the three-dimensional position data WSD.
- the path generating unit 315 may display the point cloud model of the target object OBJ_tgt together with the target object model OM_tgt in the virtual three-dimensional space SP5 constructed in the path generating screen 3550.
- the point cloud model includes a plurality of points associated with three-dimensional coordinates in the global coordinate system.
- the path generation unit 315 may convert the three-dimensional coordinates of each of the multiple points included in the point cloud model into three-dimensional coordinates in a virtual three-dimensional space SP5 in the path generation screen 3550, and then place each of the multiple points included in the point cloud model in the three-dimensional space SP5 at a position indicated by the three-dimensional coordinates obtained by the conversion, thereby placing the point cloud model of the target object OBJ_tgt in the three-dimensional space SP5.
- the path generation screen 3550 displays the target object model OM_tgt arranged at the position calculated by the 2D matching process based on the image data IMG_2D and with the attitude calculated by the 2D matching process based on the image data IMG_2D, and the point cloud model of the target object OBJ_tgt generated based on the image data IMG_3D.
- the path generation unit 315 may evaluate the reliability of the result of the 2D matching process (particularly, the calculation result of at least one of the position and attitude of the target object OBJ_tgt) based on the deviation between the target object model OM_tgt and the point cloud model of the target object OBJ_tgt in the virtual three-dimensional space SP5. For example, the path generation unit 315 may evaluate the reliability of the result of the 2D matching process as lower the greater the deviation between the target object model OM_tgt and the point cloud model of the target object OBJ_tgt in the virtual three-dimensional space SP5.
- the user may check the deviation between the target object model OM_tgt and the point cloud model of the target object OBJ_tgt in the virtual three-dimensional space SP5. In this case, the user may evaluate the reliability of the result of the 2D matching process before the processing system SYS starts the above-mentioned operation.
- the user may check the deviation between the target object model OM_tgt and the point cloud model of the target object OBJ_tgt in the virtual three-dimensional space SP5. In this case, the user may also evaluate the reliability of the result of the 2D matching process.
- the user may set the non-target object OBJ_ntgt as the new target object OBJ_tgt. This is because when there is a large deviation between the target object model OM_tgt and the point cloud model of the target object OBJ_tgt, there is a possibility that there will be a small deviation between the non-target object model OM_ntgt and the point cloud model of the target object OBJ_tgt.
- the objects OBJ other than the target object OBJ_tgt are treated as non-target objects OBJ_ntgt for which the end effector 4 does not currently perform a predetermined process by the matching process.
- the path generating unit 315 may display information about the non-target object OBJ_ntgt on the path generating screen 3550.
- the imaging device 2 is located in a position where it can capture the image of the non-target object OBJ_ntgt during the period in which the path generating process shown in FIG. 19 is being performed.
- the non-target object OBJ_ntgt is included in the imaging angle of view of the imaging device 2 (i.e., the imaging angle of view of at least one of the cameras 21 and 22) during the period in which the path generating process shown in FIG. 19 is being performed.
- the path generating unit 315 may display a non-target object model OM_ntgt, which is a three-dimensional model showing the non-target object OBJ_ntgt, as information related to the non-target object OBJ_ntgt.
- the path generating unit 315 may place the non-target object model OM_ntgt in a position corresponding to the position of the non-target object OBJ_ntgt calculated by the matching process in a virtual three-dimensional space SP5 constructed in the path generating screen 3550.
- the path generating unit 315 may place the non-target object model OM_ntgt in a posture corresponding to the posture of the non-target object OBJ_ntgt calculated by the matching process in a virtual three-dimensional space SP5 constructed in the path generating screen 3550.
- the user can appropriately grasp at least one of the actual position and posture of the non-target object OBJ_ntgt.
- the user can input instructions to the control device 3 to change the movement path MRr while taking into account at least one of the actual position and orientation of the non-target object OBJ_ntgt.
- the non-target object model OM_ntgt may be a CAD model of the non-target object OBJ_ntgt.
- the path generation unit 315 may use the three-dimensional model used as a template in the above-mentioned matching process as the non-target object model OM_ntgt.
- the path generating unit 315 may generate a point cloud model of the non-target object OBJ_ntgt based on image data IMG_3D generated by capturing an image of the non-target object OBJ_ntgt. Specifically, the path generating unit 315 may generate three-dimensional position data WSD based on the image data IMG_3D. In this case, the three-dimensional position data WSD may be data indicating the three-dimensional positions of each of the multiple points of the non-target object OBJ_ntgt. The path generating unit 315 may generate data indicating a point cloud model of the non-target object OBJ_ntgt as the three-dimensional position data WSD.
- the path generating unit 315 may generate three-dimensional position data WSD (e.g., depth image data) different from the data indicating the point cloud model of the non-target object OBJ_ntgt, and generate a point cloud model of the non-target object OBJ_ntgt based on the three-dimensional position data WSD.
- the point cloud model of the non-target object OBJ_ntgt may be regarded as information based on the three-dimensional position data WSD.
- the path generating unit 315 may display the point cloud model of the non-target object OBJ_ntgt together with the non-target object model OM_ntgt in a virtual three-dimensional space SP5 constructed in the path generating screen 3550.
- the point cloud model includes a plurality of points associated with three-dimensional coordinates in the global coordinate system.
- the path generating unit 315 may convert the three-dimensional coordinates of each of the plurality of points included in the point cloud model into three-dimensional coordinates in the virtual three-dimensional space SP5 in the path generating screen 3550, and then, in the three-dimensional space SP5, place each of the plurality of points included in the point cloud model at a position indicated by the three-dimensional coordinates obtained by the conversion, thereby placing the point cloud model of the non-target object OBJ_ntgt in the three-dimensional space SP5.
- the path generation screen 3550 displays the non-target object model OM_ntgt arranged at the position calculated by the 2D matching process based on the image data IMG_2D and with the attitude calculated by the 2D matching process based on the image data IMG_2D, and the point cloud model of the non-target object OBJ_ntgt generated based on the image data IMG_3D.
- the path generation unit 315 may evaluate the reliability of the result of the 2D matching process (in particular, the calculation result of at least one of the position and attitude of the non-target object OBJ_ntgt) based on the deviation between the non-target object model OM_ntgt and the point cloud model of the non-target object OBJ_ntgt in the virtual three-dimensional space SP5. For example, the path generation unit 315 may evaluate the reliability of the results of the 2D matching process as being lower the greater the deviation between the non-target object model OM_ntgt and the non-target object point cloud model OBJ_ntgt in the virtual three-dimensional space SP5.
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Automation & Control Theory (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Numerical Control (AREA)
- Manipulator (AREA)
Priority Applications (5)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP23930676.4A EP4691704A1 (en) | 2023-03-31 | 2023-03-31 | Computing device, robot control method, processing method, display method, computer program, recording medium, path generation method, and conveyance method |
| PCT/JP2023/013635 WO2024202042A1 (ja) | 2023-03-31 | 2023-03-31 | 演算装置、ロボット制御方法、処理方法、表示方法、コンピュータプログラム、記録媒体、経路生成方法及び搬送方法 |
| JP2025509624A JPWO2024202042A1 (cg-RX-API-DMAC7.html) | 2023-03-31 | 2023-03-31 | |
| KR1020257035897A KR20250167039A (ko) | 2023-03-31 | 2023-03-31 | 연산 장치, 로봇 제어 방법, 처리 방법, 표시 방법, 컴퓨터 프로그램, 기록 매체, 경로 생성 방법 및 반송 방법 |
| CN202380096488.XA CN120957843A (zh) | 2023-03-31 | 2023-03-31 | 运算装置、机器人控制方法、处理方法、显示方法、计算机程序、记录介质、路径生成方法及搬运方法 |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2023/013635 WO2024202042A1 (ja) | 2023-03-31 | 2023-03-31 | 演算装置、ロボット制御方法、処理方法、表示方法、コンピュータプログラム、記録媒体、経路生成方法及び搬送方法 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2024202042A1 true WO2024202042A1 (ja) | 2024-10-03 |
Family
ID=92904539
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2023/013635 Ceased WO2024202042A1 (ja) | 2023-03-31 | 2023-03-31 | 演算装置、ロボット制御方法、処理方法、表示方法、コンピュータプログラム、記録媒体、経路生成方法及び搬送方法 |
Country Status (5)
| Country | Link |
|---|---|
| EP (1) | EP4691704A1 (cg-RX-API-DMAC7.html) |
| JP (1) | JPWO2024202042A1 (cg-RX-API-DMAC7.html) |
| KR (1) | KR20250167039A (cg-RX-API-DMAC7.html) |
| CN (1) | CN120957843A (cg-RX-API-DMAC7.html) |
| WO (1) | WO2024202042A1 (cg-RX-API-DMAC7.html) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06126662A (ja) * | 1992-10-22 | 1994-05-10 | Toshiba Corp | 産業用ロボット |
| JPH0736519A (ja) * | 1993-07-23 | 1995-02-07 | Kobe Steel Ltd | ロボットのニアミスチェック方法 |
| US20130311154A1 (en) | 2012-05-18 | 2013-11-21 | Fanuc Corporation | Operation simulation system of robot system |
| JP6915543B2 (ja) * | 2015-10-30 | 2021-08-04 | 株式会社安川電機 | ロボット教示装置、コンピュータプログラム及びロボット教示方法 |
| WO2022131335A1 (ja) * | 2020-12-18 | 2022-06-23 | 川崎重工業株式会社 | 制御装置、ロボットシステム及び学習装置 |
| JP2022122648A (ja) * | 2021-02-10 | 2022-08-23 | 株式会社キーエンス | 制御装置 |
-
2023
- 2023-03-31 WO PCT/JP2023/013635 patent/WO2024202042A1/ja not_active Ceased
- 2023-03-31 CN CN202380096488.XA patent/CN120957843A/zh active Pending
- 2023-03-31 EP EP23930676.4A patent/EP4691704A1/en active Pending
- 2023-03-31 KR KR1020257035897A patent/KR20250167039A/ko active Pending
- 2023-03-31 JP JP2025509624A patent/JPWO2024202042A1/ja active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06126662A (ja) * | 1992-10-22 | 1994-05-10 | Toshiba Corp | 産業用ロボット |
| JPH0736519A (ja) * | 1993-07-23 | 1995-02-07 | Kobe Steel Ltd | ロボットのニアミスチェック方法 |
| US20130311154A1 (en) | 2012-05-18 | 2013-11-21 | Fanuc Corporation | Operation simulation system of robot system |
| JP6915543B2 (ja) * | 2015-10-30 | 2021-08-04 | 株式会社安川電機 | ロボット教示装置、コンピュータプログラム及びロボット教示方法 |
| WO2022131335A1 (ja) * | 2020-12-18 | 2022-06-23 | 川崎重工業株式会社 | 制御装置、ロボットシステム及び学習装置 |
| JP2022122648A (ja) * | 2021-02-10 | 2022-08-23 | 株式会社キーエンス | 制御装置 |
Also Published As
| Publication number | Publication date |
|---|---|
| KR20250167039A (ko) | 2025-11-28 |
| JPWO2024202042A1 (cg-RX-API-DMAC7.html) | 2024-10-03 |
| EP4691704A1 (en) | 2026-02-11 |
| CN120957843A (zh) | 2025-11-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Pagano et al. | A vision guided robotic system for flexible gluing process in the footwear industry | |
| Barth et al. | Design of an eye-in-hand sensing and servo control framework for harvesting robotics in dense vegetation | |
| Rong et al. | A peduncle detection method of tomato for autonomous harvesting | |
| US11386289B2 (en) | Systems and methods for robot-aided product inspection | |
| US20210129326A1 (en) | Inspection planning using robotic systems | |
| US20210129339A1 (en) | Calibration and zeroing in robotic systems | |
| CN118542162B (zh) | 基于机器视觉的果树自动修剪系统 | |
| CN111061266A (zh) | 一种实时场景分析和空间避障的夜间值守机器人 | |
| Singh et al. | A deep learning-based approach to strawberry grasping using a telescopic-link differential drive mobile robot in ros-gazebo for greenhouse digital twin environments | |
| US12585292B2 (en) | Location based change detection within image data by a mobile robot | |
| US20210129328A1 (en) | Braking stepper motors in robotic systems | |
| ES2663537T3 (es) | Procedimiento y dispositivo de simulación para la medición de un componente y la optimización de la medición real correspondiente | |
| CN106780566B (zh) | 一种基于Delta机器人下的目标跟踪的方法及系统 | |
| Bao et al. | 3D perception-based collision-free robotic leaf probing for automated indoor plant phenotyping | |
| US20210129340A1 (en) | End effectors in robotic systems | |
| Huang et al. | Autonomous navigation system in various greenhouse scenarios based on improved FAST-LIO2 | |
| Csaba et al. | Differences between Kinect and structured lighting sensor in robot navigation | |
| KR20250167039A (ko) | 연산 장치, 로봇 제어 방법, 처리 방법, 표시 방법, 컴퓨터 프로그램, 기록 매체, 경로 생성 방법 및 반송 방법 | |
| CN106558070B (zh) | 一种基于Delta机器人下的视觉跟踪的方法及系统 | |
| CN115235470A (zh) | 一种基于视觉检测的农业机器人定位导航方法及系统 | |
| Bao et al. | Robotic 3D plant perception and leaf probing with collision-free motion planning for automated indoor plant phenotyping | |
| US20250022158A1 (en) | Optimized method for identificating robot pouring regions based on hierarchical processing and connectivity maximization and system thereof | |
| Baslan et al. | Navigation and vision system of a mobile robot | |
| Geer et al. | Novel software architecture for an autonomous agricultural robotic fruit harvesting system | |
| Tselishchev et al. | Autonomous navigation of a mobile robot using overhead camera and computer vision methods for time-critical tasks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23930676 Country of ref document: EP Kind code of ref document: A1 |
|
| ENP | Entry into the national phase |
Ref document number: 2025509624 Country of ref document: JP Kind code of ref document: A |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 2025509624 Country of ref document: JP |
|
| ENP | Entry into the national phase |
Ref document number: 1020257035897 Country of ref document: KR Free format text: ST27 STATUS EVENT CODE: A-0-1-A10-A15-NAP-PA0105 (AS PROVIDED BY THE NATIONAL OFFICE) |
|
| WWE | Wipo information: entry into national phase |
Ref document number: KR1020257035897 Country of ref document: KR |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 2023930676 Country of ref document: EP |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| ENP | Entry into the national phase |
Ref document number: 2023930676 Country of ref document: EP Effective date: 20251031 |
|
| ENP | Entry into the national phase |
Ref document number: 2023930676 Country of ref document: EP Effective date: 20251031 |
|
| ENP | Entry into the national phase |
Ref document number: 2023930676 Country of ref document: EP Effective date: 20251031 |
|
| ENP | Entry into the national phase |
Ref document number: 2023930676 Country of ref document: EP Effective date: 20251031 |
|
| ENP | Entry into the national phase |
Ref document number: 2023930676 Country of ref document: EP Effective date: 20251031 |
|
| ENP | Entry into the national phase |
Ref document number: 2023930676 Country of ref document: EP Effective date: 20251031 |
|
| ENP | Entry into the national phase |
Ref document number: 2023930676 Country of ref document: EP Effective date: 20251031 |
|
| ENP | Entry into the national phase |
Ref document number: 2023930676 Country of ref document: EP Effective date: 20251031 |
|
| WWP | Wipo information: published in national office |
Ref document number: 2023930676 Country of ref document: EP |